Re: [swift-evolution] Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices

[Karl Wagner via swift-evolution]

> On 24 Mar 2017, at 07:49, Andrew Trick  wrote:
> 
> 
>> On Mar 23, 2017, at 7:34 PM, Karl Wagner > > wrote:
>> 
>> Oh, one more thing (about this specific change): If we do this, we should 
>> add an “offset” parameter to 
>> UnsafeMutableBufferPointer.[move]initialize/assign (with a default of 0 for 
>> source compatibility). Otherwise, it becomes awkward to initialise a region 
>> of a buffer from another buffer.
>> 
>> What I want to write:
>> buffer.suffix(from: filled).initialize(from: newData)
>> 
>> If SubSequence is not another unsafe pointer, I’d have to do this:
>> buffer.baseAddress!.advanced(by: filled).initialize(from: newData)  // 
>> Warning: deprecated in Swift 4.0
> 
> Oh no. Don't do that. You're supposed to do:
> 
> UnsafeMutableRawBufferPointer(rebasing: buffer.suffix(from: 
> filled)).initialize(from: newData)
> 
> I don't blame you for not wanting to write out the type name.
> 
>> 
>> So instead, we should have the ability to write this:
>> buffer.initialize(startingAt: filled, from: newData) // awkward labels, but 
>> src compat…
> 
> I agree that's worth doing as a result of changing the slice type. You
> shouldn't need to "rebase" the buffer each time you want to initialize
> a region. My concern is that there are a handful of APIs to fix
> (initialize, initializeMemory, copyBytes, load). We should handle them
> all in one proposal, and we may want time to bikeshed the API. I'm
> unconvinced this is worth pursuing for Swift 4. Can you file a bug?
> 

It might be related to the existing bug about unsafely initialising an Array’s 
storage (SR-3087). We could add these methods as a protocol extension or 
refined protocol on “UnsafelyInitializable”.

Then, in a similar fashion to SR-3631, which would introduce ContiguouslyStored 
and remove ArraySlice, Slice could conditionally conform to that protocol 
when its base does. That would make the first example — 
buffer.suffix().initialize(from:), work.

https://bugs.swift.org/browse/SR-3087
https://bugs.swift.org/browse/SR-3631


Fleshing out those unsafe-initialisation protocols would be useful in general, 
not just for slices of unsafe buffers. For example, if you’re initialising such 
any such Collection from something which is ContiguouslyStored, we could 
optimise that operation quite a lot. Currently, the standard library has 
private, underscored methods on Sequence to do that: 
“_copyContentsToContiguousArray” and “_copyContents(initializing:)”.

I hope unsafe-initialisation is something which can make Swift 4. It would 
align well with other features in that release.

Where ownership is concerned, I feel we still need to express how code which 
wishes to avoid copies should get its data in to standard-library datatypes: 
the ownership manifesto only covers data at the Swift-type level, and provides 
no way to express ownership over an arbitrary buffer of allocated memory (that 
you might, for example, be able to safely wrap as an Array or String). Unsafe 
initialisation is a great answer to this - the standard library type owns its 
backing, which remains a private implementation detail, but you get the 
opportunity to have it allocate a buffer for you to fill up directly, using 
whichever unsafe operations you like and eliminating copies.

In particular, the new String model will expose its backing as a contiguous 
RandomAccessCollection of CodeUnits. It would be totally badass if we could 
unsafely initialise a standard Swift.String, using its backing storage directly 
as the buffer for file read/network recv operations.

- Karl

>> 
>> But yeah, this post just reminded me that there are a number of small 
>> consistency tweaks we could make to the unsafe-buffer API.
> 
> Yeah, but these are convenience issues, not correctness issues.
> 
> I think these are the ones that are important, obvious, and trivial
> enough that might make sense in Swift 4.
> 
> ** UnsafeBufferPointer should have init from mutable
> https://bugs.swift.org/browse/SR-3929 
> 
> ** UnsafeMutableBufferPointer doesn't have an allocating init
> https://bugs.swift.org/browse/SR-3088 
> 
> ** UnsafeBufferPointer needs a withMemoryRebound method
> https://bugs.swift.org/browse/SR-4340 
> 
> ** Implicit Conversion:  to UnsafePointer
> https://bugs.swift.org/browse/SR-3590 
> 
> -Andy
> 
>> 
>> - Karl
>> 
>>> On 24 Mar 2017, at 03:22, Karl Wagner >> > wrote:
>>> 
>>> The convenience initialiser should exist on all of the unsafe buffers, not 
>>> just the raw (untyped) ones.
>>> 
>>> I’ve run in to this problem a few times, and I think it would get worse if 
>>> we adopted a ContiguouslyStored protocol to formalise accessing the 
>>> raw-pointers of generic collections. It would 

Re: [swift-evolution] Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices

[Andrew Trick via swift-evolution]

> On Mar 23, 2017, at 7:34 PM, Karl Wagner  wrote:
> 
> Oh, one more thing (about this specific change): If we do this, we should add 
> an “offset” parameter to UnsafeMutableBufferPointer.[move]initialize/assign 
> (with a default of 0 for source compatibility). Otherwise, it becomes awkward 
> to initialise a region of a buffer from another buffer.
> 
> What I want to write:
> buffer.suffix(from: filled).initialize(from: newData)
> 
> If SubSequence is not another unsafe pointer, I’d have to do this:
> buffer.baseAddress!.advanced(by: filled).initialize(from: newData)  // 
> Warning: deprecated in Swift 4.0

Oh no. Don't do that. You're supposed to do:

UnsafeMutableRawBufferPointer(rebasing: buffer.suffix(from: 
filled)).initialize(from: newData)

I don't blame you for not wanting to write out the type name.

> 
> So instead, we should have the ability to write this:
> buffer.initialize(startingAt: filled, from: newData) // awkward labels, but 
> src compat…

I agree that's worth doing as a result of changing the slice type. You
shouldn't need to "rebase" the buffer each time you want to initialize
a region. My concern is that there are a handful of APIs to fix
(initialize, initializeMemory, copyBytes, load). We should handle them
all in one proposal, and we may want time to bikeshed the API. I'm
unconvinced this is worth pursuing for Swift 4. Can you file a bug?

> 
> But yeah, this post just reminded me that there are a number of small 
> consistency tweaks we could make to the unsafe-buffer API.

Yeah, but these are convenience issues, not correctness issues.

I think these are the ones that are important, obvious, and trivial
enough that might make sense in Swift 4.

** UnsafeBufferPointer should have init from mutable
https://bugs.swift.org/browse/SR-3929

** UnsafeMutableBufferPointer doesn't have an allocating init
https://bugs.swift.org/browse/SR-3088

** UnsafeBufferPointer needs a withMemoryRebound method
https://bugs.swift.org/browse/SR-4340

** Implicit Conversion:  to UnsafePointer
https://bugs.swift.org/browse/SR-3590

-Andy

> 
> - Karl
> 
>> On 24 Mar 2017, at 03:22, Karl Wagner > > wrote:
>> 
>> The convenience initialiser should exist on all of the unsafe buffers, not 
>> just the raw (untyped) ones.
>> 
>> I’ve run in to this problem a few times, and I think it would get worse if 
>> we adopted a ContiguouslyStored protocol to formalise accessing the 
>> raw-pointers of generic collections. It would mean that you couldn’t write 
>> code that works with UnsafeRawBufferPointer/Data/DispatchData generically, 
>> or with UnsafeBufferPointer/Array.
>> 
>> Also, there seem to be some implicit conversions for the unsafe-pointer 
>> types, but UMBP -> UBP requires an awkward initialiser. We should introduce 
>> an implicit conversion for that case or add an “immutable” computed property 
>> to UMBP.
>> 
>> And while we’re on the subject, memory allocation/deallocation functions are 
>> weirdly dispersed. In order to allocate an UnsafeMutableBufferPointer, 
>> for instance, you have to do:
>> 
>> var buffer: UnsafeMutableBufferPointer
>> init(length: Int) {
>>   let b  = UnsafeMutablePointer.allocate(capacity: length)
>>   buffer = UnsafeMutableBufferPointer(start: b, count: length)
>> }
>> 
>> Also, the deallocate API feels weird - since it deallocates n items from the 
>> head of the pointer, it is a consuming operation and I feel like it should 
>> return a new pointer (with @discardableResult). Once you’ve deallocated a 
>> memory address, you can never re-allocate that specific location so there is 
>> no reason to know about it any more.
>> 
>> - Karl
>> 
>>> On 21 Mar 2017, at 03:21, Andrew Trick via swift-evolution 
>>> > wrote:
>>> 
>>> This proposal amends SE-0138: Normalize UnsafeRawBufferPointer Slices
>>> to fix a design bug: https://github.com/apple/swift-evolution/pull/651 
>>> 
>>> 
>>> The issue was discussed on swift-evolution in Nov/Dec:
>>> See [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers
>>> https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161128/029108.html
>>>  
>>> 
>>> 
>>> The implementation of this fix is in PR #8222:
>>> https://github.com/apple/swift/pull/8222 
>>> 
>>> 
>>> Fix: Change Unsafe[Mutable]RawBufferPointer's SubSequence type
>>> 
>>> Original: Unsafe[Mutable]RawBufferPointer.SubSequence = 
>>> Unsafe[Mutable]RawBufferPointer
>>> 
>>> Fixed: Unsafe[Mutable]RawBufferPointer.SubSequence = 
>>> [Mutable]RandomAccessSlice
>>> 
>>> This is a source breaking bug fix that only applies to
>>> post-3.0.1. It's extremely unlikely that any Swift 3 code would rely
>>> on the 

Re: [swift-evolution] Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices

[Andrew Trick via swift-evolution]

This is all valuable feedback. I also have a bunch of convenience APIs in
mind but haven't been pushing them yet because:

- the Swift 4 schedule is tight

- I don't want to speculatively add API surface until enough users
  have had experience with the feature. Maybe there are better ideas.
…

> On Mar 23, 2017, at 7:22 PM, Karl Wagner  wrote:
> 
> The convenience initialiser should exist on all of the unsafe buffers, not 
> just the raw (untyped) ones.

I think it's fair to add the UnsafeBuffer.init(rebasing:) initializer
as part of this SE-0138 amendment. It's not so much a new API as a
consistency fix. I’ll update the current proposal.

> I’ve run in to this problem a few times, and I think it would get worse if we 
> adopted a ContiguouslyStored protocol to formalise accessing the raw-pointers 
> of generic collections. It would mean that you couldn’t write code that works 
> with UnsafeRawBufferPointer/Data/DispatchData generically, or with 
> UnsafeBufferPointer/Array.
> 
> Also, there seem to be some implicit conversions for the unsafe-pointer 
> types, but UMBP -> UBP requires an awkward initialiser. We should introduce 
> an implicit conversion for that case or add an “immutable” computed property 
> to UMBP.

UnsafeBufferPointer should have init from mutable. We already had a bug for 
that.
Thanks for reminding me: https://bugs.swift.org/browse/SR-3929

I think that needs a new proposal, but it's a trivial addition. I can work on 
that.

I agree that implicit conversions would be nice but does merit some
discussion and involves a bit of type system work. You have to be a
bit careful with implicit conversion because of potentially ambigous
overloads. Implicit conversion is primarily for C interop. Eventually
I do think it would be nice to optionally import some pointer+length
arguments as UnsafeBuffer. We just don't do it now. Regardless of
interop, this would also be handy way to pass an inout argument as a
buffer without wrapping it in a closure.

Your specific concern seems to be about the mutable -> immutable
conversion being implicit though. But that just comes down to writing out
the type name. As bad as the type name is, to me that's not worth an
evolution proposal at the moment. I suggest filing a bug for any
implicit conversion issues, or `immutable` property suggestions for now.

> And while we’re on the subject, memory allocation/deallocation functions are 
> weirdly dispersed. In order to allocate an UnsafeMutableBufferPointer, for 
> instance, you have to do:
> 
> var buffer: UnsafeMutableBufferPointer
> init(length: Int) {
>   let b  = UnsafeMutablePointer.allocate(capacity: length)
>   buffer = UnsafeMutableBufferPointer(start: b, count: length)
> }

We've had a bug open on this since introducing raw buffers:
https://bugs.swift.org/browse/SR-3088

This is obvious enough that I can probably sneak it into Swift 4 if I just get 
around to writing the proposal.

> Also, the deallocate API feels weird - since it deallocates n items from the 
> head of the pointer, it is a consuming operation and I feel like it should 
> return a new pointer (with @discardableResult). Once you’ve deallocated a 
> memory address, you can never re-allocate that specific location so there is 
> no reason to know about it any more.

Now we're outside UnsafeBufferPointer territory.
UnsafePointer.deallocate(capacity:) needs to be passed the
same capacity as allocate. If that's confusing, please file a
documentation bug.

Or you may be referring to UnsafePointer.deinitialize(count:). That
returns a raw pointer to the same memory address to indicate that you
can now initialize the same memory as a different type.

With move-only types, I think we'll be able to revisit some of the
pointer initialize/deinitialize API to be more robust.

-Andy

> 
> - Karl
> 
>> On 21 Mar 2017, at 03:21, Andrew Trick via swift-evolution 
>> > wrote:
>> 
>> This proposal amends SE-0138: Normalize UnsafeRawBufferPointer Slices
>> to fix a design bug: https://github.com/apple/swift-evolution/pull/651 
>> 
>> 
>> The issue was discussed on swift-evolution in Nov/Dec:
>> See [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers
>> https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161128/029108.html
>>  
>> 
>> 
>> The implementation of this fix is in PR #8222:
>> https://github.com/apple/swift/pull/8222
>> 
>> Fix: Change Unsafe[Mutable]RawBufferPointer's SubSequence type
>> 
>> Original: Unsafe[Mutable]RawBufferPointer.SubSequence = 
>> Unsafe[Mutable]RawBufferPointer
>> 
>> Fixed: Unsafe[Mutable]RawBufferPointer.SubSequence = 
>> [Mutable]RandomAccessSlice
>> 
>> This is a source breaking bug fix that only applies to
>> post-3.0.1. It's extremely unlikely that 

Re: [swift-evolution] Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices

[Karl Wagner via swift-evolution]

Oh, one more thing (about this specific change): If we do this, we should add 
an “offset” parameter to UnsafeMutableBufferPointer.[move]initialize/assign 
(with a default of 0 for source compatibility). Otherwise, it becomes awkward 
to initialise a region of a buffer from another buffer.

What I want to write:
buffer.suffix(from: filled).initialize(from: newData)

If SubSequence is not another unsafe pointer, I’d have to do this:
buffer.baseAddress!.advanced(by: filled).initialize(from: newData)  // Warning: 
deprecated in Swift 4.0

So instead, we should have the ability to write this:
buffer.initialize(startingAt: filled, from: newData) // awkward labels, but src 
compat...

But yeah, this post just reminded me that there are a number of small 
consistency tweaks we could make to the unsafe-buffer API.

- Karl

> On 24 Mar 2017, at 03:22, Karl Wagner  wrote:
> 
> The convenience initialiser should exist on all of the unsafe buffers, not 
> just the raw (untyped) ones.
> 
> I’ve run in to this problem a few times, and I think it would get worse if we 
> adopted a ContiguouslyStored protocol to formalise accessing the raw-pointers 
> of generic collections. It would mean that you couldn’t write code that works 
> with UnsafeRawBufferPointer/Data/DispatchData generically, or with 
> UnsafeBufferPointer/Array.
> 
> Also, there seem to be some implicit conversions for the unsafe-pointer 
> types, but UMBP -> UBP requires an awkward initialiser. We should introduce 
> an implicit conversion for that case or add an “immutable” computed property 
> to UMBP.
> 
> And while we’re on the subject, memory allocation/deallocation functions are 
> weirdly dispersed. In order to allocate an UnsafeMutableBufferPointer, for 
> instance, you have to do:
> 
> var buffer: UnsafeMutableBufferPointer
> init(length: Int) {
>   let b  = UnsafeMutablePointer.allocate(capacity: length)
>   buffer = UnsafeMutableBufferPointer(start: b, count: length)
> }
> 
> Also, the deallocate API feels weird - since it deallocates n items from the 
> head of the pointer, it is a consuming operation and I feel like it should 
> return a new pointer (with @discardableResult). Once you’ve deallocated a 
> memory address, you can never re-allocate that specific location so there is 
> no reason to know about it any more.
> 
> - Karl
> 
>> On 21 Mar 2017, at 03:21, Andrew Trick via swift-evolution 
>> > wrote:
>> 
>> This proposal amends SE-0138: Normalize UnsafeRawBufferPointer Slices
>> to fix a design bug: https://github.com/apple/swift-evolution/pull/651 
>> 
>> 
>> The issue was discussed on swift-evolution in Nov/Dec:
>> See [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers
>> https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161128/029108.html
>>  
>> 
>> 
>> The implementation of this fix is in PR #8222:
>> https://github.com/apple/swift/pull/8222
>> 
>> Fix: Change Unsafe[Mutable]RawBufferPointer's SubSequence type
>> 
>> Original: Unsafe[Mutable]RawBufferPointer.SubSequence = 
>> Unsafe[Mutable]RawBufferPointer
>> 
>> Fixed: Unsafe[Mutable]RawBufferPointer.SubSequence = 
>> [Mutable]RandomAccessSlice
>> 
>> This is a source breaking bug fix that only applies to
>> post-3.0.1. It's extremely unlikely that any Swift 3 code would rely
>> on the SubSequence type beyond the simple use case of passing a
>> raw buffer subrange to an another raw buffer argument:
>> 
>> `takesRawBuffer(buffer[i..> 
>> A diagnostic message now instructs users to convert the slice to a
>> buffer using a `rebasing` initializer:
>> 
>> `takesRawBuffer(UnsafeRawBufferPointer(rebasing: buffer[i..> 
>> To support this, the following `rebasing` initializers are added:
>> 
>> extension UnsafeRawBufferPointer {
>>  public init(rebasing slice: RandomAccessSlice)
>>  public init(
>>rebasing slice: MutableRandomAccessSlice
>>  )
>> }
>> 
>> extension UnsafeMutableRawBufferPointer {
>>  public init(
>>rebasing slice: MutableRandomAccessSlice
>>  )
>> }
>> 
>> The source compatibility test builds are unnaffected by this change.
>> 
>> -Andy
>> ___
>> swift-evolution mailing list
>> swift-evolution@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
> 

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices

[Karl Wagner via swift-evolution]

The convenience initialiser should exist on all of the unsafe buffers, not just 
the raw (untyped) ones.

I’ve run in to this problem a few times, and I think it would get worse if we 
adopted a ContiguouslyStored protocol to formalise accessing the raw-pointers 
of generic collections. It would mean that you couldn’t write code that works 
with UnsafeRawBufferPointer/Data/DispatchData generically, or with 
UnsafeBufferPointer/Array.

Also, there seem to be some implicit conversions for the unsafe-pointer types, 
but UMBP -> UBP requires an awkward initialiser. We should introduce an 
implicit conversion for that case or add an “immutable” computed property to 
UMBP.

And while we’re on the subject, memory allocation/deallocation functions are 
weirdly dispersed. In order to allocate an UnsafeMutableBufferPointer, for 
instance, you have to do:

var buffer: UnsafeMutableBufferPointer
init(length: Int) {
  let b  = UnsafeMutablePointer.allocate(capacity: length)
  buffer = UnsafeMutableBufferPointer(start: b, count: length)
}

Also, the deallocate API feels weird - since it deallocates n items from the 
head of the pointer, it is a consuming operation and I feel like it should 
return a new pointer (with @discardableResult). Once you’ve deallocated a 
memory address, you can never re-allocate that specific location so there is no 
reason to know about it any more.

- Karl

> On 21 Mar 2017, at 03:21, Andrew Trick via swift-evolution 
>  wrote:
> 
> This proposal amends SE-0138: Normalize UnsafeRawBufferPointer Slices
> to fix a design bug: https://github.com/apple/swift-evolution/pull/651
> 
> The issue was discussed on swift-evolution in Nov/Dec:
> See [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers
> https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161128/029108.html
> 
> The implementation of this fix is in PR #8222:
> https://github.com/apple/swift/pull/8222
> 
> Fix: Change Unsafe[Mutable]RawBufferPointer's SubSequence type
> 
> Original: Unsafe[Mutable]RawBufferPointer.SubSequence = 
> Unsafe[Mutable]RawBufferPointer
> 
> Fixed: Unsafe[Mutable]RawBufferPointer.SubSequence = 
> [Mutable]RandomAccessSlice
> 
> This is a source breaking bug fix that only applies to
> post-3.0.1. It's extremely unlikely that any Swift 3 code would rely
> on the SubSequence type beyond the simple use case of passing a
> raw buffer subrange to an another raw buffer argument:
> 
> `takesRawBuffer(buffer[i.. 
> A diagnostic message now instructs users to convert the slice to a
> buffer using a `rebasing` initializer:
> 
> `takesRawBuffer(UnsafeRawBufferPointer(rebasing: buffer[i.. 
> To support this, the following `rebasing` initializers are added:
> 
> extension UnsafeRawBufferPointer {
>  public init(rebasing slice: RandomAccessSlice)
>  public init(
>rebasing slice: MutableRandomAccessSlice
>  )
> }
> 
> extension UnsafeMutableRawBufferPointer {
>  public init(
>rebasing slice: MutableRandomAccessSlice
>  )
> }
> 
> The source compatibility test builds are unnaffected by this change.
> 
> -Andy
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution