Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Kevin Ballard via swift-evolution]

On Fri, Dec 9, 2016, at 11:50 AM, Dave Abrahams via swift-evolution wrote:
> 
> on Fri Dec 09 2016, Andrew Trick  wrote:
> 
> >> On Dec 9, 2016, at 10:27 AM, Dave Abrahams via swift-evolution
> >  wrote:
> >> 
> >> 
> >> on Thu Dec 08 2016, Xiaodi Wu  >> > wrote:
> >> 
> >
> >>> On Thu, Dec 8, 2016 at 6:53 PM, Ben Cohen via swift-evolution <
> >>> swift-evolution@swift.org> wrote:
> >>> 
>  
>  On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution <
>  swift-evolution@swift.org> wrote:
>  
>  Um, Sequence doesn’t have a subscript (or indexes). Sequences are
>  single-pass. So if this is important, it needs to stay a Collection.
>  
>  
>  Just because something fulfills one of the requirements of a Collection
>  does not mean it should be one. It needs to tick all the boxes before its
>  allowed to be elevated.
>  
>  But it’s still allowed to have subscripts (UnsafePointer has subscripting
>  but isn’t a collection) or be multi-pass (strides are multiples but are
>  only sequences). That’s OK
>  
>  In this case, yes it’s multi-pass, yes it has a subscript, but no it 
>  isn’t
>  a collection because it doesn’t meet the requirements for slicing i.e. 
>  that
>  indices of the slice be indices of the parent.
>  (relatedly… it appears this requirement is documented on the concrete
>  Slice type rather than on Collection… which is a documentation bug we
>  should fix).
>  
> >>> 
> >>> If this is indeed a requirement for Collection, then my vote would be for
> >>> Nate's option #1 and Andy's option #2, to give UnsafeRawBufferPointer a
> >>> Slice type that fulfills the requirement. It's the smallest change,
> >>> preserves the use of integer indices, and preserves what Andy stated as 
> >>> the
> >>> desired use case of making it easy for users to switch out code written 
> >>> for
> >>> [UInt8].
> >>> 
> >>> I'm not sure I fully understand yet why Dave finds the idea of Collection
> >>> conformance fishy, 
> >> 
> >> Because the memory can easily be already bound to another type than
> >> UInt8, and there's no obvious reason why UInt8 should be privileged as a
> >> type you can get out of a raw buffer without binding the memory.
> >
> > I strongly disagree with that statement. The overwhelmingly common use
> > case for raw buffers is to view them as a sequence of UInt8 *without*
> > binding the type.  Generally, at the point that you're dealing with a
> > raw buffer it's impossible to (re)bind memory because you don't know
> > what type it holds. 
> 
> Oh, you can't just rebind to UInt8 because that's not defined as
> universally compatible with all data.  OK, sorry.

If you're dealing with raw bytes to begin with then I'd hope you're working 
with types that can be safely expressed as a collection of bytes. And since 
this is an unsafe type, I don't think there's any problem with having the 
possibility of trying to express e.g. [NSObject] as a raw byte buffer.

> > The reason it's so important to have an UnsafeRawBufferPointer data
> > type is precisely so that users don't need mess about with binding
> > memory. It's easy to get that wrong even when it's possible.
> >
> > The only reason that UInt8 is special is that when users create
> > temporary typed buffers for bytes (e.g. they sometimes want a growable
> > array or just don't want to bother with manual allocation) they always
> > use UInt8 as the element type.
> >
> > That said, we could easily divide these concerns into two types as
> > you suggested. A raw buffer, which doesn't have any special UInt8
> > features, and a RawBytes collection that handles both buffer slicing
> > and UInt8 interoperability.
> 
> But, now that I think of it, that wouldn't really solve any problems,
> would it?

Agreed. If we have a separate RawBytes type, then I'm not really sure what the 
point of UnsafeRawBufferPointer is anymore. The whole point of that type is 
it's a collection of bytes rather than just being a tuple `(baseAddress: 
UnsafeRawPointer, count: Int)`.

-Kevin Ballard
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Andrew Trick via swift-evolution]

> On Dec 9, 2016, at 11:50 AM, Dave Abrahams via swift-evolution 
>  wrote:
> 
> 
> on Fri Dec 09 2016, Andrew Trick  wrote:
> 
>>> On Dec 9, 2016, at 10:27 AM, Dave Abrahams via swift-evolution
>>  wrote:
>>> 
>>> 
>>> on Thu Dec 08 2016, Xiaodi Wu >> > wrote:
>>> 
>> 
 On Thu, Dec 8, 2016 at 6:53 PM, Ben Cohen via swift-evolution <
 swift-evolution@swift.org> wrote:
 
> 
> On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution <
> swift-evolution@swift.org> wrote:
> 
> Um, Sequence doesn’t have a subscript (or indexes). Sequences are
> single-pass. So if this is important, it needs to stay a Collection.
> 
> 
> Just because something fulfills one of the requirements of a Collection
> does not mean it should be one. It needs to tick all the boxes before its
> allowed to be elevated.
> 
> But it’s still allowed to have subscripts (UnsafePointer has subscripting
> but isn’t a collection) or be multi-pass (strides are multiples but are
> only sequences). That’s OK
> 
> In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t
> a collection because it doesn’t meet the requirements for slicing i.e. 
> that
> indices of the slice be indices of the parent.
> (relatedly… it appears this requirement is documented on the concrete
> Slice type rather than on Collection… which is a documentation bug we
> should fix).
> 
 
 If this is indeed a requirement for Collection, then my vote would be for
 Nate's option #1 and Andy's option #2, to give UnsafeRawBufferPointer a
 Slice type that fulfills the requirement. It's the smallest change,
 preserves the use of integer indices, and preserves what Andy stated as the
 desired use case of making it easy for users to switch out code written for
 [UInt8].
 
 I'm not sure I fully understand yet why Dave finds the idea of Collection
 conformance fishy, 
>>> 
>>> Because the memory can easily be already bound to another type than
>>> UInt8, and there's no obvious reason why UInt8 should be privileged as a
>>> type you can get out of a raw buffer without binding the memory.
>> 
>> I strongly disagree with that statement. The overwhelmingly common use
>> case for raw buffers is to view them as a sequence of UInt8 *without*
>> binding the type.  Generally, at the point that you're dealing with a
>> raw buffer it's impossible to (re)bind memory because you don't know
>> what type it holds. 
> 
> Oh, you can't just rebind to UInt8 because that's not defined as
> universally compatible with all data.  OK, sorry.
> 
>> The reason it's so important to have an UnsafeRawBufferPointer data
>> type is precisely so that users don't need mess about with binding
>> memory. It's easy to get that wrong even when it's possible.
>> 
>> The only reason that UInt8 is special is that when users create
>> temporary typed buffers for bytes (e.g. they sometimes want a growable
>> array or just don't want to bother with manual allocation) they always
>> use UInt8 as the element type.
>> 
>> That said, we could easily divide these concerns into two types as
>> you suggested. A raw buffer, which doesn't have any special UInt8
>> features, and a RawBytes collection that handles both buffer slicing
>> and UInt8 interoperability.
> 
> But, now that I think of it, that wouldn't really solve any problems,
> would it?


A new Collection type doesn't solve any practical problems. It does solve a
conceptual problem if you think that a raw buffer is not *inherently* a
collection of bytes. There is an elegance in separating the raw buffer
semantics from the byte collection semantics, but that elegance does
not simplify anything for users--it's just more abstraction to figure
out. Certainly, the most straightforward way to fix this is to simply
change raw buffer's slice type, so I'm inclined to favor that
approach. Creating a new collection type would involve
rethinking/redesigning some of the related APIs.

Also note that I'm leaning toward slice -> buffer conversion via an
unlabeled initializer because I think it's the most obvious with least
API surface.

I don't think we absolutely need a new proposal for this easy fix,
since it's not really introducing a new API. The additional
initializer merely allows code that used to work to be migrated via a
fixit.

Here's a quick summary. If there aren't any strong objections, I'll
post an ammendment to the original proposal along with a PR for more
formal review.

Proposed ammendment to SE-0138:


Fix: Change Unsafe${Mutable}RawBufferPointer's SubSequnce type

Original: Unsafe${Mutable}RawBufferPointer.SubSequence = 
Unsafe${Mutable}RawBufferPointer

Fixed: 

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Dave Abrahams via swift-evolution]

on Fri Dec 09 2016, Andrew Trick  wrote:

>> On Dec 9, 2016, at 10:27 AM, Dave Abrahams via swift-evolution
>  wrote:
>> 
>> 
>> on Thu Dec 08 2016, Xiaodi Wu > > wrote:
>> 
>
>>> On Thu, Dec 8, 2016 at 6:53 PM, Ben Cohen via swift-evolution <
>>> swift-evolution@swift.org> wrote:
>>> 
 
 On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution <
 swift-evolution@swift.org> wrote:
 
 Um, Sequence doesn’t have a subscript (or indexes). Sequences are
 single-pass. So if this is important, it needs to stay a Collection.
 
 
 Just because something fulfills one of the requirements of a Collection
 does not mean it should be one. It needs to tick all the boxes before its
 allowed to be elevated.
 
 But it’s still allowed to have subscripts (UnsafePointer has subscripting
 but isn’t a collection) or be multi-pass (strides are multiples but are
 only sequences). That’s OK
 
 In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t
 a collection because it doesn’t meet the requirements for slicing i.e. that
 indices of the slice be indices of the parent.
 (relatedly… it appears this requirement is documented on the concrete
 Slice type rather than on Collection… which is a documentation bug we
 should fix).
 
>>> 
>>> If this is indeed a requirement for Collection, then my vote would be for
>>> Nate's option #1 and Andy's option #2, to give UnsafeRawBufferPointer a
>>> Slice type that fulfills the requirement. It's the smallest change,
>>> preserves the use of integer indices, and preserves what Andy stated as the
>>> desired use case of making it easy for users to switch out code written for
>>> [UInt8].
>>> 
>>> I'm not sure I fully understand yet why Dave finds the idea of Collection
>>> conformance fishy, 
>> 
>> Because the memory can easily be already bound to another type than
>> UInt8, and there's no obvious reason why UInt8 should be privileged as a
>> type you can get out of a raw buffer without binding the memory.
>
> I strongly disagree with that statement. The overwhelmingly common use
> case for raw buffers is to view them as a sequence of UInt8 *without*
> binding the type.  Generally, at the point that you're dealing with a
> raw buffer it's impossible to (re)bind memory because you don't know
> what type it holds. 

Oh, you can't just rebind to UInt8 because that's not defined as
universally compatible with all data.  OK, sorry.

> The reason it's so important to have an UnsafeRawBufferPointer data
> type is precisely so that users don't need mess about with binding
> memory. It's easy to get that wrong even when it's possible.
>
> The only reason that UInt8 is special is that when users create
> temporary typed buffers for bytes (e.g. they sometimes want a growable
> array or just don't want to bother with manual allocation) they always
> use UInt8 as the element type.
>
> That said, we could easily divide these concerns into two types as
> you suggested. A raw buffer, which doesn't have any special UInt8
> features, and a RawBytes collection that handles both buffer slicing
> and UInt8 interoperability.

But, now that I think of it, that wouldn't really solve any problems,
would it?

-- 
-Dave

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Dave Abrahams via swift-evolution]

on Thu Dec 08 2016, Ben Cohen  wrote:

> Probably needs an argument label since it's performing an explicit purpose, 
> not just a vanilla
> conversion initializer. So maybe
> UnsafeRawBufferPointer.init(rebasing:)

I disagree that we need a label here.  It's a value-preserving
conversion; the result even means the same thing as the argument.

>
>
> Or since we have same-type constrained extensions now on master maybe you 
> could do it as a property:
>
> extension RandomAccessSlice where Base == UnsafeRawBufferPointer {
> var rebased: UnsafeRawBufferPointer {
> return UnsafeRawBufferPointer(start: base.baseAddress, count: count)
> }
> }
>
> (written by hand without a compiler so unlikely to be correct :)
>
>> On Dec 8, 2016, at 18:07, Andrew Trick  wrote:
>> 
>> 
>>> On Dec 8, 2016, at 5:44 PM, Xiaodi Wu via swift-evolution
>  wrote:
>>> 
 On Thu, Dec 8, 2016 at 6:53 PM, Ben Cohen via swift-evolution 
  wrote:
 
> On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution 
>  wrote:
> 
> Um, Sequence doesn’t have a subscript (or indexes). Sequences are 
> single-pass. So if this is important, it needs to stay a Collection.
 
 Just because something fulfills one of the requirements of a Collection 
 does not mean it should be one. It needs to tick all the boxes before its 
 allowed to be elevated.
 
 But it’s still allowed to have subscripts (UnsafePointer has subscripting 
 but isn’t a collection) or be multi-pass (strides are multiples but are 
 only sequences). That’s OK
 
 In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t 
 a collection because it doesn’t meet the requirements for slicing i.e. 
 that indices of the slice be indices of the parent.
 (relatedly… it appears this requirement is documented on the concrete 
 Slice type rather than on Collection… which is a documentation bug we 
 should fix).
>>> 
>>> If this is indeed a requirement for Collection, then my vote would be for 
>>> Nate's option #1 and Andy's option #2, to give UnsafeRawBufferPointer a 
>>> Slice type that fulfills the requirement. It's the smallest change, 
>>> preserves the use of integer indices, and preserves what Andy stated as the 
>>> desired use case of making it easy for users to switch out code written for 
>>> [UInt8].
>> 
>> Ok, but there needs to be an easy way in a nongeneric context to convert 
>> from a Slice into an URBP (with normalized byte offsets).
>> 
>> Does anyone object to adding an initializer for this? Any suggestions on 
>> naming? Do we need an
> argument label? etc?
>> 
>> UnsafeRawBufferPointer(_ : Slice)
>> 
>> as in:
>> 
>> let region = UnsafeRawBufferPointer(buffer[i..> 
>> -Andy
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>

-- 
-Dave

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Andrew Trick via swift-evolution]

> On Dec 9, 2016, at 10:27 AM, Dave Abrahams via swift-evolution 
>  wrote:
> 
> 
> on Thu Dec 08 2016, Xiaodi Wu  > wrote:
> 
>> On Thu, Dec 8, 2016 at 6:53 PM, Ben Cohen via swift-evolution <
>> swift-evolution@swift.org> wrote:
>> 
>>> 
>>> On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution <
>>> swift-evolution@swift.org> wrote:
>>> 
>>> Um, Sequence doesn’t have a subscript (or indexes). Sequences are
>>> single-pass. So if this is important, it needs to stay a Collection.
>>> 
>>> 
>>> Just because something fulfills one of the requirements of a Collection
>>> does not mean it should be one. It needs to tick all the boxes before its
>>> allowed to be elevated.
>>> 
>>> But it’s still allowed to have subscripts (UnsafePointer has subscripting
>>> but isn’t a collection) or be multi-pass (strides are multiples but are
>>> only sequences). That’s OK
>>> 
>>> In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t
>>> a collection because it doesn’t meet the requirements for slicing i.e. that
>>> indices of the slice be indices of the parent.
>>> (relatedly… it appears this requirement is documented on the concrete
>>> Slice type rather than on Collection… which is a documentation bug we
>>> should fix).
>>> 
>> 
>> If this is indeed a requirement for Collection, then my vote would be for
>> Nate's option #1 and Andy's option #2, to give UnsafeRawBufferPointer a
>> Slice type that fulfills the requirement. It's the smallest change,
>> preserves the use of integer indices, and preserves what Andy stated as the
>> desired use case of making it easy for users to switch out code written for
>> [UInt8].
>> 
>> I'm not sure I fully understand yet why Dave finds the idea of Collection
>> conformance fishy, 
> 
> Because the memory can easily be already bound to another type than
> UInt8, and there's no obvious reason why UInt8 should be privileged as a
> type you can get out of a raw buffer without binding the memory.

I strongly disagree with that statement. The overwhelmingly common use
case for raw buffers is to view them as a sequence of UInt8 *without*
binding the type.  Generally, at the point that you're dealing with a
raw buffer it's impossible to (re)bind memory because you don't know
what type it holds. The reason it's so important to have an
UnsafeRawBufferPointer data type is precisely so that users don't need
mess about with binding memory. It's easy to get that wrong even when
it's possible.

The only reason that UInt8 is special is that when users create
temporary typed buffers for bytes (e.g. they sometimes want a growable
array or just don't want to bother with manual allocation) they always
use UInt8 as the element type.

That said, we could easily divide these concerns into two types as
you suggested. A raw buffer, which doesn't have any special UInt8
features, and a RawBytes collection that handles both buffer slicing
and UInt8 interoperability.

-Andy

> 
>> but I'm comfortable with a type that's clearly labeled as unsafe not
>> being fully footgun-proof.
>> 
>>> 
>>> ___
>>> swift-evolution mailing list
>>> swift-evolution@swift.org 
>>> https://lists.swift.org/mailman/listinfo/swift-evolution 
>>> 
>>> 
>>> 
> 
> -- 
> -Dave
> ___
> swift-evolution mailing list
> swift-evolution@swift.org 
> https://lists.swift.org/mailman/listinfo/swift-evolution 
> 
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Andrew Trick via swift-evolution]

> On Dec 9, 2016, at 10:22 AM, Dave Abrahams  wrote:
> 
> 
> on Thu Dec 08 2016, Jordan Rose  wrote:
> 
>>> On Dec 8, 2016, at 16:22, Andrew Trick via swift-evolution 
>>>  wrote:
>>> In practice, it needs to be able to interoperate with [UInt8] and be 
>>> interchangeable in the same
>> generic context.
>>> e.g. `byteBuffer += rawBuffer[payloadIndex..>> I think Sequence is sufficient for that purpose.
>> 
>> Um, Sequence doesn’t have a subscript (or indexes). Sequences are 
>> single-pass. So if this is
>> important, it needs to stay a Collection.
> 
> Yes.  But I don't see why the raw buffer should be a collection in the first 
> place.  Why not
> 
> byteBuffer += rawBuffer.bytes[payloadIndex.. 
> ?
> 
> -- 
> -Dave

`RawBytes : Collection` could be a view over the raw buffer with `SubSequence : 
RandomAccessSlice` if that's what you mean.

I hadn’t considered that just because it’s yet another type that needs to be 
introduced. Now interface authors need to decide whether they want to take a 
collection of bytes or a buffer. (In a non-generic context, the correct answer 
is to pass the buffer, not the collection--the collection would be just a 
convenient temporary view).

We still need an initializer or extension to handle the expected use case of 
nested buffers:

extension RandomAccessSlice where Base == RawBytes {
var rebased: UnsafeRawBufferPointer {
return UnsafeRawBufferPointer(start: base.baseAddress, count: count)
}
}

I would even be willing to eliminate raw buffer subscripting altogether, which 
I think is what you're getting at, since this isn't too awful:

`buffer.bytes[i]`

Pro: More explicit division between raw memory semantics and Collection 
semantics.

Con: Raw buffer users need to juggle two different types and know how convert 
between them.

-Andy
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Dave Abrahams via swift-evolution]

on Thu Dec 08 2016, Xiaodi Wu  wrote:


On Thu, Dec 8, 2016 at 6:53 PM, Ben Cohen via swift-evolution <
swift-evolution@swift.org> wrote:



On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution <
swift-evolution@swift.org> wrote:

Um, Sequence doesn’t have a subscript (or indexes). Sequences 
are
single-pass. So if this is important, it needs to stay a 
Collection.



Just because something fulfills one of the requirements of a 
Collection
does not mean it should be one. It needs to tick all the boxes 
before its

allowed to be elevated.

But it’s still allowed to have subscripts (UnsafePointer has 
subscripting
but isn’t a collection) or be multi-pass (strides are multiples 
but are

only sequences). That’s OK

In this case, yes it’s multi-pass, yes it has a subscript, but 
no it isn’t
a collection because it doesn’t meet the requirements for 
slicing i.e. that

indices of the slice be indices of the parent.
(relatedly… it appears this requirement is documented on the 
concrete
Slice type rather than on Collection… which is a documentation 
bug we

should fix).



If this is indeed a requirement for Collection, then my vote 
would be for
Nate's option #1 and Andy's option #2, to give 
UnsafeRawBufferPointer a
Slice type that fulfills the requirement. It's the smallest 
change,
preserves the use of integer indices, and preserves what Andy 
stated as the
desired use case of making it easy for users to switch out code 
written for

[UInt8].

I'm not sure I fully understand yet why Dave finds the idea of 
Collection
conformance fishy, 


Because the memory can easily be already bound to another type 
than
UInt8, and there's no obvious reason why UInt8 should be 
privileged as a

type you can get out of a raw buffer without binding the memory.

but I'm comfortable with a type that's clearly labeled as unsafe 
not

being fully footgun-proof.



___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution




--
-Dave
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Dave Abrahams via swift-evolution]

on Thu Dec 08 2016, Ben Cohen  wrote:

On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution 
 wrote:


Um, Sequence doesn’t have a subscript (or indexes). Sequences 
are single-pass. So if this is

important, it needs to stay a Collection.




Just because something fulfills one of the requirements of a 
Collection does not mean it should be
one. 


Um, it sorta does.

It needs to tick all the boxes before its allowed to be 
elevated.


But it’s still allowed to have subscripts (UnsafePointer has 
subscripting but isn’t a collection) or
be multi-pass (strides are multiples but are only 
sequences). That’s OK


A sequence that is multipass can always be a collection, and 
there's no

good reason to limit its expressivity by keeping it a sequence.

In this case, yes it’s multi-pass, yes it has a subscript, but 
no it

isn’t a collection because it doesn’t meet the requirements for
slicing i.e. that indices of the slice be indices of the parent.


But it could.  It's just a bug that we failed to get that right.



(relatedly… it appears this requirement is documented on the 
concrete
Slice type rather than on Collection… which is a documentation 
bug we

should fix).


--
-Dave
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Dave Abrahams via swift-evolution]

on Thu Dec 08 2016, Jordan Rose  wrote:

On Dec 8, 2016, at 16:22, Andrew Trick via swift-evolution 
 wrote:


In practice, it needs to be able to interoperate with [UInt8] 
and be interchangeable in the same

generic context.


e.g. `byteBuffer += rawBuffer[payloadIndex..

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Ben Cohen via swift-evolution]

Probably needs an argument label since it's performing an explicit purpose, not 
just a vanilla conversion initializer. So maybe 
UnsafeRawBufferPointer.init(rebasing:)

Or since we have same-type constrained extensions now on master maybe you could 
do it as a property: 

extension RandomAccessSlice where Base == UnsafeRawBufferPointer {
var rebased: UnsafeRawBufferPointer {
return UnsafeRawBufferPointer(start: base.baseAddress, count: count)
}
}

(written by hand without a compiler so unlikely to be correct :)

> On Dec 8, 2016, at 18:07, Andrew Trick  wrote:
> 
> 
>> On Dec 8, 2016, at 5:44 PM, Xiaodi Wu via swift-evolution 
>>  wrote:
>> 
>>> On Thu, Dec 8, 2016 at 6:53 PM, Ben Cohen via swift-evolution 
>>>  wrote:
>>> 
 On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution 
  wrote:
 
 Um, Sequence doesn’t have a subscript (or indexes). Sequences are 
 single-pass. So if this is important, it needs to stay a Collection.
>>> 
>>> Just because something fulfills one of the requirements of a Collection 
>>> does not mean it should be one. It needs to tick all the boxes before its 
>>> allowed to be elevated.
>>> 
>>> But it’s still allowed to have subscripts (UnsafePointer has subscripting 
>>> but isn’t a collection) or be multi-pass (strides are multiples but are 
>>> only sequences). That’s OK
>>> 
>>> In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t 
>>> a collection because it doesn’t meet the requirements for slicing i.e. that 
>>> indices of the slice be indices of the parent.
>>> (relatedly… it appears this requirement is documented on the concrete Slice 
>>> type rather than on Collection… which is a documentation bug we should fix).
>> 
>> If this is indeed a requirement for Collection, then my vote would be for 
>> Nate's option #1 and Andy's option #2, to give UnsafeRawBufferPointer a 
>> Slice type that fulfills the requirement. It's the smallest change, 
>> preserves the use of integer indices, and preserves what Andy stated as the 
>> desired use case of making it easy for users to switch out code written for 
>> [UInt8].
> 
> Ok, but there needs to be an easy way in a nongeneric context to convert from 
> a Slice into an URBP (with normalized byte offsets).
> 
> Does anyone object to adding an initializer for this? Any suggestions on 
> naming? Do we need an argument label? etc?
> 
> UnsafeRawBufferPointer(_ : Slice)
> 
> as in:
> 
> let region = UnsafeRawBufferPointer(buffer[i.. 
> -Andy
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Ben Cohen via swift-evolution]

So now that I look at it, it appears UnsafeRawBufferPointer(copyBytes:) has the 
same problems we are trying to solve on initialize(as:from) i.e. it is 
completely at the mercy of the passed-in collection's count being accurate, and 
if it isn't it'll scribble over memory. We should probably apply similar fixes 
and change it to take a sequence.

I realize it's a sticking plaster on this particular issue though, so still 
doesn't answer whether it's better for UnsafeRawBufferPointer to be a 
collection, just created more work...

> On Dec 8, 2016, at 17:06, Andrew Trick  wrote:
> 
> 
>>> On Dec 8, 2016, at 4:54 PM, Jordan Rose  wrote:
>>> 
>>> 
 On Dec 8, 2016, at 16:53, Ben Cohen  wrote:
 
 
 On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution 
  wrote:
 
 Um, Sequence doesn’t have a subscript (or indexes). Sequences are 
 single-pass. So if this is important, it needs to stay a Collection.
 
>>> 
>>> Just because something fulfills one of the requirements of a Collection 
>>> does not mean it should be one. It needs to tick all the boxes before its 
>>> allowed to be elevated.
>>> 
>>> But it’s still allowed to have subscripts (UnsafePointer has subscripting 
>>> but isn’t a collection) or be multi-pass (strides are multiples but are 
>>> only sequences). That’s OK
>>> 
>>> In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t 
>>> a collection because it doesn’t meet the requirements for slicing i.e. that 
>>> indices of the slice be indices of the parent.
>>> (relatedly… it appears this requirement is documented on the concrete Slice 
>>> type rather than on Collection… which is a documentation bug we should fix).
>> 
>> Ah, right, thank you. Retracted.
> 
> Let me restate, because I think Jordan's question was valid given my 
> statement.
> 
> It would be *nice* for raw buffers to be Collection because they’re 
> meant to be a replacement for code that is typically written for [UInt8], and 
> anything you can do with an array that applies to raw buffers is covered by 
> Collection.
> 
> However, I don’t expect the raw buffer to be used in a generic context except 
> being passed to utilities that copy the bytes out. That will either be done 
> by directly iterating over the collection or invoking some other API that 
> could take a Sequence. The most important is probably 
> Array.append(contentsOf:), which is moving over to Sequence. However, we 
> would also need to change UnsafeRawBufferPointer(copyBytes:), 
> NSData(replaceSubrange:), and whatever else I haven't thought of. That's a 
> small disadvantage to this solution.
> 
> I'm also a little concerned that Sequence is immutable, so generic code has 
> no way to copy bytes into the buffer.
> 
> My bigger concern is still that the range subscript’s inconsistent behavior 
> may still lead to bugs in practice in nongeneric code.
> 
> -Andy
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Andrew Trick via swift-evolution]

> On Dec 8, 2016, at 5:44 PM, Xiaodi Wu via swift-evolution 
>  wrote:
> 
> On Thu, Dec 8, 2016 at 6:53 PM, Ben Cohen via swift-evolution 
> > wrote:
> 
>> On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution 
>> > wrote:
>> 
>> Um, Sequence doesn’t have a subscript (or indexes). Sequences are 
>> single-pass. So if this is important, it needs to stay a Collection.
>> 
> 
> Just because something fulfills one of the requirements of a Collection does 
> not mean it should be one. It needs to tick all the boxes before its allowed 
> to be elevated.
> 
> But it’s still allowed to have subscripts (UnsafePointer has subscripting but 
> isn’t a collection) or be multi-pass (strides are multiples but are only 
> sequences). That’s OK
> 
> In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t a 
> collection because it doesn’t meet the requirements for slicing i.e. that 
> indices of the slice be indices of the parent.
> (relatedly… it appears this requirement is documented on the concrete Slice 
> type rather than on Collection… which is a documentation bug we should fix).
> 
> If this is indeed a requirement for Collection, then my vote would be for 
> Nate's option #1 and Andy's option #2, to give UnsafeRawBufferPointer a Slice 
> type that fulfills the requirement. It's the smallest change, preserves the 
> use of integer indices, and preserves what Andy stated as the desired use 
> case of making it easy for users to switch out code written for [UInt8].

Ok, but there needs to be an easy way in a nongeneric context to convert from a 
Slice into an URBP (with normalized byte offsets).

Does anyone object to adding an initializer for this? Any suggestions on 
naming? Do we need an argument label? etc?

UnsafeRawBufferPointer(_ : Slice)

as in:

let region = UnsafeRawBufferPointer(buffer[i..

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Xiaodi Wu via swift-evolution]

On Thu, Dec 8, 2016 at 6:53 PM, Ben Cohen via swift-evolution <
swift-evolution@swift.org> wrote:

>
> On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution <
> swift-evolution@swift.org> wrote:
>
> Um, Sequence doesn’t have a subscript (or indexes). Sequences are
> single-pass. So if this is important, it needs to stay a Collection.
>
>
> Just because something fulfills one of the requirements of a Collection
> does not mean it should be one. It needs to tick all the boxes before its
> allowed to be elevated.
>
> But it’s still allowed to have subscripts (UnsafePointer has subscripting
> but isn’t a collection) or be multi-pass (strides are multiples but are
> only sequences). That’s OK
>
> In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t
> a collection because it doesn’t meet the requirements for slicing i.e. that
> indices of the slice be indices of the parent.
> (relatedly… it appears this requirement is documented on the concrete
> Slice type rather than on Collection… which is a documentation bug we
> should fix).
>

If this is indeed a requirement for Collection, then my vote would be for
Nate's option #1 and Andy's option #2, to give UnsafeRawBufferPointer a
Slice type that fulfills the requirement. It's the smallest change,
preserves the use of integer indices, and preserves what Andy stated as the
desired use case of making it easy for users to switch out code written for
[UInt8].

I'm not sure I fully understand yet why Dave finds the idea of Collection
conformance fishy, but I'm comfortable with a type that's clearly labeled
as unsafe not being fully footgun-proof.


>
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
>
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Xiaodi Wu via swift-evolution]

Sorry, let me clarify. Can you, Dave, elaborate as to why the issue you
stated as "this" ("untyped memory without bounds checks" conforming to
Collection) seems, as you say, "suspect"?

On Thu, Dec 8, 2016 at 18:32 Dave Abrahams  wrote:


on Thu Dec 08 2016, Xiaodi Wu  wrote:

> Can you elaborate on this? Why aren't you sure this is a wise
> idea?

Whom are you asking?  What is “this?”

--
-Dave
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Andrew Trick via swift-evolution]

> On Dec 8, 2016, at 4:54 PM, Jordan Rose  wrote:
> 
> 
>> On Dec 8, 2016, at 16:53, Ben Cohen > > wrote:
>> 
>> 
>>> On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution 
>>> > wrote:
>>> 
>>> Um, Sequence doesn’t have a subscript (or indexes). Sequences are 
>>> single-pass. So if this is important, it needs to stay a Collection.
>>> 
>> 
>> Just because something fulfills one of the requirements of a Collection does 
>> not mean it should be one. It needs to tick all the boxes before its allowed 
>> to be elevated.
>> 
>> But it’s still allowed to have subscripts (UnsafePointer has subscripting 
>> but isn’t a collection) or be multi-pass (strides are multiples but are only 
>> sequences). That’s OK
>> 
>> In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t a 
>> collection because it doesn’t meet the requirements for slicing i.e. that 
>> indices of the slice be indices of the parent.
>> (relatedly… it appears this requirement is documented on the concrete Slice 
>> type rather than on Collection… which is a documentation bug we should fix).
> 
> Ah, right, thank you. Retracted.

Let me restate, because I think Jordan's question was valid given my statement.

It would be *nice* for raw buffers to be Collection because they’re 
meant to be a replacement for code that is typically written for [UInt8], and 
anything you can do with an array that applies to raw buffers is covered by 
Collection.

However, I don’t expect the raw buffer to be used in a generic context except 
being passed to utilities that copy the bytes out. That will either be done by 
directly iterating over the collection or invoking some other API that could 
take a Sequence. The most important is probably Array.append(contentsOf:), 
which is moving over to Sequence. However, we would also need to change 
UnsafeRawBufferPointer(copyBytes:), NSData(replaceSubrange:), and whatever else 
I haven't thought of. That's a small disadvantage to this solution.

I'm also a little concerned that Sequence is immutable, so generic code has no 
way to copy bytes into the buffer.

My bigger concern is still that the range subscript’s inconsistent behavior may 
still lead to bugs in practice in nongeneric code.

-Andy___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Jordan Rose via swift-evolution]

> On Dec 8, 2016, at 16:53, Ben Cohen  wrote:
> 
> 
>> On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution 
>> > wrote:
>> 
>> Um, Sequence doesn’t have a subscript (or indexes). Sequences are 
>> single-pass. So if this is important, it needs to stay a Collection.
>> 
> 
> Just because something fulfills one of the requirements of a Collection does 
> not mean it should be one. It needs to tick all the boxes before its allowed 
> to be elevated.
> 
> But it’s still allowed to have subscripts (UnsafePointer has subscripting but 
> isn’t a collection) or be multi-pass (strides are multiples but are only 
> sequences). That’s OK
> 
> In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t a 
> collection because it doesn’t meet the requirements for slicing i.e. that 
> indices of the slice be indices of the parent.
> (relatedly… it appears this requirement is documented on the concrete Slice 
> type rather than on Collection… which is a documentation bug we should fix).

Ah, right, thank you. Retracted.___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Ben Cohen via swift-evolution]

> On Dec 8, 2016, at 4:35 PM, Jordan Rose via swift-evolution 
>  wrote:
> 
> Um, Sequence doesn’t have a subscript (or indexes). Sequences are 
> single-pass. So if this is important, it needs to stay a Collection.
> 

Just because something fulfills one of the requirements of a Collection does 
not mean it should be one. It needs to tick all the boxes before its allowed to 
be elevated.

But it’s still allowed to have subscripts (UnsafePointer has subscripting but 
isn’t a collection) or be multi-pass (strides are multiples but are only 
sequences). That’s OK

In this case, yes it’s multi-pass, yes it has a subscript, but no it isn’t a 
collection because it doesn’t meet the requirements for slicing i.e. that 
indices of the slice be indices of the parent.
(relatedly… it appears this requirement is documented on the concrete Slice 
type rather than on Collection… which is a documentation bug we should fix).___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Jordan Rose via swift-evolution]

> On Dec 8, 2016, at 16:22, Andrew Trick via swift-evolution 
>  wrote:
> 
> In practice, it needs to be able to interoperate with [UInt8] and be 
> interchangeable in the same generic context.
> 
> e.g. `byteBuffer += rawBuffer[payloadIndex.. 
> I think Sequence is sufficient for that purpose.

Um, Sequence doesn’t have a subscript (or indexes). Sequences are single-pass. 
So if this is important, it needs to stay a Collection.

Jordan___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Dave Abrahams via swift-evolution]

on Thu Dec 08 2016, Xiaodi Wu  wrote:

Can you elaborate on this? Why aren't you sure this is a wise 
idea?


Whom are you asking?  What is “this?”

--
-Dave
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Andrew Trick via swift-evolution]

> On Dec 8, 2016, at 12:50 PM, Dave Abrahams via swift-evolution 
>  wrote:
> 
> 
> on Thu Dec 08 2016, Ben Cohen  wrote:
> 
>>> On Dec 2, 2016, at 8:27 PM, Nate Cook  wrote:
>>> 
 On Dec 2, 2016, at 2:12 PM, Ben Cohen via swift-evolution
 >
 wrote:
>> 
 
> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution
> >
> wrote:
> 
> 3) Make all buffer pointers their own slices but use a different
> index type. If the indices were just wrapped pointers, that would
> handle the index sharing without needing an additional property on
> the buffer. We could also maintain integer-based stridable
> conformance (which greatly simplifies index arithmetic), since the
> indices would just offset by a byte for raw buffers or a stride
> for typed buffers.
> 
 
 Unfortunately, switching to non-integer indices would change this
 from being mildly source-breaking to being extremely
 source-breaking, as there’s lots of code out there using buffers
 today indexing them with integers (including integer literals).
 
 The big win with UnsafeBufferPointer having an integer index is
 it’s a drop-in replacement for arrays, so when you hit a
 performance problem using an array you can quickly switch to using
 a buffer under most circumstances instead without having to change
 much of your code – including code that uses for i in
 0..>> 
>>> It is definitely very source-breaking, though with relatively simple fixits:
>>> 
>>> buf[0] ---> buf[buf.startIndex]
>>> buf[3] ---> buf[buf.startIndex + 3]
>>> buf[i] ---> buf[buf.startIndex + i]
>>> 
>>> Any integer arithmetic happening outside the subscript could be left
>>> unchanged. If that cost isn't worth the benefit, then making
>>> UnsafeRawBufferPointer use Slice as its slice type is probably the
>>> best way to resolve that issue.
>>> 
>>> Nate
>> 
>> The fixits aren’t quite that simple for slices, though:
>> 
>>  let slice = buf[3..<6]
>>  slice[3] —> slice[slice.startIndex + 0] // fixit would somehow need to 
>> know this is 0 not 3
>>  slice[i] —> slice[slice.startIndex + ??] // or even need to
>> know this is, erm, I haven’t had enough coffee this morning
>> 
>> The other downside is it would thwart speculatively switching an Array
>> to an UnsafeBuffer to see if that was a bottleneck, then switching
>> back.
>> 
>>> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution 
>>>  wrote:
>>> 
>>> 1) Switch to using Slice as a wrapper for UnsafeRawBufferPointer.
>>> 
>> 
>> Based on the above, it seems like this is the least bad option, and we
>> need to do this ASAP as currently UnsafeRawBufferPointer is
>> non-compliant with the requirements of slicing and needs changing
>> before it’s more widely adopted.
> 
> Or we could say that UnsafeRawBufferPointer isn't a Collection.  Making
> it a Collection in the first place has always seemed suspect to me.
> 
> -- 
> -Dave

UnsafeRawBufferPointer does not need to be a Collection, but should at least be 
a Sequence. It is a Collection now simply because it fits the criteria 
(nondestructively accessed and subscriptable).

In practice, it needs to be able to interoperate with [UInt8] and be 
interchangeable in the same generic context.

e.g. `byteBuffer += rawBuffer[payloadIndex.. out-of-bounds

To some extent, this ship has sailed. I can see a few of options now:

1. We make UnsafeRawBufferPointer a Sequence and just live with that 
inconsistency. Users need discern the importance of range subscript's 

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Xiaodi Wu via swift-evolution]

Can you elaborate on this? Why aren't you sure this is a wise idea?

On Thu, Dec 8, 2016 at 16:19 Dave Abrahams via swift-evolution <
swift-evolution@swift.org> wrote:

>
> on Thu Dec 08 2016, Alexis Beingessner  wrote:
>
> >> On Dec 8, 2016, at 3:50 PM, Dave Abrahams via swift-evolution <
> swift-evolution@swift.org> wrote:
> >>
> >>
> >> on Thu Dec 08 2016, Ben Cohen  wrote:
> >>
> >
>  On Dec 2, 2016, at 8:27 PM, Nate Cook  wrote:
> 
> > On Dec 2, 2016, at 2:12 PM, Ben Cohen via swift-evolution
> > >
> > wrote:
> >>>
> >
> >> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution
> >> >
> >> wrote:
> >>
> >> 3) Make all buffer pointers their own slices but use a different
> >> index type. If the indices were just wrapped pointers, that would
> >> handle the index sharing without needing an additional property on
> >> the buffer. We could also maintain integer-based stridable
> >> conformance (which greatly simplifies index arithmetic), since the
> >> indices would just offset by a byte for raw buffers or a stride
> >> for typed buffers.
> >>
> >
> > Unfortunately, switching to non-integer indices would change this
> > from being mildly source-breaking to being extremely
> > source-breaking, as there’s lots of code out there using buffers
> > today indexing them with integers (including integer literals).
> >
> > The big win with UnsafeBufferPointer having an integer index is
> > it’s a drop-in replacement for arrays, so when you hit a
> > performance problem using an array you can quickly switch to using
> > a buffer under most circumstances instead without having to change
> > much of your code – including code that uses for i in
> > 0.. > wild. Switching to an opaque index would break anyone doing that.
> 
>  It is definitely very source-breaking, though with relatively simple
> fixits:
> 
> buf[0] ---> buf[buf.startIndex]
> buf[3] ---> buf[buf.startIndex + 3]
> buf[i] ---> buf[buf.startIndex + i]
> 
>  Any integer arithmetic happening outside the subscript could be left
>  unchanged. If that cost isn't worth the benefit, then making
>  UnsafeRawBufferPointer use Slice as its slice type is probably the
>  best way to resolve that issue.
> 
>  Nate
> >>>
> >>> The fixits aren’t quite that simple for slices, though:
> >>>
> >>>let slice = buf[3..<6]
> >>>slice[3] —> slice[slice.startIndex + 0] // fixit would somehow need
> to know this is 0 not 3
> >>>slice[i] —> slice[slice.startIndex + ??] // or even need to
> >>> know this is, erm, I haven’t had enough coffee this morning
> >>>
> >>> The other downside is it would thwart speculatively switching an Array
> >>> to an UnsafeBuffer to see if that was a bottleneck, then switching
> >>> back.
> >>>
>  On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution <
> swift-evolution@swift.org> wrote:
> 
>  1) Switch to using Slice as a wrapper for UnsafeRawBufferPointer.
> 
> >>>
> >>> Based on the above, it seems like this is the least bad option, and we
> >>> need to do this ASAP as currently UnsafeRawBufferPointer is
> >>> non-compliant with the requirements of slicing and needs changing
> >>> before it’s more widely adopted.
> >>
> >> Or we could say that UnsafeRawBufferPointer isn't a Collection.  Making
> >> it a Collection in the first place has always seemed suspect to me.
> >>
> >
> > If this is considered a viable option, it's the one I want. Passing
> > types without bounds checks into generic "safe" code shouldn't be this
> > easy.
>
> I don't think I agree, but that's a separate argument, about
> UnsafeBufferPointer.  This is about passing *untyped* memory without
> bounds checks.
>
> > You should need to explicitly wrap it up in something safe. And I
> > really don't want the known-to-be-error-prone indexing model in
> > concrete unsafe code.
>
>
> --
> -Dave
>
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Dave Abrahams via swift-evolution]

on Thu Dec 08 2016, Alexis Beingessner  wrote:

>> On Dec 8, 2016, at 3:50 PM, Dave Abrahams via swift-evolution 
>>  wrote:
>> 
>> 
>> on Thu Dec 08 2016, Ben Cohen  wrote:
>> 
>
 On Dec 2, 2016, at 8:27 PM, Nate Cook  wrote:
 
> On Dec 2, 2016, at 2:12 PM, Ben Cohen via swift-evolution
> >
> wrote:
>>> 
> 
>> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution
>> >
>> wrote:
>> 
>> 3) Make all buffer pointers their own slices but use a different
>> index type. If the indices were just wrapped pointers, that would
>> handle the index sharing without needing an additional property on
>> the buffer. We could also maintain integer-based stridable
>> conformance (which greatly simplifies index arithmetic), since the
>> indices would just offset by a byte for raw buffers or a stride
>> for typed buffers.
>> 
> 
> Unfortunately, switching to non-integer indices would change this
> from being mildly source-breaking to being extremely
> source-breaking, as there’s lots of code out there using buffers
> today indexing them with integers (including integer literals).
> 
> The big win with UnsafeBufferPointer having an integer index is
> it’s a drop-in replacement for arrays, so when you hit a
> performance problem using an array you can quickly switch to using
> a buffer under most circumstances instead without having to change
> much of your code – including code that uses for i in
> 0.. wild. Switching to an opaque index would break anyone doing that.
 
 It is definitely very source-breaking, though with relatively simple 
 fixits:
 
buf[0] ---> buf[buf.startIndex]
buf[3] ---> buf[buf.startIndex + 3]
buf[i] ---> buf[buf.startIndex + i]
 
 Any integer arithmetic happening outside the subscript could be left
 unchanged. If that cost isn't worth the benefit, then making
 UnsafeRawBufferPointer use Slice as its slice type is probably the
 best way to resolve that issue.
 
 Nate
>>> 
>>> The fixits aren’t quite that simple for slices, though:
>>> 
>>>let slice = buf[3..<6]
>>>slice[3] —> slice[slice.startIndex + 0] // fixit would somehow need to 
>>> know this is 0 not 3
>>>slice[i] —> slice[slice.startIndex + ??] // or even need to
>>> know this is, erm, I haven’t had enough coffee this morning
>>> 
>>> The other downside is it would thwart speculatively switching an Array
>>> to an UnsafeBuffer to see if that was a bottleneck, then switching
>>> back.
>>> 
 On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution 
  wrote:
 
 1) Switch to using Slice as a wrapper for UnsafeRawBufferPointer.
 
>>> 
>>> Based on the above, it seems like this is the least bad option, and we
>>> need to do this ASAP as currently UnsafeRawBufferPointer is
>>> non-compliant with the requirements of slicing and needs changing
>>> before it’s more widely adopted.
>> 
>> Or we could say that UnsafeRawBufferPointer isn't a Collection.  Making
>> it a Collection in the first place has always seemed suspect to me.
>> 
>
> If this is considered a viable option, it's the one I want. Passing
> types without bounds checks into generic "safe" code shouldn't be this
> easy. 

I don't think I agree, but that's a separate argument, about
UnsafeBufferPointer.  This is about passing *untyped* memory without
bounds checks.

> You should need to explicitly wrap it up in something safe. And I
> really don't want the known-to-be-error-prone indexing model in
> concrete unsafe code.


-- 
-Dave

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Alexis Beingessner via swift-evolution]

> On Dec 8, 2016, at 3:50 PM, Dave Abrahams via swift-evolution 
>  wrote:
> 
> 
> on Thu Dec 08 2016, Ben Cohen  wrote:
> 
>>> On Dec 2, 2016, at 8:27 PM, Nate Cook  wrote:
>>> 
 On Dec 2, 2016, at 2:12 PM, Ben Cohen via swift-evolution
 >
 wrote:
>> 
 
> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution
> >
> wrote:
> 
> 3) Make all buffer pointers their own slices but use a different
> index type. If the indices were just wrapped pointers, that would
> handle the index sharing without needing an additional property on
> the buffer. We could also maintain integer-based stridable
> conformance (which greatly simplifies index arithmetic), since the
> indices would just offset by a byte for raw buffers or a stride
> for typed buffers.
> 
 
 Unfortunately, switching to non-integer indices would change this
 from being mildly source-breaking to being extremely
 source-breaking, as there’s lots of code out there using buffers
 today indexing them with integers (including integer literals).
 
 The big win with UnsafeBufferPointer having an integer index is
 it’s a drop-in replacement for arrays, so when you hit a
 performance problem using an array you can quickly switch to using
 a buffer under most circumstances instead without having to change
 much of your code – including code that uses for i in
 0..>> 
>>> It is definitely very source-breaking, though with relatively simple fixits:
>>> 
>>>buf[0] ---> buf[buf.startIndex]
>>>buf[3] ---> buf[buf.startIndex + 3]
>>>buf[i] ---> buf[buf.startIndex + i]
>>> 
>>> Any integer arithmetic happening outside the subscript could be left
>>> unchanged. If that cost isn't worth the benefit, then making
>>> UnsafeRawBufferPointer use Slice as its slice type is probably the
>>> best way to resolve that issue.
>>> 
>>> Nate
>> 
>> The fixits aren’t quite that simple for slices, though:
>> 
>>let slice = buf[3..<6]
>>slice[3] —> slice[slice.startIndex + 0] // fixit would somehow need to 
>> know this is 0 not 3
>>slice[i] —> slice[slice.startIndex + ??] // or even need to
>> know this is, erm, I haven’t had enough coffee this morning
>> 
>> The other downside is it would thwart speculatively switching an Array
>> to an UnsafeBuffer to see if that was a bottleneck, then switching
>> back.
>> 
>>> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution 
>>>  wrote:
>>> 
>>> 1) Switch to using Slice as a wrapper for UnsafeRawBufferPointer.
>>> 
>> 
>> Based on the above, it seems like this is the least bad option, and we
>> need to do this ASAP as currently UnsafeRawBufferPointer is
>> non-compliant with the requirements of slicing and needs changing
>> before it’s more widely adopted.
> 
> Or we could say that UnsafeRawBufferPointer isn't a Collection.  Making
> it a Collection in the first place has always seemed suspect to me.
> 

If this is considered a viable option, it's the one I want. Passing types 
without bounds checks into generic "safe" code shouldn't be this easy. You 
should need to explicitly wrap it up in something safe. And I really don't want 
the known-to-be-error-prone indexing model in concrete unsafe code.

> -- 
> -Dave
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Dave Abrahams via swift-evolution]

on Thu Dec 08 2016, Ben Cohen  wrote:

>> On Dec 2, 2016, at 8:27 PM, Nate Cook  wrote:
>> 
>>> On Dec 2, 2016, at 2:12 PM, Ben Cohen via swift-evolution
>>> >
>>> wrote:
>
>>> 
 On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution
 >
 wrote:
 
 3) Make all buffer pointers their own slices but use a different
 index type. If the indices were just wrapped pointers, that would
 handle the index sharing without needing an additional property on
 the buffer. We could also maintain integer-based stridable
 conformance (which greatly simplifies index arithmetic), since the
 indices would just offset by a byte for raw buffers or a stride
 for typed buffers.
 
>>> 
>>> Unfortunately, switching to non-integer indices would change this
>>> from being mildly source-breaking to being extremely
>>> source-breaking, as there’s lots of code out there using buffers
>>> today indexing them with integers (including integer literals).
>>> 
>>> The big win with UnsafeBufferPointer having an integer index is
>>> it’s a drop-in replacement for arrays, so when you hit a
>>> performance problem using an array you can quickly switch to using
>>> a buffer under most circumstances instead without having to change
>>> much of your code – including code that uses for i in
>>> 0..>> wild. Switching to an opaque index would break anyone doing that.
>> 
>> It is definitely very source-breaking, though with relatively simple fixits:
>> 
>>  buf[0] ---> buf[buf.startIndex]
>>  buf[3] ---> buf[buf.startIndex + 3]
>>  buf[i] ---> buf[buf.startIndex + i]
>> 
>> Any integer arithmetic happening outside the subscript could be left
>> unchanged. If that cost isn't worth the benefit, then making
>> UnsafeRawBufferPointer use Slice as its slice type is probably the
>> best way to resolve that issue.
>> 
>> Nate
>
> The fixits aren’t quite that simple for slices, though:
>
>   let slice = buf[3..<6]
>   slice[3] —> slice[slice.startIndex + 0] // fixit would somehow need to 
> know this is 0 not 3
>   slice[i] —> slice[slice.startIndex + ??] // or even need to
> know this is, erm, I haven’t had enough coffee this morning
>
> The other downside is it would thwart speculatively switching an Array
> to an UnsafeBuffer to see if that was a bottleneck, then switching
> back.
>
>> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution 
>>  wrote:
>> 
>> 1) Switch to using Slice as a wrapper for UnsafeRawBufferPointer.
>> 
>
> Based on the above, it seems like this is the least bad option, and we
> need to do this ASAP as currently UnsafeRawBufferPointer is
> non-compliant with the requirements of slicing and needs changing
> before it’s more widely adopted.

Or we could say that UnsafeRawBufferPointer isn't a Collection.  Making
it a Collection in the first place has always seemed suspect to me.

-- 
-Dave
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Ben Cohen via swift-evolution]

> On Dec 2, 2016, at 8:27 PM, Nate Cook  wrote:
> 
>> On Dec 2, 2016, at 2:12 PM, Ben Cohen via swift-evolution 
>> > wrote:
>> 
>>> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution 
>>> > wrote:
>>> 
>>> 3) Make all buffer pointers their own slices but use a different index 
>>> type. If the indices were just wrapped pointers, that would handle the 
>>> index sharing without needing an additional property on the buffer. We 
>>> could also maintain integer-based stridable conformance (which greatly 
>>> simplifies index arithmetic), since the indices would just offset by a byte 
>>> for raw buffers or a stride for typed buffers.
>>> 
>> 
>> Unfortunately, switching to non-integer indices would change this from being 
>> mildly source-breaking to being extremely source-breaking, as there’s lots 
>> of code out there using buffers today indexing them with integers (including 
>> integer literals).
>> 
>> The big win with UnsafeBufferPointer having an integer index is it’s a 
>> drop-in replacement for arrays, so when you hit a performance problem using 
>> an array you can quickly switch to using a buffer under most circumstances 
>> instead without having to change much of your code – including code that 
>> uses for i in 0..> wild. Switching to an opaque index would break anyone doing that.
> 
> It is definitely very source-breaking, though with relatively simple fixits:
> 
>   buf[0] ---> buf[buf.startIndex]
>   buf[3] ---> buf[buf.startIndex + 3]
>   buf[i] ---> buf[buf.startIndex + i]
> 
> Any integer arithmetic happening outside the subscript could be left 
> unchanged. If that cost isn't worth the benefit, then making 
> UnsafeRawBufferPointer use Slice as its slice type is probably the best way 
> to resolve that issue.
> 
> Nate



The fixits aren’t quite that simple for slices, though:

let slice = buf[3..<6]
slice[3] —> slice[slice.startIndex + 0] // fixit would somehow need to 
know this is 0 not 3
slice[i] —> slice[slice.startIndex + ??] // or even need to know this 
is, erm, I haven’t had enough coffee this morning

The other downside is it would thwart speculatively switching an Array to an 
UnsafeBuffer to see if that was a bottleneck, then switching back.

> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution 
>  wrote:
> 
> 1) Switch to using Slice as a wrapper for UnsafeRawBufferPointer.
> 

Based on the above, it seems like this is the least bad option, and we need to 
do this ASAP as currently UnsafeRawBufferPointer is non-compliant with the 
requirements of slicing and needs changing before it’s more widely adopted.

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Nate Cook via swift-evolution]

> On Dec 2, 2016, at 2:12 PM, Ben Cohen via swift-evolution 
> > wrote:
> 
>> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution 
>> > wrote:
>> 
>> 3) Make all buffer pointers their own slices but use a different index type. 
>> If the indices were just wrapped pointers, that would handle the index 
>> sharing without needing an additional property on the buffer. We could also 
>> maintain integer-based stridable conformance (which greatly simplifies index 
>> arithmetic), since the indices would just offset by a byte for raw buffers 
>> or a stride for typed buffers.
>> 
> 
> Unfortunately, switching to non-integer indices would change this from being 
> mildly source-breaking to being extremely source-breaking, as there’s lots of 
> code out there using buffers today indexing them with integers (including 
> integer literals).
> 
> The big win with UnsafeBufferPointer having an integer index is it’s a 
> drop-in replacement for arrays, so when you hit a performance problem using 
> an array you can quickly switch to using a buffer under most circumstances 
> instead without having to change much of your code – including code that uses 
> for i in 0.. Switching to an opaque index would break anyone doing that.

It is definitely very source-breaking, though with relatively simple fixits:

buf[0] ---> buf[buf.startIndex]
buf[3] ---> buf[buf.startIndex + 3]
buf[i] ---> buf[buf.startIndex + i]

Any integer arithmetic happening outside the subscript could be left unchanged. 
If that cost isn't worth the benefit, then making UnsafeRawBufferPointer use 
Slice as its slice type is probably the best way to resolve that issue.

Nate___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Hooman Mehr via swift-evolution]

How about providing an Int subscript in addition to switching the index type 
for the purpose of Subsequence?

> On Dec 2, 2016, at 2:12 PM, Ben Cohen via swift-evolution 
>  wrote:
> 
> 
>> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution 
>> > wrote:
>> 
>> 3) Make all buffer pointers their own slices but use a different index type. 
>> If the indices were just wrapped pointers, that would handle the index 
>> sharing without needing an additional property on the buffer. We could also 
>> maintain integer-based stridable conformance (which greatly simplifies index 
>> arithmetic), since the indices would just offset by a byte for raw buffers 
>> or a stride for typed buffers.
>> 
> 
> Unfortunately, switching to non-integer indices would change this from being 
> mildly source-breaking to being extremely source-breaking, as there’s lots of 
> code out there using buffers today indexing them with integers (including 
> integer literals).
> 
> The big win with UnsafeBufferPointer having an integer index is it’s a 
> drop-in replacement for arrays, so when you hit a performance problem using 
> an array you can quickly switch to using a buffer under most circumstances 
> instead without having to change much of your code – including code that uses 
> for i in 0.. Switching to an opaque index would break anyone doing that.
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Hooman Mehr via swift-evolution]

+1 on the third option it being clearly the best way to model pointers. Making 
the index strideable makes its use in index computations easy. It takes a bit 
care to make it work as conveniently for UnsafeBufferPointer where the stride 
will need to be special.


> On Dec 1, 2016, at 11:33 PM, Nate Cook via swift-evolution 
>  wrote:
> 
> 3) Make all buffer pointers their own slices but use a different index type. 
> If the indices were just wrapped pointers, that would handle the index 
> sharing without needing an additional property on the buffer. We could also 
> maintain integer-based stridable conformance (which greatly simplifies index 
> arithmetic), since the indices would just offset by a byte for raw buffers or 
> a stride for typed buffers.
> 
> The first option would certainly be the smallest change; the third seems like 
> it would do the most good, for the reasons Dave laid out. I've tried the 
> third option out here for raw buffers:
>   
> https://github.com/apple/swift/compare/master...natecook1000:nc-buffer-indices
>  
> 
> 
> Thanks!
> Nate
> 

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Nate Cook via swift-evolution]

> On Nov 30, 2016, at 6:15 PM, Dave Abrahams via swift-evolution 
>  wrote:
> 
> on Wed Nov 30 2016, Kevin Ballard  > wrote:
> 
>> This sounds like a sensible idea. But there is one behavioral change you
>> haven't addressed, which is that this changes how indexes work on the
>> slice. With all other slice types that come to mind, the slice shares
>> the same indexes as the base, e.g.
>> 
>>  let ary = Array(0..<10)
>> 
>>  print(ary[3]) // prints 3
>> 
>>  print(ary[2..<5][3]) // still prints 3
> 
> This is an important invariant that we need to maintain.
> 
>> UnsafeBufferPointer is indexed using 0-based integers, so with your
>> proposal, slicing an UnsafeBufferPointer produces a value that uses
>> different indexes. We could solve this by adding a new field, but that
>> would break the expectation that startIndex is always zero. 
> 
> I'm not sure that's an expectation we're obligated to honor.  Of course,
> once you get into “unsafe” territory like this, breaking even the
> expectations that aren't based on documented guarantees can be really
> dangerous.
> 
> We probably ought to have wrapped those integers in some Index type
> specific to UnsafeBufferPointer, so zero wasn't even a value.
> 
>> But we can't just ignore this problem, because algorithms that are
>> designed around collections may assume that slices preserve indexes.
>> 
>> In addition, since you point out that UnsafeRawBufferPointer is already
>> its own subsequence, and that type also guarantees that startIndex is
>> always zero, it sounds like we already have an instance of this problem
>> in the stdlib, and so this needs to be addressed with
>> UnsafeRawBufferPointer as well.
> 
> Sounds like it!

Argh, thanks to Kevin for pointing all this out!

I can see at least three approaches to resolving the inconsistency between the 
two sets of buffer types and correcting index sharing for raw buffer pointers.

1) Switch to using Slice as a wrapper for UnsafeRawBufferPointer.
2) Make all buffer pointers their own slices, keeping integer-based indices, 
but remove the zero-based index expectation. With this we'd need to add an 
additional stored property to keep track of the relative offset from the 
"original" buffer.
3) Make all buffer pointers their own slices but use a different index type. If 
the indices were just wrapped pointers, that would handle the index sharing 
without needing an additional property on the buffer. We could also maintain 
integer-based stridable conformance (which greatly simplifies index 
arithmetic), since the indices would just offset by a byte for raw buffers or a 
stride for typed buffers.

The first option would certainly be the smallest change; the third seems like 
it would do the most good, for the reasons Dave laid out. I've tried the third 
option out here for raw buffers:

https://github.com/apple/swift/compare/master...natecook1000:nc-buffer-indices

Thanks!
Nate


___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Dave Abrahams via swift-evolution]

on Thu Dec 01 2016, Jordan Rose  wrote:

>> On Nov 30, 2016, at 16:15, Dave Abrahams via swift-evolution
>  wrote:
>> 
>> 
>> on Wed Nov 30 2016, Kevin Ballard
>> 
>> >
>> wrote:
>> 
>>> This sounds like a sensible idea. But there is one behavioral change you
>>> haven't addressed, which is that this changes how indexes work on the
>>> slice. With all other slice types that come to mind, the slice shares
>>> the same indexes as the base, e.g.
>>> 
>>>  let ary = Array(0..<10)
>>> 
>>>  print(ary[3]) // prints 3
>>> 
>>>  print(ary[2..<5][3]) // still prints 3
>> 
>> This is an important invariant that we need to maintain.
>> 
>>> UnsafeBufferPointer is indexed using 0-based integers, so with your
>>> proposal, slicing an UnsafeBufferPointer produces a value that uses
>>> different indexes. We could solve this by adding a new field, but that
>>> would break the expectation that startIndex is always zero. 
>> 
>> I'm not sure that's an expectation we're obligated to honor.  Of course,
>> once you get into “unsafe” territory like this, breaking even the
>> expectations that aren't based on documented guarantees can be really
>> dangerous.
>> 
>> We probably ought to have wrapped those integers in some Index type
>> specific to UnsafeBufferPointer, so zero wasn't even a value.
>
> Since UnsafeBufferPointer is like an Array, I think it is supposed to
> have integer indexes from the start of the buffer. 

I'm not sure it's important that the indices be integers. Once we get
into this unsafe corner of the language, it might be better if they
weren't.

-- 
-Dave

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Jordan Rose via swift-evolution]

> On Nov 30, 2016, at 16:15, Dave Abrahams via swift-evolution 
>  wrote:
> 
> 
> on Wed Nov 30 2016, Kevin Ballard  > wrote:
> 
>> This sounds like a sensible idea. But there is one behavioral change you
>> haven't addressed, which is that this changes how indexes work on the
>> slice. With all other slice types that come to mind, the slice shares
>> the same indexes as the base, e.g.
>> 
>>  let ary = Array(0..<10)
>> 
>>  print(ary[3]) // prints 3
>> 
>>  print(ary[2..<5][3]) // still prints 3
> 
> This is an important invariant that we need to maintain.
> 
>> UnsafeBufferPointer is indexed using 0-based integers, so with your
>> proposal, slicing an UnsafeBufferPointer produces a value that uses
>> different indexes. We could solve this by adding a new field, but that
>> would break the expectation that startIndex is always zero. 
> 
> I'm not sure that's an expectation we're obligated to honor.  Of course,
> once you get into “unsafe” territory like this, breaking even the
> expectations that aren't based on documented guarantees can be really
> dangerous.
> 
> We probably ought to have wrapped those integers in some Index type
> specific to UnsafeBufferPointer, so zero wasn't even a value.

Since UnsafeBufferPointer is like an Array, I think it is supposed to have 
integer indexes from the start of the buffer. (It's what we're telling people 
to use as the currency type for what would be pointer/size pairs in C.) So I 
think Kevin's point is valid, and UnsafeBufferPointer cannot be its own slice 
type.

Jordan

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Dave Abrahams via swift-evolution]

on Wed Nov 30 2016, Kevin Ballard  wrote:

> This sounds like a sensible idea. But there is one behavioral change you
> haven't addressed, which is that this changes how indexes work on the
> slice. With all other slice types that come to mind, the slice shares
> the same indexes as the base, e.g.
>
>   let ary = Array(0..<10)
>
>   print(ary[3]) // prints 3
>
>   print(ary[2..<5][3]) // still prints 3

This is an important invariant that we need to maintain.

> UnsafeBufferPointer is indexed using 0-based integers, so with your
> proposal, slicing an UnsafeBufferPointer produces a value that uses
> different indexes. We could solve this by adding a new field, but that
> would break the expectation that startIndex is always zero. 

I'm not sure that's an expectation we're obligated to honor.  Of course,
once you get into “unsafe” territory like this, breaking even the
expectations that aren't based on documented guarantees can be really
dangerous.

We probably ought to have wrapped those integers in some Index type
specific to UnsafeBufferPointer, so zero wasn't even a value.

> But we can't just ignore this problem, because algorithms that are
> designed around collections may assume that slices preserve indexes.
>
> In addition, since you point out that UnsafeRawBufferPointer is already
> its own subsequence, and that type also guarantees that startIndex is
> always zero, it sounds like we already have an instance of this problem
> in the stdlib, and so this needs to be addressed with
> UnsafeRawBufferPointer as well.

Sounds like it!

> -Kevin Ballard
>
> On Wed, Nov 30, 2016, at 09:15 AM, Nate Cook via swift-evolution wrote:
>> Hello all—
>
>> This is a proposal for a fairly minor change in slicing behavior for
>> unsafe buffers.
>> Nate
>
>> 
>
>> --
>
>> 
>
>> 
>
>> This proposal changes Swift's typed UnsafeBufferPointers to be their
>> own slice type, like the UnsafeRawBufferPointer types. This is a minor
>> change in the subscript API of UnsafeBufferPointer and
>> UnsafeMutableBufferPointer, but constitutes a change to the standard
>> library's ABI, as it can't be solved through type aliasing.
>> Motivation
>
>> The standard library has parallel pointer and buffer types for working
>> with raw and typed memory. These types have broadly similar APIs that
>> streamline working with pointers, as some kinds of memory manipulation
>> involve moving back and forth between the two. One significant
>> difference between the two groups of buffer types, however, is that
>> while UnsafeRawBufferPointers are their own slice type,
>> UnsafeBufferPointers use the default Slice type as a wrapper.
>> Using a Slice wrapper is a needless addition when working with
>> buffers—the wrapper is most useful when used to prevent copying of a
>> collection's stored data, but since UnsafeBufferPointers aren't owners
>> of the memory they reference, there is no copying performed when
>> simply creating a new buffer over a subrange of the memory. Moreover,
>> the overhead of a Slice wrapper around an UnsafeBufferPointer is
>> almost certainly higher than another UnsafeBufferPointer. instance.
>> The Slice wrapper makes using buffer pointers as parameters more
>> cumbersome than necessary. To pass a slice of a buffer to a function
>> taking a buffer, you need to create a new buffer manually:
>> func _operateOnBuffer(_ buffer: UnsafeMutableBufferPointer) {
>> // ... }
>
>> let buffer: UnsafeMutableBufferPointer = ...
>> _operateOnBuffer(buffer)// okay
>> _operateOnBuffer(buffer[..<16])// error: type mismatch let
>> subBuffer = UnsafeMutableBufferPointer(start: buffer, count: 16)
>> _operateOnBuffer(subBuffer) // okay
>> The wrapper complicates subscript assignment, as well. Instead of
>> using simple assignment to copy all the elements of one buffer into a
>> memory range of another, you must either manually create a slice or
>> subscript the source buffer with its full range:
>> let biggerBuffer: UnsafeMutableBufferPointer = ... let
>> smallerBuffer: UnsafeMutableBufferPointer = ...
>> biggerBuffer[..> smallerBuffer[..> Proposed solution
>
>> The proposed solution is to switch the UnsafeBufferPointers to be
>> their own slice type. This uses less overhead than the Slice type,
>> which needs to store both the original buffer and a bounding range.
>> The operations above are simpler with this change:
>
>> _operateOnBuffer(buffer[..<16])// subscripting okay
>>
>> // no need to subscript 'smallerBuffer'
>> biggerBuffer[..> Detailed design
>
>> The change follows the example of the raw buffer pointer types:
>
>> struct UnsafeBufferPointer : Collection, ... {  // other
>> declarationssubscript(bounds: Range) -> UnsafeBufferPointer {
>> get {  // check boundsreturn UnsafeMutableBufferPointer(
>> start: self + bounds.lowerBound,  count: bounds.count) } } }
>
>> struct 

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Kevin Ballard via swift-evolution]

This sounds like a sensible idea. But there is one behavioral change you
haven't addressed, which is that this changes how indexes work on the
slice. With all other slice types that come to mind, the slice shares
the same indexes as the base, e.g.


  let ary = Array(0..<10)

  print(ary[3]) // prints 3

  print(ary[2..<5][3]) // still prints 3



UnsafeBufferPointer is indexed using 0-based integers, so with your
proposal, slicing an UnsafeBufferPointer produces a value that uses
different indexes. We could solve this by adding a new field, but that
would break the expectation that startIndex is always zero. But we can't
just ignore this problem, because algorithms that are designed around
collections may assume that slices preserve indexes.


In addition, since you point out that UnsafeRawBufferPointer is already
its own subsequence, and that type also guarantees that startIndex is
always zero, it sounds like we already have an instance of this problem
in the stdlib, and so this needs to be addressed with
UnsafeRawBufferPointer as well.


-Kevin Ballard



On Wed, Nov 30, 2016, at 09:15 AM, Nate Cook via swift-evolution wrote:
> Hello all—



> This is a proposal for a fairly minor change in slicing behavior for
> unsafe buffers.
> Nate



> 

> --

> 

> 

> This proposal changes Swift's typed UnsafeBufferPointers to be their
> own slice type, like the UnsafeRawBufferPointer types. This is a minor
> change in the subscript API of UnsafeBufferPointer and
> UnsafeMutableBufferPointer, but constitutes a change to the standard
> library's ABI, as it can't be solved through type aliasing.
> Motivation



> The standard library has parallel pointer and buffer types for working
> with raw and typed memory. These types have broadly similar APIs that
> streamline working with pointers, as some kinds of memory manipulation
> involve moving back and forth between the two. One significant
> difference between the two groups of buffer types, however, is that
> while UnsafeRawBufferPointers are their own slice type,
> UnsafeBufferPointers use the default Slice type as a wrapper.
> Using a Slice wrapper is a needless addition when working with
> buffers—the wrapper is most useful when used to prevent copying of a
> collection's stored data, but since UnsafeBufferPointers aren't owners
> of the memory they reference, there is no copying performed when
> simply creating a new buffer over a subrange of the memory. Moreover,
> the overhead of a Slice wrapper around an UnsafeBufferPointer is
> almost certainly higher than another UnsafeBufferPointer. instance.
> The Slice wrapper makes using buffer pointers as parameters more
> cumbersome than necessary. To pass a slice of a buffer to a function
> taking a buffer, you need to create a new buffer manually:
> func _operateOnBuffer(_ buffer: UnsafeMutableBufferPointer) {
> // ... }

> let buffer: UnsafeMutableBufferPointer = ...
> _operateOnBuffer(buffer)// okay
> _operateOnBuffer(buffer[..<16])// error: type mismatch let
> subBuffer = UnsafeMutableBufferPointer(start: buffer, count: 16)
> _operateOnBuffer(subBuffer) // okay
> The wrapper complicates subscript assignment, as well. Instead of
> using simple assignment to copy all the elements of one buffer into a
> memory range of another, you must either manually create a slice or
> subscript the source buffer with its full range:
> let biggerBuffer: UnsafeMutableBufferPointer = ... let
> smallerBuffer: UnsafeMutableBufferPointer = ...
> biggerBuffer[.. smallerBuffer[.. Proposed solution



> The proposed solution is to switch the UnsafeBufferPointers to be
> their own slice type. This uses less overhead than the Slice type,
> which needs to store both the original buffer and a bounding range.
> The operations above are simpler with this change:



> _operateOnBuffer(buffer[..<16])// subscripting okay
>
> // no need to subscript 'smallerBuffer'
> biggerBuffer[.. Detailed design



> The change follows the example of the raw buffer pointer types:



> struct UnsafeBufferPointer : Collection, ... {  // other
> declarationssubscript(bounds: Range) -> UnsafeBufferPointer {
> get {  // check boundsreturn UnsafeMutableBufferPointer(
> start: self + bounds.lowerBound,  count: bounds.count) } } }

> struct UnsafeMutableBufferPointer : Collection, ... {  //
> other declarationssubscript(bounds: Range) ->
> UnsafeMutableBufferPointer {  get {  // check boundsreturn
> UnsafeMutableBufferPointer(  start: self + bounds.lowerBound,  count:
> bounds.count) }  set {  // check bounds
> _writeBackMutableSlice(, bounds: bounds, slice: newValue) } } }
> Impact on existing code



> Any existing code that works with slices of UnsafeMutableBufferPointer
> and specifies the Slice type explicitly will need to change that
> specification. This isn't a terribly common thing to do (I can't find
> 

Re: [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

[Karl via swift-evolution]

+1. Sensible change.

> On 30 Nov 2016, at 18:15, Nate Cook via swift-evolution 
>  wrote:
> 
> Hello all—
> 
> This is a proposal for a fairly minor change in slicing behavior for unsafe 
> buffers.
> 
> Nate
> 
> 
> --
> 
> 
> This proposal changes Swift's typed UnsafeBufferPointers to be their own 
> slice type, like the UnsafeRawBufferPointer types. This is a minor change in 
> the subscript API of UnsafeBufferPointer and UnsafeMutableBufferPointer, but 
> constitutes a change to the standard library's ABI, as it can't be solved 
> through type aliasing.
>  
> Motivation
> 
> The standard library has parallel pointer and buffer types for working with 
> raw and typed memory. These types have broadly similar APIs that streamline 
> working with pointers, as some kinds of memory manipulation involve moving 
> back and forth between the two. One significant difference between the two 
> groups of buffer types, however, is that while UnsafeRawBufferPointers are 
> their own slice type, UnsafeBufferPointers use the default Slice type as a 
> wrapper.
> 
> Using a Slice wrapper is a needless addition when working with buffers—the 
> wrapper is most useful when used to prevent copying of a collection's stored 
> data, but since UnsafeBufferPointers aren't owners of the memory they 
> reference, there is no copying performed when simply creating a new buffer 
> over a subrange of the memory. Moreover, the overhead of a Slice wrapper 
> around an UnsafeBufferPointer is almost certainly higher than another 
> UnsafeBufferPointer. instance.
> 
> The Slice wrapper makes using buffer pointers as parameters more cumbersome 
> than necessary. To pass a slice of a buffer to a function taking a buffer, 
> you need to create a new buffer manually:
> 
> func _operateOnBuffer(_ buffer: UnsafeMutableBufferPointer) {
> // ...
> }
> 
> let buffer: UnsafeMutableBufferPointer = ...
> _operateOnBuffer(buffer)// okay
> _operateOnBuffer(buffer[0..<16])// error: type mismatch
> let subBuffer = UnsafeMutableBufferPointer(start: buffer, count: 16)
> _operateOnBuffer(subBuffer) // okay
> The wrapper complicates subscript assignment, as well. Instead of using 
> simple assignment to copy all the elements of one buffer into a memory range 
> of another, you must either manually create a slice or subscript the source 
> buffer with its full range:
> 
> let biggerBuffer: UnsafeMutableBufferPointer = ...
> let smallerBuffer: UnsafeMutableBufferPointer = ...
> 
> biggerBuffer[0.. smallerBuffer[0..  
> Proposed
>  solution
> 
> The proposed solution is to switch the UnsafeBufferPointers to be their own 
> slice type. This uses less overhead than the Slice type, which needs to store 
> both the original buffer and a bounding range.
> 
> The operations above are simpler with this change:
> 
> _operateOnBuffer(buffer[0..<16])// subscripting okay
> 
> // no need to subscript 'smallerBuffer'
> biggerBuffer[0..  
> Detailed
>  design
> 
> The change follows the example of the raw buffer pointer types:
> 
> struct UnsafeBufferPointer : Collection, ... {
> // other declarations
> subscript(bounds: Range) -> UnsafeBufferPointer {
> get {
> // check bounds
> return UnsafeMutableBufferPointer(
> start: self + bounds.lowerBound,
> count: bounds.count)
> }
> }
> }
> 
> struct UnsafeMutableBufferPointer : Collection, ... {
> // other declarations
> subscript(bounds: Range) -> UnsafeMutableBufferPointer {
> get {
> // check bounds
> return UnsafeMutableBufferPointer(
> start: self + bounds.lowerBound,
> count: bounds.count)
> }
> set {
> // check bounds
> _writeBackMutableSlice(, bounds: bounds, slice: newValue)
> }
> }
> }
>  
> Impact
>  on existing code
> 
> Any existing code that works with slices of UnsafeMutableBufferPointer and 
> specifies the Slice type explicitly will need to change that specification. 
> This isn't a terribly common thing to do (I can't find any in the standard 
> library or test suite), so the impact of the change should be minor.
> 
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org