Re: [swift-evolution] [REVIEW] SE-0193 - Cross-module inlining and specialization

[Johannes Weiß via swift-evolution]

Hi Slava,

> On 22 Dec 2017, at 7:13 am, Slava Pestov <spes...@apple.com> wrote:
> 
> Hi Johannes,
> 
> Thanks for reviewing this proposal!
> 
>> On Dec 21, 2017, at 8:06 AM, Johannes Weiß via swift-evolution 
>> <swift-evolution@swift.org> wrote:
> 
>> The library I'm working on will presumably never have stable ABI as you'd 
>> naturally build it with your application. However we also don't want to miss 
>> on the cross-module optimisation & specialisation and I suspect there are 
>> quite a few (mostly open-source) libraries in the same space. I'm pretty 
>> sure everybody would just end up littering their code with 
>> @abiPublic/@inlinable (or the @available(...) syntax Chris Lattner proposed) 
>> without actually meaning that.
>> 
>> Summing up: I think this feature is crucial but shouldn't come without a 
>> compiler "where all declarations become implicitly @inlinable, and all 
>> private and internal declarations become @abiPublic". I really don't want to 
>> litter the code with attributes that aren't what I mean. (basically `swift 
>> build --global-resilience-domain`) Having this compiler mode also makes 
>> these attributes IMHO really niche and therefore I can only sympathise 
>> with's Chris' sentiment to not litter the global attribute namespace.
> 
> I agree that a ‘completely non-resilient’ compiler mode would be great when 
> building libraries that are always shipped together, and I hope Swift gains 
> such a feature one day, possibly built on top of the very infrastructure used 
> to implement this proposal!

Cool. I just meant that the compiler mode and the public (as in 
non-underscored) attributes should go hand in hand to not tempt people into 
littering their code without thinking.


> However, the goal of this proposal is to formalize some language features 
> that already exist and are used by the standard library. Clearly making 
> everything fragile is a non-starter for the standard library in an ABI-stable 
> world.

totally agree that these attributes are necessary, they should be in Swift as 
soon as we bike shedded the right name. But to prevent them from being placed 
everywhere without thinking I propose to introduce a compiler mode at the same 
time.
Performance testing is hard and takes time. So I'm pretty sure that as soon as 
one @abiPublic/@inlinable has proven to improve performance, library authors 
will just start putting them anywhere as there's no downside besides making the 
code harder to read.

What is the downside of bringing the attributes and the compiler mode at the 
same time. Am I massively underestimating the amount of work required for such 
mode?


> I do hope that this attribute is not abused in the manner in which you 
> describe, but I’m not sure the potential for abuse is reason enough to not 
> run the proposal — people are already using the underscored attribute today, 
> risking source breakage and bugs due to insufficient test coverage in the 
> future.

> 
>> C(++) as described in the proposal and Haskell 
>> (https://wiki.haskell.org/Inlining_and_Specialisation), where {-# INLINABLE 
>> myFunction #-} (quoting the docs) causes exactly two things to happens.
>> 
>>  • The function's (exact) definition is included in the interface file 
>> for the module.
>>  • The function will be specialised at use sites -- even across modules.
>> Note that [the Haskell compiler] GHC is no more keen to inline an INLINABLE 
>> function than any other.
> 
> Note that Swift’s compiler is the same — @inlinable does not influence 
> optimizer decisions to inline or not.

Cool, that's what I suspected. Good to know that the Swift & GHC attributes are 
the same and even the name is :).
So we just need -fspecialise-aggressively 邏.


> Also currently the proposal is implemented by binary serialization of the SIL 
> IR, but nothing in it precludes serializing inlinable function bodies as 
> source code in the future — in fact we are likely to go in that direction if 
> we implement the proposed textual ‘stable’ module format.

Cool. Just to be sure I understand you correctly: Assuming your proposal gets 
implemented as proposed, a function that does _not_ have the @inlinable 
attribute it won't be specialised across modules, ever. Correct?


-- Johannes

> 
> Slava
> 

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [REVIEW] SE-0193 - Cross-module inlining and specialization

[Johannes Weiß via swift-evolution]

> On 21 Dec 2017, at 4:06 pm, Johannes Weiß via swift-evolution 
> <swift-evolution@swift.org> wrote:
> 
>> 
>> On 21 Dec 2017, at 12:19 am, Ted Kremenek via swift-evolution 
>> <swift-evolution@swift.org> wrote:
>> 
>> The review of "SE-0193 - Cross-module inlining and specialization" begins 
>> now and runs through January 5, 2018.
>> 
>> The proposal is available here:
>> 
>> https://github.com/apple/swift-evolution/blob/master/proposals/0193-cross-module-inlining-and-specialization.md
>> Reviews are an important part of the Swift evolution process. All review 
>> feedback should be sent to the swift-evolution mailing list at:
>> 
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>> or, if you would like to keep your feedback private, directly to the review 
>> manager. 
>> 
>> When replying, please try to keep the proposal link at the top of the 
>> message:
>> 
>> Proposal link: 
>> https://github.com/apple/swift-evolution/blob/master/proposals/0193-cross-module-inlining-and-specialization.md
>> ...
>> Reply text
>> ...
>> Other replies
>> What goes into a review of a proposal?
>> 
>> The goal of the review process is to improve the proposal under review 
>> through constructive criticism and, eventually, determine the direction of 
>> Swift. 
>> 
>> When reviewing a proposal, here are some questions to consider:
>> 
>>  • What is your evaluation of the proposal?
> 
> I'm working on a performance sensitive library and we're sometimes bitten 
> quite hard by not being able to cross-module inline & specialise. Therefore, 
> it's thrilling to see that you're working in this area.
> 
> However, I have to admit that I believe this language feature will most 
> likely be grossly abused. The library I'm working on will presumably never 
> have stable ABI as you'd naturally build it with your application. However we 
> also don't want to miss on the cross-module optimisation & specialisation and 
> I suspect there are quite a few (mostly open-source) libraries in the same 
> space. I'm pretty sure everybody would just end up littering their code with 
> @abiPublic/@inlinable (or the @available(...) syntax Chris Lattner proposed) 
> without actually meaning that.
> 
> Summing up: I think this feature is crucial but shouldn't come without a 
> compiler "where all declarations become implicitly @inlinable, and all 
> private and internal declarations become @abiPublic". I really don't want to 
> litter the code with attributes that aren't what I mean. (basically `swift 
> build --global-resilience-domain`) Having this compiler mode also makes these 
> attributes IMHO really niche and therefore I can only sympathise with's 
> Chris' sentiment to not litter the global attribute namespace.
> 
> 
>>  • Is the problem being addressed significant enough to warrant a change 
>> to Swift?
> 
> see above.
> 
> 
>>  • Does this proposal fit well with the feel and direction of Swift?
> 
> to back up the 'swift' claim, cross-module inlining & specialisation is 
> absolutely necessary. However this should also be achievable with a 'I don't 
> need a stable ABI for this product' mode in the compiler :).
> 
> 
>>  • If you have used other languages or libraries with a similar feature, 
>> how do you feel that this proposal compares to those?
> 
> C(++) as described in the proposal and Haskell 
> (https://wiki.haskell.org/Inlining_and_Specialisation), where {-# INLINABLE 
> myFunction #-} (quoting the docs) causes exactly two things to happens.
> 
>   • The function's (exact) definition is included in the interface file 
> for the module.
>   • The function will be specialised at use sites -- even across modules.
> Note that [the Haskell compiler] GHC is no more keen to inline an INLINABLE 
> function than any other.

forgot to mention GHC's -fspecialise-aggressively which is its way of saying, 
just specialise things across modules even without '{-# INLINABLE ... #-}' 
which is basically the compilation mode I'm asking for :).

> 
> 
>>  • How much effort did you put into your review? A glance, a quick 
>> reading, or an in-depth study?
> 
> read the proposal (and believe to understand it).
> 
> -- Johannes
> 
>> 
>> Thanks,
>> Ted Kremenek
>> Review Manager
>> 
>> ___
>> swift-evolution mailing list
>> swift-evolution@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [REVIEW] SE-0193 - Cross-module inlining and specialization

[Johannes Weiß via swift-evolution]

> On 21 Dec 2017, at 12:19 am, Ted Kremenek via swift-evolution 
>  wrote:
> 
> The review of "SE-0193 - Cross-module inlining and specialization" begins now 
> and runs through January 5, 2018.
> 
> The proposal is available here:
> 
> https://github.com/apple/swift-evolution/blob/master/proposals/0193-cross-module-inlining-and-specialization.md
> Reviews are an important part of the Swift evolution process. All review 
> feedback should be sent to the swift-evolution mailing list at:
> 
> https://lists.swift.org/mailman/listinfo/swift-evolution
> or, if you would like to keep your feedback private, directly to the review 
> manager. 
> 
> When replying, please try to keep the proposal link at the top of the message:
> 
> Proposal link: 
> https://github.com/apple/swift-evolution/blob/master/proposals/0193-cross-module-inlining-and-specialization.md
> ...
> Reply text
> ...
> Other replies
> What goes into a review of a proposal?
> 
> The goal of the review process is to improve the proposal under review 
> through constructive criticism and, eventually, determine the direction of 
> Swift. 
> 
> When reviewing a proposal, here are some questions to consider:
> 
>   • What is your evaluation of the proposal?

I'm working on a performance sensitive library and we're sometimes bitten quite 
hard by not being able to cross-module inline & specialise. Therefore, it's 
thrilling to see that you're working in this area.

However, I have to admit that I believe this language feature will most likely 
be grossly abused. The library I'm working on will presumably never have stable 
ABI as you'd naturally build it with your application. However we also don't 
want to miss on the cross-module optimisation & specialisation and I suspect 
there are quite a few (mostly open-source) libraries in the same space. I'm 
pretty sure everybody would just end up littering their code with 
@abiPublic/@inlinable (or the @available(...) syntax Chris Lattner proposed) 
without actually meaning that.

Summing up: I think this feature is crucial but shouldn't come without a 
compiler "where all declarations become implicitly @inlinable, and all private 
and internal declarations become @abiPublic". I really don't want to litter the 
code with attributes that aren't what I mean. (basically `swift build 
--global-resilience-domain`) Having this compiler mode also makes these 
attributes IMHO really niche and therefore I can only sympathise with's Chris' 
sentiment to not litter the global attribute namespace.


>   • Is the problem being addressed significant enough to warrant a change 
> to Swift?

see above.


>   • Does this proposal fit well with the feel and direction of Swift?

to back up the 'swift' claim, cross-module inlining & specialisation is 
absolutely necessary. However this should also be achievable with a 'I don't 
need a stable ABI for this product' mode in the compiler :).


>   • If you have used other languages or libraries with a similar feature, 
> how do you feel that this proposal compares to those?

C(++) as described in the proposal and Haskell 
(https://wiki.haskell.org/Inlining_and_Specialisation), where {-# INLINABLE 
myFunction #-} (quoting the docs) causes exactly two things to happens.

• The function's (exact) definition is included in the interface file 
for the module.
• The function will be specialised at use sites -- even across modules.
Note that [the Haskell compiler] GHC is no more keen to inline an INLINABLE 
function than any other.


>   • How much effort did you put into your review? A glance, a quick 
> reading, or an in-depth study?

read the proposal (and believe to understand it).

-- Johannes

> 
> Thanks,
> Ted Kremenek
> Review Manager
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [RFC] Associated type inference

[Johannes Weiß via swift-evolution]

Hi Douglas,

First of all, thanks very much for looking into this seemingly dark corner of 
the compiler. This caused us a lot of trouble already.

Comments inline.

> On 1 Dec 2017, at 12:28 am, Douglas Gregor via swift-evolution 
>  wrote:
> 
> Hello Swift community,
> 
> Associated type inference, which is the process by which the Swift compiler 
> attempts to infer typealiases to satisfy associated-type requirements based 
> on other requirements, has caused both implementation problems and user 
> confusion for a long time. Some of you might remember a previous (failed) 
> attempt to remove this feature from the Swift language, in SE-0108 “Remove 
> associated type inference”. 
> 
> I’m not sure we can remove this feature outright (i.e., the concerns that 
> sank that proposal are still absolutely valid), because it is so very 
> convenient and a lot of user code likely depends on it in some form or other. 
> So, here I’d like to describe the uses of the feature, its current (very 
> problematic) implementation approach, and a half-baked proposal to narrow the 
> scope of associated type inference to something that I think is more 
> tractable. I need help with the design of this feature, because I feel like 
> it’s going to be a delicate balance between implementability and 
> expressiveness.
> 
> A Motivating Example
> As a motivating example, let’s consider a “minimal” random access collection:
> 
> struct MyCollection {
> var contents: [T]
> }
> 
> extension MyCollection: RandomAccessCollection {
> var startIndex: Int { return contents.startIndex }
> var endIndex: Int { return contents.endIndex }
> subscript(index: Int) -> T { return contents[index] }
> }
> 
> This is actually pretty awesome: by providing just two properties and a 
> subscript, we get the full breadth of the random access collection API! This 
> is relying heavily on associated type inference (for associated type 
> requirements) and default implementations specified on protocol extensions. 
> Without associated type inference, we would have had to write:
> 
> 
> extension MyCollection: RandomAccessCollection {
> typealias Element = T
> typealias Index = Int
> typealias Indices = CountableRange
> typealias Iterator = IndexingIterator
> typealias SubSequence = RandomAccessSlice
> 
> var startIndex: Int { return contents.startIndex }
> var endIndex: Int { return contents.endIndex }
> subscript(index: Int) -> T { return contents[index] }
> }
> 
> where the bolded typealiases are currently inferred. It was worse back when 
> we reviewed SE-0108, because IIRC there were a few underscored associated 
> types (e.g., _Element) that have since been removed. Still, that’s a bit of 
> additional work to define a “minimal” collection, and requires quite a bit 
> more understanding: how do I know to choose IndexingIterator, and 
> CountableRange, and RandomAccessSlice?
> 
> The issue would get worse with, e.g., SE-0174 “Change filter to return an 
> associated type”, which adds an associated type Filtered that almost nobody 
> will ever customize, and isn’t really fundamental to the way collections 
> work. Adding Filtered to the standard library would be a source-breaking 
> change, because users would have to write a typealias giving it its default.
> 
> Associated Type Defaults
> One of the ways in which we avoid having to specify typealiases is to use 
> associated type defaults. For example, the standard library contains 
> associated type defaults for Indices, Iterator, and SubSequence. Let’s focus 
> on Indices:
> 
> protocol Collection : Sequence {
>   associatedtype Indices = DefaultIndices
>   // ...
> }
> 
> protocol BidirectionalCollection : Collection {
>   associatedtype Indices = DefaultBidirectionalIndices
>   // ...
> }
> 
> protocol RandomAccessCollection : BidirectionalCollection {
>   associatedtype Indices = DefaultRandomAccessIndices
>   // ...
> }
> 
> The basic idea here is that different protocols in the hierarchy provide 
> different defaults, and you presumably want the default from the most 
> specific protocol. If I define a type and make it conform to 
> BidirectionalCollection, I’d expect to get DefaultBidirectionalIndices 
> for Indices. If a define a type and make it conform to RandomAccessIterator, 
> I’d expect to get DefaultRandomAccessIndices.
> 
> (Aside: DefaultRandomAccessIndices and DefaultBidirectionalIndices got 
> collapsed into DefaultIndices now that we have conditional conformances for 
> the standard library, but the issues I’m describing remain).
> 
> Associated type defaults seem like a reasonable feature that fits well enough 
> into the design. However, it’s not the only thing in place with our 
> MyCollection example, for which Indices was inferred to CountableRange. How’s 
> that happen?
> 
> Associated Type Inference
> Associated type inference attempts to look at the 

Re: [swift-evolution] [swift-evolution-announce] [Review] SE-0187: Introduce Sequence.filterMap(_:)

[Johannes Weiß via swift-evolution]

+1 and agree with Stephen on why

> On 8 Nov 2017, at 8:17 am, Stephen Celis via swift-evolution 
>  wrote:
> 
> +1
> 
>> On Nov 7, 2017, at 6:23 PM, John McCall  wrote:
>> 
>>  • What is your evaluation of the proposal?
> 
> BJ summarized my thoughts nicely. I think “flatMap” in its current form is 
> confusing to both newcomers and functional programmers familiar with the 
> nomenclature. I’ve been aware of this disparity and thought about it for 
> awhile. I typically redefine “flatMap” as “mapOptional” in my projects for 
> clarity.
> 
>>  • Is the problem being addressed significant enough to warrant a change 
>> to Swift?
> 
> I think so! It improves the user experience in a significant way to those 
> that encounter the function in the wild.
> 
>>  • Does this proposal fit well with the feel and direction of Swift?
> 
> Yes.
> 
>>  • If you have used other languages or libraries with a similar feature, 
>> how do you feel that this proposal compares to those?
> 
> It fits in well. Swift’s current extra implementation of “flatMap” is the 
> anomaly.
> 
>>  • How much effort did you put into your review? A glance, a quick 
>> reading, or an in-depth study?
> 
> Gave a quick reading to the proposal.
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] Making capturing semantics of local

[Johannes Weiß via swift-evolution]

> On 29 Oct 2017, at 3:01 pm, Mike Kluev  wrote:
> 
>> On 29 October 2017 at 14:02, Johannes Weiß  wrote:
>> 
>> Definitely not arguing with that. But there are (valid?) cases when you want 
>> a recursive closure which doesn’t have a native recursion mechanism and then 
>> `fix` can be useful I’d argue. I think more straightforward than
>> 
>> recursive.c = { x in
>>(x == 0) ? 1 : x * recursive.c(x - 1)
>>}
>> 
> 
> you can do without "recursive.c":
> 
> var factorial: ((Int) -> Int)!
> 
> factorial = { n in
> n == 0 ? 1 : n * factorial(n - 1)
> }

That’s nicer, thanks!

> 
> factorial(5) // 120
> 
>  
>> . But fortunately have local functions, I can only recall wanting a 
>> recursive closure once.
>> 
> 
> in down to earth practical programming even if i didn't have local functions 
> i probably wouldn't bother abusing closures to implement recursion, would 
> just used what's available: methods. and if there is some extra state to pass 
> from the outer scopes, well, so be it, either via parameters of via some 
> instance variables, etc. wasn't too much of a problem in good old C/C++. in a 
> way, knowing exact state you want to pass and passing it explicitly organises 
> the code and the reasoning about it.
> 
> Mike
> 
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] Making capturing semantics of local

[Johannes Weiß via swift-evolution]

> On 28 Oct 2017, at 10:14 pm, John McCall <rjmcc...@apple.com> wrote:
> 
> 
>> On Oct 28, 2017, at 6:05 AM, Johannes Weiß via swift-evolution 
>> <swift-evolution@swift.org> wrote:
>> 
>> Hi Mike,
>> 
>>> On 27 Oct 2017, at 7:05 pm, Mike Kluev <mike.kl...@gmail.com> wrote:
>>> 
>>> on Date: Fri, 27 Oct 2017 17:52:54 +0100 Johannes Weiß 
>>> <johanneswe...@apple.com> wrote:
>>> 
>>>> On 27 Oct 2017, at 6:27 am, Howard Lovatt via swift-evolution 
>>>> <swift-evolution@swift.org> wrote:
>>>> 
>>>> In terms of recursion you can fiddle it:
>>>> 
>>>> struct RecursiveClosure {
>>>>var c: C! = nil
>>>> }
>>>> func factorial(_ n: Int) -> Int {
>>>>var recursive = RecursiveClosure<(Int) -> Int>()
>>>>recursive.c = { x in
>>>>(x == 0) ? 1 : x * recursive.c(x - 1)
>>>>}
>>>>return recursive.c(n)
>>>> }
>>>> factorial(5) // 120
>>> 
>>> what a hack and a half :)
>>> 
>>> sorry, offtopic to the thread but that you can have easier with the 
>>> fixed-point combinator 
>>> (https://en.wikipedia.org/wiki/Fixed-point_combinator)
>>> 
>>> // the fixed-point combinator
>>> func fix(_ f: @escaping ((@escaping (T) -> T) -> (T) -> T)) -> (T) -> T {
>>>return { (x: T) in (f(fix(f)))(x) }
>>> }
>>> 
>>> // demo
>>> let fact = fix { fact_ in { n in n == 1 ? 1 : n * fact_(n-1) } }
>>> for i in 1..<10 {
>>>print(fact(i))
>>> }
>>> 
>>> that would be a serious crime against humanity if swift allows this type of 
>>> code at all :-)
>> 
>> the fixed-point combinator and Y combinator are pretty important in 
>> functional languages.
> 
> They're important in the theory of functional languages.  Anyone seriously 
> promoting using the Y combinator in actual programming instead of using the 
> language's native recursive-binding features is, frankly, someone you should 
> not being taking advice from.

Definitely not arguing with that. But there are (valid?) cases when you want a 
recursive closure which doesn’t have a native recursion mechanism and then 
`fix` can be useful I’d argue. I think more straightforward than

>>>> recursive.c = { x in
>>>>(x == 0) ? 1 : x * recursive.c(x - 1)
>>>>}

. But fortunately have local functions, I can only recall wanting a recursive 
closure once.

— Johannes
> 
> John.
> 
>> And the above code works in Swift. The good thing is that you need to write 
>> `fix` only once and you can then use it for all closures that need to be 
>> recursive.
>> 
>> 
>> -- Johannes
>> 
>>> 
>>> Mike
>>> 
>> 
>> ___
>> swift-evolution mailing list
>> swift-evolution@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
> 
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] Making capturing semantics of local

[Johannes Weiß via swift-evolution]

Hi Mike,

> On 27 Oct 2017, at 7:05 pm, Mike Kluev  wrote:
> 
> on Date: Fri, 27 Oct 2017 17:52:54 +0100 Johannes Weiß 
>  wrote:
> 
> > On 27 Oct 2017, at 6:27 am, Howard Lovatt via swift-evolution 
> >  wrote:
> >
> > In terms of recursion you can fiddle it:
> >
> > struct RecursiveClosure {
> > var c: C! = nil
> > }
> > func factorial(_ n: Int) -> Int {
> > var recursive = RecursiveClosure<(Int) -> Int>()
> > recursive.c = { x in
> > (x == 0) ? 1 : x * recursive.c(x - 1)
> > }
> > return recursive.c(n)
> > }
> > factorial(5) // 120
> 
> what a hack and a half :)
> 
> sorry, offtopic to the thread but that you can have easier with the 
> fixed-point combinator (https://en.wikipedia.org/wiki/Fixed-point_combinator)
> 
> // the fixed-point combinator
> func fix(_ f: @escaping ((@escaping (T) -> T) -> (T) -> T)) -> (T) -> T {
> return { (x: T) in (f(fix(f)))(x) }
> }
> 
> // demo
> let fact = fix { fact_ in { n in n == 1 ? 1 : n * fact_(n-1) } }
> for i in 1..<10 {
> print(fact(i))
> }
> 
> that would be a serious crime against humanity if swift allows this type of 
> code at all :-)

the fixed-point combinator and Y combinator are pretty important in functional 
languages. And the above code works in Swift. The good thing is that you need 
to write `fix` only once and you can then use it for all closures that need to 
be recursive.


-- Johannes

> 
> Mike
> 

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] Making capturing semantics of local

[Johannes Weiß via swift-evolution]

> On 27 Oct 2017, at 6:27 am, Howard Lovatt via swift-evolution 
>  wrote:
> 
> > Closures cannot replace all uses of local functions. Local functions can be 
> > recursive, and have a generic parameter list.
> 
> My response would be add these featurtes to closures, it will make closures 
> better.
> 
> In terms of recursion you can fiddle it:
> 
> struct RecursiveClosure {
> var c: C! = nil
> }
> func factorial(_ n: Int) -> Int {
> var recursive = RecursiveClosure<(Int) -> Int>()
> recursive.c = { x in
> (x == 0) ? 1 : x * recursive.c(x - 1)
> }
> return recursive.c(n)
> }
> factorial(5) // 120

sorry, offtopic to the thread but that you can have easier with the fixed-point 
combinator (https://en.wikipedia.org/wiki/Fixed-point_combinator)

// the fixed-point combinator
func fix(_ f: @escaping ((@escaping (T) -> T) -> (T) -> T)) -> (T) -> T {
return { (x: T) in (f(fix(f)))(x) }
}

// demo
let fact = fix { fact_ in { n in n == 1 ? 1 : n * fact_(n-1) } }
for i in 1..<10 {
print(fact(i))
}

-- Johannes


> 
>   -- Howard.
> 
> On 27 October 2017 at 15:53, Slava Pestov  wrote:
> 
> 
>> On Oct 26, 2017, at 9:52 PM, Howard Lovatt  wrote:
>> 
>> It would be easy to provide a fixit.
> 
> That is still a source breaking change that requires migration, though.
> 
>> How often are they actually used? I rarely use them and all my use cases 
>> could be a closure instead. 
>> 
>> Also see Mike Kluev example; local functions are arguably worse than 
>> closures in all cases (provided that you can appropriately annotate the 
>> function type).
> 
> Closures cannot replace all uses of local functions. Local functions can be 
> recursive, and have a generic parameter list.
> 
> Slava
> 
>> 
>>   -- Howard.
>> 
>> On 27 October 2017 at 12:26, Slava Pestov  wrote:
>> 
>> 
>>> On Oct 26, 2017, at 4:45 PM, Howard Lovatt via swift-evolution 
>>>  wrote:
>>> 
>>> Rather than expand local function syntax why not deprecate local functions 
>>> completely
>> 
>> I don’t think at this point such a drastic change to the language is likely 
>> to happen.
>> 
>> Slava
>> 
> 
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] Making capturing semantics of local functions explicit

[Johannes Weiß via swift-evolution]

Hi,

> On 25 Oct 2017, at 6:01 pm, John McCall via swift-evolution 
>  wrote:
> 
>> On Oct 25, 2017, at 7:41 AM, David Hart via swift-evolution 
>>  wrote:
>> I got bit again by a sneaky memory leak concerning local functions and would 
>> like to discuss a small language change. I vaguely remember this being 
>> discussed in the past, but can’t find the thread (if anybody could point me 
>> to it, I’d appreciate it). Basically, here’s an example of the leak:
>> 
>> class A
>>  {
>> 
>> func foo
>> () {
>> 
>> func local
>> () {
>> 
>> bar
>> ()
>> }
>> 
>> methodWithEscapingClosure { [
>> unowned self] _ in
>> 
>> 
>> self.bar
>> ()
>> 
>> local() // this leaks because local captures self
>> }
>> }
>> 
>> 
>> func bar
>> () {
>> }
>> }
>> 
>> 
>> Its sneaky because local’s capturing of self is not obvious if you’ve 
>> trained your brain to watch out for calls prefixed with self. I would 
>> suggest having the compiler force users to make self capturing explicit, the 
>> same way it does for closures:
> 
> I think this is a good idea.  Ideally the proposal would also allow explicit 
> capture lists in local functions.

Yes, please. Optionally allowing exhaustive capture lists would be great. I 
think they work really well as compiler checked documentation too.
Sometimes a programmer makes sure that a given closure doesn't capture 
something and maybe even adds a comment explaining that and why this might be 
important. Still, it's fairly easy for another programmer to accidentally 
capture something without even noticing. And given that we mostly review on 
diffs with limited context it's even possible that this change slips though the 
review as the comment doesn't necessarily appear in the review.

With optional exhaustive capture lists, this problem would go away because a) a 
programmer is more likely to think about the consequences when changing the 
capture list b) it's obvious in code review.
That's especially true if capture lists are optional because the mere fact that 
there is a capture list suggests that the code might rely on the very captures 
only.

-- Johannes


> 
> John.
> 
>> 
>> class A
>>  {
>> 
>> func foo
>> () {
>> 
>> func local
>> () {
>> 
>> bar
>> () // error: Call to method ‘bar' in function ‘local' requires explicit 
>> 'self.' to make capture semantics explicit
>> }
>> 
>> 
>>  
>> // ...
>> }
>> }
>> 
>> 
>> What do you think?
>> 
>> David.
>> 
>> ___
>> swift-evolution mailing list
>> swift-evolution@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] Pitch: Deprecate/remove AnyObject method dispatch

[Johannes Weiß via swift-evolution]

yes, please. Especially having quite a long time (4.1 until 5.0) where this 
works but issues a warning sounds like a great transition plan.

> On 24 Oct 2017, at 11:02 pm, Slava Pestov via swift-evolution 
>  wrote:
> 
> Hi all,
> 
> Dynamic dispatch of methods through AnyObject is a source of implementation 
> complexity, has many known bugs which we are not going to fix, and no longer 
> makes sense now in an id-as-Any world; AnyObject is not the ‘common currency’ 
> type for arbitrary Objective-C objects anymore.
> 
> I would like to propose we deprecate it as follows:
> 
> - Unconditional warning in Swift 4.1, with a fixit to add an ‘as’ cast to 
> cast the base value to the right type
> - Error in Swift 5 in -swift-version 5 mode
> 
> Thoughts? Does anyone actually rely on this feature, instead of just 
> stumbling on it by accident once in a while?
> 
> Slava
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] Pitch: Member lookup on String should not find members of NSString

[Johannes Weiß via swift-evolution]

+1, it's currently really non-obvious where these automatic bridges are 
happening which keeps costing me time.

> On 24 Oct 2017, at 11:00 pm, Slava Pestov via swift-evolution 
>  wrote:
> 
> Hi,
> 
> Members of NSString, except those defined in Foundation, are available on 
> values of type String. For example,
> 
> extension NSString {
>  @objc func foo() {}
> }
> 
> let s: String = “hello”
> 
> s.foo()
> 
> We don’t do this for any other bridged types, for instance NSArray methods 
> are not imported as Array methods. It’s literally a special case in the type 
> checker for member lookup on String.
> 
> This behavior doesn’t really much sense conceptually and it was put in as a 
> stop-gap in Swift 1 to beef up the String API. I would like to phase it out 
> as follows:
> 
> - Unconditional warning in Swift 4.1, with a fixit to insert an ‘as NSString’ 
> cast
> - Error in Swift 5 with -swift-version 5
> 
> What does everyone think about this?
> 
> Slava
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] Opaque Pointers in Swift

[Johannes Weiß via swift-evolution]

Hi Cory,

I think we're dealing with two separate issues here.

1) that all forward declared struct pointers get imported as an OpaquePointer 
which makes us lose all type-safety
2) that it's a fairly frequent case that C libraries evolve from 'pointers to 
fully declared structs' to 'pointers to forward declared structs'


Regarding 1)

I fully agree that is pretty bad and I believe I have an idea what the Swift 
importer should do for that case:

For the following C types

struct foo_s;
typedef foo_s *foo;

struct bar_s;
typedef bar_s *bar;

the C compiler today imports both `foo` and `bar` as `OpaquePointer` which 
isn't very helpful. Instead I believe the importer should declare to phantom 
types

enum foo_s {}
enum bar_s {}

and import `foo` as `typealias foo = OpaquePointer` and `bar` as 
`typealias bar = OpaquePointer`.

That seems to preserve the right semantics:

 - we can't have any values of `foo_s` or `bar_s` as they're phantom
 - we can however have pointers to them
 - in this example the pointers are `OpaquePointer` and 
`OpaquePointer`, deliberately not `UnsafePointer` and 
`UnsafePointer` so we don't have an issue that the user can do 
'unsafePtr.pointee'

How do people think about this proposed change?

Regarding 2)


I feel you'd prefer if we'd import the above as `typealias foo = 
UnsafePointer` which I would be happy with but I can also see why 
`OpaquePointer` exists: It stops us from dereferencing the pointer at compile 
time (just like C does).

So yes, I agree we should fix (1). For the time being I believe I have 
something that might work for you use-case even today (and would work even 
nicer if the above changes are implemented):

Today, we already have the following initialisers

struct UnsafePointer {
init?(_ ptr: OpaquePointer)
}

and

struct OpaquePointer {
init?(_ ptr: UnsafePointer)
}

by just adding the following two trivial ones:

extension UnsafePointer {
init?(_ ptr: UnsafePointer) { return ptr }
}
extension OpaquePointer {
init?(_ ptr: OpaquePointer) { return ptr }
}

this should solve your problem:

- when you receive a pointer from the C library, store it as OpaquePointer
let myOpaquePointer: OpaquePointer = 
OpaquePointer(c_library_create_function())
  which now should work regardless of whether you're linking the old or the new 
version of the C library
- when you pass a pointer to the C library:
c_library_consuming_function(.init(myOpaquePointer))
  which should (modulo it doesn't right now 
https://bugs.swift.org/browse/SR-6211) select the right initialiser for you, 
only it doesn't 

But fortunately we can work around it:

--- SNIP ---
extension UnsafePointer {
static func make(_ ptr: UnsafePointer) -> UnsafePointer {
return ptr
}
static func make(_ ptr: OpaquePointer) -> UnsafePointer {
return UnsafePointer(ptr)
}
}
extension OpaquePointer {
static func make(_ ptr: UnsafePointer) -> OpaquePointer {
return OpaquePointer(ptr)!
}
static func make(_ ptr: OpaquePointer) -> OpaquePointer {
return ptr
}
}

func mockCLibraryCreateOld() -> UnsafePointer {
return UnsafePointer(UnsafeMutablePointer.allocate(capacity: 1))
}

func mockCLibraryCreateNew() -> OpaquePointer {
return OpaquePointer(mockCLibraryCreateOld())
}

func mockCLibraryConsumeOld(_ x: UnsafePointer) {}
func mockCLibraryConsumeNew(_ x: OpaquePointer) {}


let fromCold: OpaquePointer = .make(mockCLibraryCreateOld())
let fromCnew: OpaquePointer = .make(mockCLibraryCreateNew())

mockCLibraryConsumeOld(.make(fromCold))
mockCLibraryConsumeNew(.make(fromCnew))
mockCLibraryConsumeOld(.make(fromCnew))
mockCLibraryConsumeNew(.make(fromCold))
--- SNAP ---

HTH

-- Johannes



> On 24 Oct 2017, at 9:14 am, Cory Benfield via swift-evolution 
>  wrote:
> 
> I wanted to discuss a recent difficulty I’ve encountered while writing a 
> Swift program that uses a C library that has recently changed its API to use 
> opaque pointers, with an eye towards asking whether there are suggestions for 
> ways to tackle the problem that I haven’t considered, or whether some 
> enhancement to Swift should be proposed to provide a solution.
> 
> A common trend in modern C code is to encapsulate application data by using 
> pointers to “opaque” data structures: that is, data structures whose complete 
> definition is not available in the header files for the library. This has 
> many benefits from the perspective of library developers, mostly notably 
> because it limits the ABI of the library, making it easier to change the 
> internals without requiring recompilation or breaking changes. Pointers to 
> these structures are translated into Swift code in the form of the 
> OpaquePointer type.
> 
> Older C libraries frequently have non-opaque structures: that is, the 
> structure definition is available in the header files for the library. When 
> using code like 

Re: [swift-evolution] [Concurrency] async/await + actors

[Johannes Weiß via swift-evolution]

> On 11 Sep 2017, at 10:04 pm, Adam Kemp via swift-evolution 
>  wrote:
> 
> 
> 
>> On Sep 11, 2017, at 1:15 PM, Kenny Leung via swift-evolution 
>>  wrote:
>> 
>> I found a decent description about async/await here:
>> 
>> https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/async/
>> 
>> So it’s much more like runloop based callbacks in Foundation that 
>> libdispatch. It’s complicated. Even the simplest example is super 
>> complicated. 
>> 
>> It’s super implicit (like, “I’m going to package up every line of code from 
>> await to the end of this function and turn it into a continuation”). That 
>> seems to go against one of the primary principles of Swift, which is to make 
>> things plain to the reader. I’d be interested to know what the call stack 
>> looks like on the line after await.
> 
> This is pretty much how it would have to work for Swift as well. The call 
> stack after the await (in C#) would either start at the runloop and go 
> through the futures API (usually Task) or it would start at whatever code 
> satisfied the async request.
> 
> It’s true that this can make it more difficult to understand stack traces. In 
> most cases the original call stack is lost. Microsoft has made changes to 
> Visual Studio in order to show kind of an alternative stack trace for tasks 
> to try to make this better.

just FYI, Xcode does that too these days. If you breakpoint/crash within 
something that got asynchronously dispatched, you'll see a synthesised stack 
frame that shows you where it got enqueued from. The same could be done for 
async/await.


> I think they also made things like F10 (step over) and F11 (step out) do the 
> natural thing (i.e., wait for the continuation).
> 
>> 
>> The doc takes away some of the mystery, but leaves major questions, like: 
>> await is used to yield control to the parent, but at the bottom of the call 
>> stack, presumably you’re going to do something blocking, so how do you call 
>> await?
> 
> One of the common misconceptions about async/await (which I also had when I 
> first encountered it) is that there must be a blocking thread somewhere. It 
> doesn’t work that way. The “bottom of the call stack” is typically either a 
> run loop or a thread pool with a work queue (really just another kind of run 
> loop). I guess you’re right in the sense that those kinds of run loops do 
> block, but they’re not blocking on any particular piece of work to be done. 
> They’re blocking waiting for ANY more work to be done (either events or items 
> placed in the work queue).
> 
> The way that the continuation works is that it is placed onto one of those 
> queues. For the UI thread it’s kind of like doing 
> performSelectorOnMainThread: (the .Net equivalent is usually called 
> BeginInvokeOnMainThread). For a thread pool there’s another API. For GCD this 
> would be like doing a dispatch_async. It’s putting the continuation callback 
> block onto a queue, and that callback will be called when the run loop or the 
> thread pool is able to do so.
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Concurrency] async/await + actors

[Johannes Weiß via swift-evolution]

Hi Chris & swift-evo,

(Given the already lengthy thread I tried to separate my points and keep them 
reasonably short to allow people to skip points they don't care about. I'm very 
happy to expand on the points.)

Thanks very much for writing up your thoughts/proposal, I've been waiting to 
see the official kick-off for the concurrency discussions :).

I) Let's start with the async/await proposal. Personally I think this is the 
right direction for Swift given the reality that we need to interface with 
incredibly large existing code-bases and APIs. Further thoughts:

- ❓ GCD: dispatching onto calling queue, how?
GCD doesn't actually allow you to dispatch back to the original queue, so I 
find it unclear how you'd achieve that. IMHO the main reason is that 
conceptually at a given time you can be on more than one queue (nested 
q.sync{}/target queues). So which is 'the' current queue?

- ⊥ first class coroutine model => async & throws should be orthogonal
given that the proposal pitches to be the beginning of a first class coroutine 
model (which I think is great), I think `async` and `throws` do need to be two 
orthogonal concepts. I wouldn't want automatically throwing generators in the 
future ;). Also I think we shouldn't throw spanner in the works of people who 
do like to use Result types to hold the errors or values. I'd be fine 
with async(nothrow) or something though.
- what do we do with functions that invoke their closure multiple times? Like 
DispatchIO.read/write.


II) the actor model part

-  Erlang runtime and the actor model go hand in hand 
I really like the Erlang actor model but I don't think it can be separated from 
Erlang's runtime. The runtime offers green threads (which allow an actor to 
block without blocking an OS thread) and prevents you from sharing memory 
(which makes it possible to kill any actor at any point and still have a 
reliable system). I don't see these two things happening in Swift. To a lesser 
extend these issues are also present in Scala/Akka, the mitigate some of the 
problems by having Akka Streams. Akka Streams are important to establish 
back-pressure if you have faster producers than consumers. Note that we often 
can't control the producer, they might be on the other side of a network 
connection. So it's often very important to not read the available bytes to 
communicate to the kernel that we can't consumes bytes that fast. If we're 
networking with TCP the kernel can then use the TCP flow-control to signal to 
the other side that they better slow down (or else packets will be dropped and 
then need to be resent later).

-  regarding fatal failure in actors
in the server world we need to be able to accept hundreds of thousands 
(millions) of connections at the same time. There are quite a few cases where 
these connections are long-lived and paused for most of the the time. So I 
don't really see the value in introducing a 'reliable' actor model where the 
system stops accepting new connections if one actor fatalError'd and then 
'just' finishes up serving the existing connections. So I believe there are 
only two possible routes: 1) treat it like C/C++ and make sure your code 
doesn't fatalError or the whole process blows up (what we have right now) 2) 
treat it like Erlang and let things die. IMHO Erlang wouldn't be successful if 
actors couldn't just die or couldn't be linked. Linking propagates failures to 
all linked processes. A common thing to do is to 1) spawn a new actor 2) link 
yourself to the newly spawned actor 3) send a message to that actor and at some 
point eventually await a reply message sent by the actor spawned earlier. As 
you mentioned in the writeup it is a problem if the actor doesn't actually 
reply which is why in Erlang you'd link them. The effect is that if the actor 
we spawned dies, any linked actor will die too which will the propagate the 
error to an appropriate place. That allows the programmer to control where an 
error should propagate too. I realise I'm doing a poor job in explaining what 
is best explained by documentation around Erlang: supervision [1] and the 
relationship between what Erlang calls a process (read 'actor') and errors [2].

- ♨️ OS threads and actors
as you mention, the actor model only really works if you can spawn lots of 
them, so it's very important to be able to run hundreds of thousands of them on 
a number of OS threads pretty much equal to your number of cores. That's only 
straightforward if there are no (OS thread) blocking operations or at least 
it's really obvious what blocks and what doesn't. And that's not the case in 
Swift today and with GCD you really feel that pain. GCD does spawn threads for 
you and has a rather arbitrary limit of 64 OS threads (by default on macOS). 
That is too many for a very scalable server application but too few to just 
tolerate blocking APIs.


[1]: 
http://erlang.org/documentation/doc-4.9.1/doc/design_principles/sup_princ.html
[2]: 

Re: [swift-evolution] typed throws

[Johannes Weiß via swift-evolution]

Hi John,

tl;dr I think throws should be optionally typed. Ie. `func someSyscall(...) 
throws(POSIXError) -> Int` should be just as legal as `func 
doSomeFancyCocoaOperation() throws -> Void`.

> [...]
>> Here is where I think things stand on it:
>> - There is consensus that untyped throws is the right thing for a large 
>> scale API like Cocoa.  NSError is effectively proven here.  Even if typed 
>> throws is introduced, Apple is unlikely to adopt it in their APIs for this 
>> reason.
>> - There is consensus that untyped throws is the right default for people to 
>> reach for for public package (#2).
>> - There is consensus that Java and other systems that encourage lists of 
>> throws error types lead to problematic APIs for a variety of reasons.
>> - There is disagreement about whether internal APIs (#3) should use it.  It 
>> seems perfect to be able to write exhaustive catches in this situation, 
>> since everything in knowable. OTOH, this could encourage abuse of error 
>> handling in cases where you really should return an enum instead of using 
>> throws.
>> - Some people are concerned that introducing typed throws would cause people 
>> to reach for it instead of using untyped throws for public package APIs.
> 
> Even for non-public code.  The only practical merit of typed throws I have 
> ever seen someone demonstrate is that it would let them use contextual lookup 
> in a throw or catch.  People always say "I'll be able to exhaustively switch 
> over my errors", and then I ask them to show me where they want to do that, 
> and they show me something that just logs the error, which of course does not 
> require typed throws.  Every. Single.  Time.

I just grep'ed through some code and what I always see is this:

} catch let e {
fatalError("unexcepted error caught: \(e)")
}

most of the code is fairly low-level and the only error thrown in there is a 
POSIXError which basically holds the errno value and a string describing the 
failed syscall. I find this a very common pattern. Also the IMHO quite 
widespread use of Result types shows that there are valid uses of a typed 
error mechanism. On the other hand requiring a high-level Cocoa method to 
declare all errors doesn't seem right either. So why not optionally typing the 
throws?



> [...]

-- Johannes
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Make `errno`-setting functions more usable from Swift

[Johannes Weiß via swift-evolution]

Hi Joe,

>> I just realised, that the problem is slightly worse than I originally 
>> described. I believed that successful calls in between the actual call the 
>> programmer wanted to make and capturing `errno` are not a problem.
>> 
>> But POSIX seems to suggest [4] that "The setting of errno after a successful 
>> call to a function is unspecified unless the description of that function 
>> specifies that errno shall not be modified." . The Linux man page [5] also 
>> mentions that "a function that succeeds is allowed to change errno."
>> 
>> To me this means that the issue is wider than just ARC. I think the problem 
>> extends to memory allocations on the heap. Failed memory allocations aren't 
>> a problem because they are terminal in Swift. However, _successful_ memory 
>> allocations might be a problem because the malloc(3) that the Swift compiler 
>> will use is absolutely free to set errno to 0 (or any other value in fact) 
>> indicating success. (Said that at least malloc doesn't change `errno` on the 
>> macOS or Linux I tested today, we probably shouldn't rely on that though.)
>> 
>> This makes it even more unpredictable to the programmer what a use of 
>> `errno` in Swift will return. IMHO it shouldn't be exported to Swift as its 
>> value is undefined almost(?) everywhere.
>> 
>> [4]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/errno.html
>> [5]: http://man7.org/linux/man-pages/man3/errno.3.html
> 
> Swift's implicit allocations don't directly use malloc/free.

Ah right, that's good to know. But still, "explicit" allocation by creating an 
object does, right?

Like

class Foo {}
let x = Foo()

does allocate through _swift_allocObject_ -> swift_slowAlloc -> malloc .


> If there is a platform where a successful malloc dirties errno, we should 
> avoid doing so in the Swift runtime's entry points.

ok, agreed, should there be a platform where malloc that dirties `errno` it 
could be worked around by saving and restoring the `errno`.

Cheers,
  Johannes
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Pitch] Make `errno`-setting functions more usable from Swift

[Johannes Weiß via swift-evolution]

Hi,

I just realised, that the problem is slightly worse than I originally 
described. I believed that successful calls in between the actual call the 
programmer wanted to make and capturing `errno` are not a problem.

But POSIX seems to suggest [4] that "The setting of errno after a successful 
call to a function is unspecified unless the description of that function 
specifies that errno shall not be modified." . The Linux man page [5] also 
mentions that "a function that succeeds is allowed to change errno."

To me this means that the issue is wider than just ARC. I think the problem 
extends to memory allocations on the heap. Failed memory allocations aren't a 
problem because they are terminal in Swift. However, _successful_ memory 
allocations might be a problem because the malloc(3) that the Swift compiler 
will use is absolutely free to set errno to 0 (or any other value in fact) 
indicating success. (Said that at least malloc doesn't change `errno` on the 
macOS or Linux I tested today, we probably shouldn't rely on that though.)

This makes it even more unpredictable to the programmer what a use of `errno` 
in Swift will return. IMHO it shouldn't be exported to Swift as its value is 
undefined almost(?) everywhere.

[4]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/errno.html
[5]: http://man7.org/linux/man-pages/man3/errno.3.html

All the best,
  Johannes

> On 2 Nov 2016, at 1:12 pm, Johannes Weiß via swift-evolution 
> <swift-evolution@swift.org> wrote:
> 
> Hey swift-evolution,
> 
> First of all apologies, this is not a full proposal yet, it's meant to kick 
> off a discussion on how to resolve the issue.
> 
> # Make `errno`-setting functions more usable from Swift
> 
> ## Introduction
> 
> This is a pitch to make [`errno`][1]-setting functions properly usable, as in 
> having a guarantee to get the correct `errno` value on failure of a [system 
> call][2]. Currently, functions which set `errno` are just exported in the 
> Darwin/Glibc modules with (as far as I understand) no guaranteed correct way 
> of handling errors as the correct `errno` value can't be retrieved.
> This means that much of the Swift code which uses Darwin/Glibc out there 
> relies on behaviour that isn't guaranteed.
> 
> 
> ## Motivation
> 
> In many Swift libraries that use the Darwin/Glibc modules there is code 
> similar to:
> 
> ```
> /* import Darwin/Glibc */
> 
> let rv = some_system_call(some, parameters)
> if rv < 0 {
>   throw SomeError(errorCode: errno) /* <-- errno use */
> }
> ```
> 
> That looks very innocent but please note that `errno` is used here. And 
> `errno` is an interesting one as it's a thread-local variable which is 
> written to by many functions. A thread-local variable is like a global 
> variable except that setting it in one thread does not affect its value in 
> any other thread. Pretty much all system calls and many library functions set 
> `errno` if something went wrong.
> 
> The problem is that as far as I see (and Swift developers have confirmed), 
> there is no guarantee that in between the call of `some_system_call` and the 
> reading of `errno`, `errno` hasn't been overwritten by some other system call 
> that has been call on the same thread.
> 
> To illustrate this further, let's consider this example
> 
> ```
> /* import Darwin/Glibc */
> public class SomeClass {
>  public let someValue: Int = 1
>  deinit {
>  /* call some failing syscall, for example */
>  write(-1, nil, 0) /* should set errno to EBADF */
>  }
> }
> 
> public func foo() {
>   let x = SomeClass()
>   let rv = write(x.someValue, nil, 0)
>   let errnoSave = errno
>   if rv != 0 {
>  throw SomeError(errorCode: errnoSave)
>   }
> }
> ```
> 
> as you see in function `foo`, the instance `x` of `SomeClass` isn't needed 
> anymore as soon as `write` has been called. So (as far as I understand) 
> there's no guarantee that ARC doesn't turn the above code into
> 
> ```
> let x = SomeClass()
> let rv = write(x.someValue, nil, 42) /* should set errno to EFAULT */
> /* ARC generated */ x.release()
> let errnoSave = errno /* wrong errno value :( */
> if rv != 0 {
>  throw SomeError(errorCode: errnoSave)
> }
> ```
> 
> And the ARC generated `x.release()` will cause `x` to be deallocated which 
> will call the failing `write` in the `deinit` of `SomeClass`. So `errnoSave` 
> might be `EBADF` instead of `EFAULT` depending on where ARC put the 
> `x.release()` call.
> 
> What `errno` value we read will depend on the optimisation settings and the 
> Swift compiler version. That's IMHO a big issue as it might make the lowest 
> layers unstable with hard-to-debug issues.
> 
> 
> ## Proposed solution

Re: [swift-evolution] [Pitch] Make `errno`-setting functions more usable from Swift

[Johannes Weiß via swift-evolution]

Hi John,

> [...]
>> This is a pitch to make [`errno`][1]-setting functions properly usable, as 
>> in having a guarantee to get the correct `errno` value on failure of a 
>> [system call][2]. Currently, functions which set `errno` are just exported 
>> in the Darwin/Glibc modules with (as far as I understand) no guaranteed 
>> correct way of handling errors as the correct `errno` value can't be 
>> retrieved.
>> This means that much of the Swift code which uses Darwin/Glibc out there 
>> relies on behaviour that isn't guaranteed.
> 
> Is this a general enough problem to be worth complicating the importer with 
> vs., say, just manually wrapping the POSIX APIs to be throwing in an overlay? 
>  POSIX uses a lot of different conventions for reporting that an error 
> occurred, and there's only a finite set of POSIX APIs. Are there "user" 
> functions that wrap POSIX and carefully report errors with errno?

That's a very good point. I'm sure there are "user libraries" out there which 
set errno but I'd hope that it's a fairly limited set. So I think an overlay 
that ships with Swift and makes the underlying module unavailable would be fine 
by me. Something like the Dispatch module (which uses the "hidden" CDispatch 
module internally) sounds reasonable to me.

The reason I proposed to include that into the clang importer is because the 
Darwin/Glibc modules contain hundreds (I haven't counted) of functions (syscall 
and library functions) that set errno. And I assumed that if it can be done 
automatically we could leverage that.


> [...]

Thanks,
  Johannes

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[swift-evolution] [Pitch] Make `errno`-setting functions more usable from Swift

[Johannes Weiß via swift-evolution]

Hey swift-evolution,

First of all apologies, this is not a full proposal yet, it's meant to kick off 
a discussion on how to resolve the issue.

# Make `errno`-setting functions more usable from Swift

## Introduction

This is a pitch to make [`errno`][1]-setting functions properly usable, as in 
having a guarantee to get the correct `errno` value on failure of a [system 
call][2]. Currently, functions which set `errno` are just exported in the 
Darwin/Glibc modules with (as far as I understand) no guaranteed correct way of 
handling errors as the correct `errno` value can't be retrieved.
This means that much of the Swift code which uses Darwin/Glibc out there relies 
on behaviour that isn't guaranteed.


## Motivation

In many Swift libraries that use the Darwin/Glibc modules there is code similar 
to:

```
/* import Darwin/Glibc */

let rv = some_system_call(some, parameters)
if rv < 0 {
   throw SomeError(errorCode: errno) /* <-- errno use */
}
```

That looks very innocent but please note that `errno` is used here. And `errno` 
is an interesting one as it's a thread-local variable which is written to by 
many functions. A thread-local variable is like a global variable except that 
setting it in one thread does not affect its value in any other thread. Pretty 
much all system calls and many library functions set `errno` if something went 
wrong.

The problem is that as far as I see (and Swift developers have confirmed), 
there is no guarantee that in between the call of `some_system_call` and the 
reading of `errno`, `errno` hasn't been overwritten by some other system call 
that has been call on the same thread.

To illustrate this further, let's consider this example

```
/* import Darwin/Glibc */
public class SomeClass {
  public let someValue: Int = 1
  deinit {
  /* call some failing syscall, for example */
  write(-1, nil, 0) /* should set errno to EBADF */
  }
}

public func foo() {
   let x = SomeClass()
   let rv = write(x.someValue, nil, 0)
   let errnoSave = errno
   if rv != 0 {
  throw SomeError(errorCode: errnoSave)
   }
}
```

as you see in function `foo`, the instance `x` of `SomeClass` isn't needed 
anymore as soon as `write` has been called. So (as far as I understand) there's 
no guarantee that ARC doesn't turn the above code into

```
let x = SomeClass()
let rv = write(x.someValue, nil, 42) /* should set errno to EFAULT */
/* ARC generated */ x.release()
let errnoSave = errno /* wrong errno value :( */
if rv != 0 {
  throw SomeError(errorCode: errnoSave)
}
```

And the ARC generated `x.release()` will cause `x` to be deallocated which will 
call the failing `write` in the `deinit` of `SomeClass`. So `errnoSave` might 
be `EBADF` instead of `EFAULT` depending on where ARC put the `x.release()` 
call.

What `errno` value we read will depend on the optimisation settings and the 
Swift compiler version. That's IMHO a big issue as it might make the lowest 
layers unstable with hard-to-debug issues.


## Proposed solution

I don't have a full story on how to actually resolve the issue but I see a few 
options:

### Option 1: always return errno

clang importer could be changed to make all `errno`-setting functions return a 
tuple of the actual return value and the `errno` value.

For example, currently write(2) is imported as:

```
public func write(_ __fd: Int32, _ __buf: UnsafeRawPointer!, _ __nbyte: Int) -> 
Int
```

which could be changed to

```
 public func write(_ __fd: Int32, _ __buf: UnsafeRawPointer!, _ __nbyte: Int) 
-> (Int, Int32 /* for errno */)
```

Correct code to use write would then look like this:

```
let (bytesWritten, writeErrno) = write(fd, buf, len)
if bytesWritten >= 0 {
   /* everything's fine */
} else {
   throw POSIXError(code: writeErrno)
}
```


### Option 2: make them throw

The second option is to teach clang importer to make the functions throwing. So 
write(2) would be imported as

```
public func write(_ __fd: Int32, _ __buf: UnsafeRawPointer!, _ __nbyte: Int) 
throws /* POSIXError */ -> Int
```

That would make these functions quite easy to use and would feel natural in 
Swift:

```
do {
   let bytesWritten = write(fd, buf, len)
} catch let e as POSIXError {
   /* handle error */
} catch {
   ...
}
```


### Discussion

The beauty of option 1 is simplicity. Clang importer would not need to know 
what exact values a system call returns on failure. Also very little additional 
code needs to be emitted for calling a system call. That seems to be the [way 
Go is going][3].

The downside of option 1 is that the API doesn't feel like idiomatic Swift. The 
returned `errno` value is only useful if the system call failed and is 
arbitrary in the case when it worked. (There is no guarantee that `errno` is 
set to `0` when a system call succeeds.)
Also there is a slight overhead in reading `errno` which would be paid for 
every `errno`-setting function, even if successful. Hence, option 2 looks nice 
as it brings these functions more in like with 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Johannes Weiß via swift-evolution]

Hi Eloy,

> > [...]
>> I have no idea how well it works but if we'll end up relying on proper 
>> semantic versioning, tool support sounds like a good idea to me.
> 
> This is what I was referring to when I mentioned that automation can only 
> take you so far. It is easily possible to do a patch release where the API 
> might not change, but the semantics of the code does.
> 
> In my opinion it requires human judgement to determine if a change is really 
> something you can trust. Trusting SemVer for that is going to lead to 
> problems and making people think that they can is just misleading in my book.
> 
> Not saying you can’t have tools to help guide choosing versions, though.

agreed. But I think that if (whether that's good or bad) we rely on semantic 
versioning, tool support can make that a lot easier.

Cheers,
  Johannes
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Johannes Weiß via swift-evolution]

Hey,

> [...]
> I see it as my responsibility to know exactly what code I’m pulling into my 
> package. In my view, it’s absolutely unsafe to trust other people’s code. 
> Even when they mean no harm, trusting them to properly apply SemVer is the 
> same issue.

maybe we should have the tooling support that? Elm does try to enforce correct 
semantic versioning. Maybe swift-pm should do that too?

See http://elm-lang.org :

Enforced Semantic Versioning

Elm can detect all API changes automatically thanks to its type system. We use 
that information to force everything in our package catalog to follow semantic 
versioning precisely. No more surprises in PATCH releases!


I have no idea how well it works but if we'll end up relying on proper semantic 
versioning, tool support sounds like a good idea to me.


> [...]

Cheers,
  Johannes
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution