Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Martin Waitz via swift-evolution]

Hi :)

> Daniel, it's all right, "There are only two hard things in Computer Science: 
> cache invalidation and naming things.“

That’s true :-).

So I throw another name into the ring: „Package.versions“.

I think the file should be named after what it contains: the versions of all 
dependencies.
This is used to get reproducible builds by always using the same pinned version.
But that is how the file is _used_, not what the file _is_.

— Martin___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Martin Waitz via swift-evolution]

Hi,

> Suppose we had a semantic notion of which packages were intended to be 
> "top-level" versus used as a dependency, and we chose our defaults 
> accordingly (in this case, we would orient workflows towards pinning by 
> default in the top-level case, in the used as a dependency case we would 
> orient away from it, e.g. warning you if you checked it in). What would you 
> think of such a design?

What is wrong with checking in the versions of your dependencies even if you 
are not a top-level package?
That’s a great way to document which versions do work together and may be 
helpful for the maintainer of the top-level package which is depending on you 
(he can easily compare the version files to find problems).

I think we agree that `swift package` within the top-level project should not 
look at the Package version files of its dependencies.
So these files will not change anything for the top-level builds.

I really think that the easiest solution is to always store the versions of all 
dependencies in SCM.
Then have some `swift package update` to move all versions to the latest 
consistent set and we have everything we need.
Each package can be built reproducibly and can be managed independently.

If packages want to track their dependencies closely, then simply set up some 
CI which does the update automatically.
You could automatically create create ‚update‘-commits, test them and 
automatically merge them if you really want to.
But every update is documented and you can even bisect bugs which were 
introduced by the updates.
This is helpful both for top-level as well as for library projects.

— Martin
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

Thanks everyone who participated in this discussion.

We took the feedback on this thread and went back and created a revised 
proposal, here:
  
https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md
which we are putting up for review today:
  
https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161031/028579.html

Ultimately, we decided to not actually change any of the proposed behavior, 
*for now*, because we couldn't come to a consensus about how that should look.

What we *did* do is try and write up much more about (a) the current behavior, 
and (b) the motivation behind the decisions (for example, the implications of 
not being able to include multiple versions of the same package in one build).

What we also agreed on was that the main thing we need right now is to 
implement the *mechanisms* in the proposal, and we didn't want the debate over 
the policy decision (to create the pin file by default) to side track that. So 
we decided to continue with the proposal as is, but to also reevaluate the 
impact of the policy decision in the proposal, in time to adjust it for Swift 4 
if we decide the way it has played out in the ecosystem so far isn't as 
intended.

We also agreed there was room for a workflow component around specifying the 
"style" of package being developed (final application versus something being 
primarily developed for use as a dependency), but we didn't want to block 
implementation of the mechanism on figuring out the right design for that. If 
we get a design for that, we will also probably use that to change the default 
policy here.

Please take a look at the new proposal, and see what you think.

Thanks,
 - Daniel

> On Oct 14, 2016, at 4:42 PM, Daniel Dunbar via swift-evolution 
>  wrote:
> 
> 
>> On Oct 14, 2016, at 4:02 PM, Paul Cantrell  wrote:
>> 
>> Sorry for the late arrival to this thread. Comments below…
>> 
>>> On Oct 14, 2016, at 3:09 PM, Daniel Dunbar via swift-build-dev 
>>>  wrote:
>>> 
>>> What this proposal about is in one sense being able to export and share 
>>> those pins.
>> 
>> This is a crucial and clarifying insight. It should be in the proposal! Near 
>> the top.
> 
> Good idea, will incorporate it.
> 
>>> 2. Huon, Alexis, and I all agree we should never *inherit* pins by default.
>> 
>> Indeed. Pins should be only be about sharing specific versions within a 
>> development team — not with client packages / apps. What’s pinned in Vegas 
>> stays in Vegas. Publishing pins to other projects would be nonsensical.
>> 
>>> 5. Given that many people agree there are two workflows (we ourselves had 
>>> talked about this a lot when writing the proposal, but didn't put it in), 
>>> we felt it makes sense to consider adding that as an explicit declaration 
>>> *somewhere*.
>>> 
>>> @Eloy, @Orta: Suppose we had a semantic notion of which packages were 
>>> intended to be "top-level" versus used as a dependency, and we chose our 
>>> defaults accordingly (in this case, we would orient workflows towards 
>>> pinning by default in the top-level case, in the used as a dependency case 
>>> we would orient away from it, e.g. warning you if you checked it in). What 
>>> would you think of such a design?
>> 
>> I’m puzzled. If a package’s pinning does not affect any other package that 
>> uses it, why should the defaults be different? A library will still suffer 
>> from all the “works for me” problems an app might.
>> 
>> Is the rationale that not pinning libraries encourages accidental testing of 
>> new versions of a library’s dependencies as they arrive? Or is there another 
>> rationale for having different defaults?
> 
> I'll defer to this comment (linked from someone else earlier in the thread), 
> which happens to match up with my perspective:
>  https://github.com/yarnpkg/yarn/issues/838#issuecomment-253362537
> 
> - Daniel
> 
>> 
>> Cheers, P
>> 
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Marcin Krzyzanowski via swift-evolution]

Daniel, it's all right, "There are only two hard things in Computer
Science: cache invalidation and *naming things*."


--
Marcin

On Fri, Oct 14, 2016 at 6:25 PM, Daniel Dunbar via swift-build-dev <
swift-build-...@swift.org> wrote:

> Let's not change this thread into a discussion of English. :)
>
>  - Daniel
>
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Paul Cantrell via swift-evolution]

> On Oct 14, 2016, at 7:18 PM, Daniel Dunbar  wrote:
> 
>> 
>> On Oct 14, 2016, at 4:51 PM, Paul Cantrell via swift-evolution 
>>  wrote:
>> 
>> That’s clearly a bigger, separate idea, not necessary to hash out right now. 
>> I mean it just to illustrate what better approaches might look like. I’m 
>> skeptical that simply disabling pinning does a good job of solving the 
>> intended problem, and don’t think it should weigh quite so heavily on the 
>> proposal at hand.
> 
> The current idea wouldn't "disable it”

Right, my bad wording. Is “turn off / discourage pinfiles by default for 
libraries but not apps” a better description of the general idea?

> it would discourage you from checking it in for a library (it really comes 
> down to you have to run two commands, not one). I agree all the other things 
> you outline are useful, and that not checking it in doesn't magically solve 
> the problem here.


If the difference were only what’s in .gitignore, I’d be completely comfortable 
with that. Enthusiastic.

And if there’s some distinction between libs and top-level packages that only 
affects a generated .gitignore and/or emitted warnings, I’d be completely 
comfortable with that.

I’m skeptical of deeper special-casing for libs vs. top-level, but it sounds 
like the special-casing may not actually be that deep. If so, I’m just fussing 
over nothing!

> On Oct 14, 2016, at 7:17 PM, Alexis Beingessner  
> wrote:
> 
> A few comments down, Yehuda even provides an example of him doing just that 
> with Bundler:
> 
> https://github.com/yarnpkg/yarn/issues/838#issuecomment-253366352 
> 
Yeah, I’d love to see SwiftPM or a companion tool automate that. Totally 
tractable problem, and I’ll bet we see quality payoffs in the lib ecosystem.

Cheers,

Paul

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

> On Oct 14, 2016, at 4:51 PM, Paul Cantrell via swift-evolution 
>  wrote:
> 
> 
> 
>> On Oct 14, 2016, at 6:34 PM, Eloy Durán  wrote:
>> 
>>> I’m puzzled. If a package’s pinning does not affect any other package that 
>>> uses it, why should the defaults be different? A library will still suffer 
>>> from all the “works for me” problems an app might.
>>> 
>>> Is the rationale that not pinning libraries encourages accidental testing 
>>> of new versions of a library’s dependencies as they arrive?
>> 
>> Yep.
> 
> I’m skeptical of whether disabling pinning by default is the right answer. It 
> seems … haphazard and incomplete, likely to confuse those who don’t 
> understand this reasoning, and only incidentally helpful to those who do.
> 
> I’d prefer an approach that is more mindful, more intentional.
> 
> Emitting warnings when newer versions of packages are available would help 
> address the same issue. This would also encourage apps to get security 
> patches promptly.
> 
> A more ambitious approach might be to provide a utility that installs and 
> tests various combinations of all the semantically allowed versions of a 
> lib’s dependencies — at the very least, all the lowest and highest allowed 
> versions. This would ignore any existing pinfile. Library devs would run this 
> prior to release, and on a CI server if they have one.
> 
> This would have the advantage of catching incompatibilities with _past_ 
> versions of libraries as well as _new_ ones. In particular, it would catch 
> too-permissive version specifications, e.g. if your library asks for foolib 
> 3.x but relies on a bugfix in 3.4.2, this approach would catch that problem.
> 
> That’s clearly a bigger, separate idea, not necessary to hash out right now. 
> I mean it just to illustrate what better approaches might look like. I’m 
> skeptical that simply disabling pinning does a good job of solving the 
> intended problem, and don’t think it should weigh quite so heavily on the 
> proposal at hand.

The current idea wouldn't "disable it", it would discourage you from checking 
it in for a library (it really comes down to you have to run two commands, not 
one). I agree all the other things you outline are useful, and that not 
checking it in doesn't magically solve the problem here.

 - Daniel

> 
> Cheers,
> 
> Paul
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.swift.org_mailman_listinfo_swift-2Devolution=CwIGaQ=Hw-EJUFt2_D9PK5csBJ29kRV40HqSDXWTLPyZ6W8u84=teZCCTj9bCvNJ4SX_xKQ4SOVlaMWK9IXMlx_HMapKUU=wl9y8ecLDwSW83LjsOje4Jz1_TjqamVYnv-lTnVooiI=ICIfe-G4A6ZgJyiYYtMnHpq6JKQpHA5nhxc8nG4GxhY=
>  

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Alexis Beingessner via swift-evolution]

> On Oct 14, 2016, at 8:00 PM, Paul Cantrell  wrote:
> 
> 
>>> On Oct 14, 2016, at 6:42 PM, Daniel Dunbar  wrote:
>>> 
>>> On Oct 14, 2016, at 4:02 PM, Paul Cantrell  wrote:
>>> 
>>> I’m puzzled. If a package’s pinning does not affect any other package that 
>>> uses it, why should the defaults be different? A library will still suffer 
>>> from all the “works for me” problems an app might.
>>> 
>>> Is the rationale that not pinning libraries encourages accidental testing 
>>> of new versions of a library’s dependencies as they arrive? Or is there 
>>> another rationale for having different defaults?
>> 
>> I'll defer to this comment (linked from someone else earlier in the thread), 
>> which happens to match up with my perspective:
>> https://github.com/yarnpkg/yarn/issues/838#issuecomment-253362537
> 
> I took that comment to be an explanation of why a library's lockfile/pinfile 
> should not propagate to other packages that use it. That is clearly the case; 
> such pin propagation would be nonsensical.
> 
> My question was not about that, but about why libraries shouldn’t use a 
> pinfile at all, even for their own _internal_ development. All the same “last 
> know good build” concerns apply.
> 
> The difference is that testing against that single last known good version 
> set is sufficient for a top-level package, whereas a library should (1) 
> ideally test against multiple valid dependency versions and (2) test often 
> against new versions of its dependencies. I don’t see, however, that this 
> implies that libraries should not have pinfiles at all — just that their 
> release / CI process should not be limited to what’s pinned.

A few comments down, Yehuda even provides an example of him doing just that 
with Bundler:

https://github.com/yarnpkg/yarn/issues/838#issuecomment-253366352

But in this case you actually want to maintain *many* lock files, and so it 
seems fine to require a bit of extra work (passing some flags) to do this. 
Drifting tests are the better default here. It makes library CI into an 
alpha-tester, empowering binaries to be more confident in upgrading frequently.

> 
> Cheers, P
> 
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Paul Cantrell via swift-evolution]

> On Oct 14, 2016, at 6:42 PM, Daniel Dunbar  wrote:
> 
>> On Oct 14, 2016, at 4:02 PM, Paul Cantrell  wrote:
>> 
>> I’m puzzled. If a package’s pinning does not affect any other package that 
>> uses it, why should the defaults be different? A library will still suffer 
>> from all the “works for me” problems an app might.
>> 
>> Is the rationale that not pinning libraries encourages accidental testing of 
>> new versions of a library’s dependencies as they arrive? Or is there another 
>> rationale for having different defaults?
> 
> I'll defer to this comment (linked from someone else earlier in the thread), 
> which happens to match up with my perspective:
>  https://github.com/yarnpkg/yarn/issues/838#issuecomment-253362537

I took that comment to be an explanation of why a library's lockfile/pinfile 
should not propagate to other packages that use it. That is clearly the case; 
such pin propagation would be nonsensical.

My question was not about that, but about why libraries shouldn’t use a pinfile 
at all, even for their own _internal_ development. All the same “last know good 
build” concerns apply.

The difference is that testing against that single last known good version set 
is sufficient for a top-level package, whereas a library should (1) ideally 
test against multiple valid dependency versions and (2) test often against new 
versions of its dependencies. I don’t see, however, that this implies that 
libraries should not have pinfiles at all — just that their release / CI 
process should not be limited to what’s pinned.

Cheers, P

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Eloy Durán via swift-evolution]

>> Not saying you can’t have tools to help guide choosing versions, though.
> 
> agreed. But I think that if (whether that's good or bad) we rely on semantic 
> versioning, tool support can make that a lot easier.

Aye, agreed.
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Paul Cantrell via swift-evolution]

> On Oct 14, 2016, at 6:34 PM, Eloy Durán  wrote:
> 
>> I’m puzzled. If a package’s pinning does not affect any other package that 
>> uses it, why should the defaults be different? A library will still suffer 
>> from all the “works for me” problems an app might.
>> 
>> Is the rationale that not pinning libraries encourages accidental testing of 
>> new versions of a library’s dependencies as they arrive?
> 
> Yep.

I’m skeptical of whether disabling pinning by default is the right answer. It 
seems … haphazard and incomplete, likely to confuse those who don’t understand 
this reasoning, and only incidentally helpful to those who do.

I’d prefer an approach that is more mindful, more intentional.

Emitting warnings when newer versions of packages are available would help 
address the same issue. This would also encourage apps to get security patches 
promptly.

A more ambitious approach might be to provide a utility that installs and tests 
various combinations of all the semantically allowed versions of a lib’s 
dependencies — at the very least, all the lowest and highest allowed versions. 
This would ignore any existing pinfile. Library devs would run this prior to 
release, and on a CI server if they have one.

This would have the advantage of catching incompatibilities with _past_ 
versions of libraries as well as _new_ ones. In particular, it would catch 
too-permissive version specifications, e.g. if your library asks for foolib 3.x 
but relies on a bugfix in 3.4.2, this approach would catch that problem.

That’s clearly a bigger, separate idea, not necessary to hash out right now. I 
mean it just to illustrate what better approaches might look like. I’m 
skeptical that simply disabling pinning does a good job of solving the intended 
problem, and don’t think it should weigh quite so heavily on the proposal at 
hand.

Cheers,

Paul

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Johannes Weiß via swift-evolution]

Hi Eloy,

> > [...]
>> I have no idea how well it works but if we'll end up relying on proper 
>> semantic versioning, tool support sounds like a good idea to me.
> 
> This is what I was referring to when I mentioned that automation can only 
> take you so far. It is easily possible to do a patch release where the API 
> might not change, but the semantics of the code does.
> 
> In my opinion it requires human judgement to determine if a change is really 
> something you can trust. Trusting SemVer for that is going to lead to 
> problems and making people think that they can is just misleading in my book.
> 
> Not saying you can’t have tools to help guide choosing versions, though.

agreed. But I think that if (whether that's good or bad) we rely on semantic 
versioning, tool support can make that a lot easier.

Cheers,
  Johannes
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

> On Oct 14, 2016, at 4:15 PM, Johannes Weiß  wrote:
> 
> Hey,
> 
>> [...]
>> I see it as my responsibility to know exactly what code I’m pulling into my 
>> package. In my view, it’s absolutely unsafe to trust other people’s code. 
>> Even when they mean no harm, trusting them to properly apply SemVer is the 
>> same issue.
> 
> maybe we should have the tooling support that? Elm does try to enforce 
> correct semantic versioning. Maybe swift-pm should do that too?

We would like to (try to), and it is on the long list of ideal future things to 
do. It requires a lot of compiler support no one has signed up for yet, 
though...

 - Daniel

> See http://elm-lang.org :
> 
> Enforced Semantic Versioning
> 
> Elm can detect all API changes automatically thanks to its type system. We 
> use that information to force everything in our package catalog to follow 
> semantic versioning precisely. No more surprises in PATCH releases!
> 
> 
> I have no idea how well it works but if we'll end up relying on proper 
> semantic versioning, tool support sounds like a good idea to me.
> 
> 
>> [...]
> 
> Cheers,
>  Johannes

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Eloy Durán via swift-evolution]

>> [...]
>> I see it as my responsibility to know exactly what code I’m pulling into my 
>> package. In my view, it’s absolutely unsafe to trust other people’s code. 
>> Even when they mean no harm, trusting them to properly apply SemVer is the 
>> same issue.
> 
> maybe we should have the tooling support that? Elm does try to enforce 
> correct semantic versioning. Maybe swift-pm should do that too?
> 
> See http://elm-lang.org :
> 
> Enforced Semantic Versioning
> 
> Elm can detect all API changes automatically thanks to its type system. We 
> use that information to force everything in our package catalog to follow 
> semantic versioning precisely. No more surprises in PATCH releases!
> 
> 
> I have no idea how well it works but if we'll end up relying on proper 
> semantic versioning, tool support sounds like a good idea to me.

This is what I was referring to when I mentioned that automation can only take 
you so far. It is easily possible to do a patch release where the API might not 
change, but the semantics of the code does.

In my opinion it requires human judgement to determine if a change is really 
something you can trust. Trusting SemVer for that is going to lead to problems 
and making people think that they can is just misleading in my book.

Not saying you can’t have tools to help guide choosing versions, though.
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

> On Oct 14, 2016, at 4:02 PM, Paul Cantrell  wrote:
> 
> Sorry for the late arrival to this thread. Comments below…
> 
>> On Oct 14, 2016, at 3:09 PM, Daniel Dunbar via swift-build-dev 
>>  wrote:
>> 
>> What this proposal about is in one sense being able to export and share 
>> those pins.
> 
> This is a crucial and clarifying insight. It should be in the proposal! Near 
> the top.

Good idea, will incorporate it.

>> 2. Huon, Alexis, and I all agree we should never *inherit* pins by default.
> 
> Indeed. Pins should be only be about sharing specific versions within a 
> development team — not with client packages / apps. What’s pinned in Vegas 
> stays in Vegas. Publishing pins to other projects would be nonsensical.
> 
>> 5. Given that many people agree there are two workflows (we ourselves had 
>> talked about this a lot when writing the proposal, but didn't put it in), we 
>> felt it makes sense to consider adding that as an explicit declaration 
>> *somewhere*.
>> 
>> @Eloy, @Orta: Suppose we had a semantic notion of which packages were 
>> intended to be "top-level" versus used as a dependency, and we chose our 
>> defaults accordingly (in this case, we would orient workflows towards 
>> pinning by default in the top-level case, in the used as a dependency case 
>> we would orient away from it, e.g. warning you if you checked it in). What 
>> would you think of such a design?
> 
> I’m puzzled. If a package’s pinning does not affect any other package that 
> uses it, why should the defaults be different? A library will still suffer 
> from all the “works for me” problems an app might.
> 
> Is the rationale that not pinning libraries encourages accidental testing of 
> new versions of a library’s dependencies as they arrive? Or is there another 
> rationale for having different defaults?

I'll defer to this comment (linked from someone else earlier in the thread), 
which happens to match up with my perspective:
  https://github.com/yarnpkg/yarn/issues/838#issuecomment-253362537

 - Daniel

> 
> Cheers, P
> 

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Eloy Durán via swift-evolution]

> I’m puzzled. If a package’s pinning does not affect any other package that 
> uses it, why should the defaults be different? A library will still suffer 
> from all the “works for me” problems an app might.
> 
> Is the rationale that not pinning libraries encourages accidental testing of 
> new versions of a library’s dependencies as they arrive?

Yep.___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Johannes Weiß via swift-evolution]

Hey,

> [...]
> I see it as my responsibility to know exactly what code I’m pulling into my 
> package. In my view, it’s absolutely unsafe to trust other people’s code. 
> Even when they mean no harm, trusting them to properly apply SemVer is the 
> same issue.

maybe we should have the tooling support that? Elm does try to enforce correct 
semantic versioning. Maybe swift-pm should do that too?

See http://elm-lang.org :

Enforced Semantic Versioning

Elm can detect all API changes automatically thanks to its type system. We use 
that information to force everything in our package catalog to follow semantic 
versioning precisely. No more surprises in PATCH releases!


I have no idea how well it works but if we'll end up relying on proper semantic 
versioning, tool support sounds like a good idea to me.


> [...]

Cheers,
  Johannes
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Paul Cantrell via swift-evolution]

Sorry for the late arrival to this thread. Comments below…

> On Oct 14, 2016, at 3:09 PM, Daniel Dunbar via swift-build-dev 
>  wrote:
> 
> What this proposal about is in one sense being able to export and share those 
> pins.

This is a crucial and clarifying insight. It should be in the proposal! Near 
the top.

> 2. Huon, Alexis, and I all agree we should never *inherit* pins by default.

Indeed. Pins should be only be about sharing specific versions within a 
development team — not with client packages / apps. What’s pinned in Vegas 
stays in Vegas. Publishing pins to other projects would be nonsensical.

> 5. Given that many people agree there are two workflows (we ourselves had 
> talked about this a lot when writing the proposal, but didn't put it in), we 
> felt it makes sense to consider adding that as an explicit declaration 
> *somewhere*.
> 
> @Eloy, @Orta: Suppose we had a semantic notion of which packages were 
> intended to be "top-level" versus used as a dependency, and we chose our 
> defaults accordingly (in this case, we would orient workflows towards pinning 
> by default in the top-level case, in the used as a dependency case we would 
> orient away from it, e.g. warning you if you checked it in). What would you 
> think of such a design?

I’m puzzled. If a package’s pinning does not affect any other package that uses 
it, why should the defaults be different? A library will still suffer from all 
the “works for me” problems an app might.

Is the rationale that not pinning libraries encourages accidental testing of 
new versions of a library’s dependencies as they arrive? Or is there another 
rationale for having different defaults?

Cheers, P

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

Ok, great! We will discuss this some more and see what we can come up with.

 - Daniel

> On Oct 14, 2016, at 2:18 PM, orta therox  wrote:
> 
> Same, yeah 
> 
>> On 14 Oct 2016, at 21:21, Eloy Durán via swift-build-dev 
>>  wrote:
>> 
>>> 5. Given that many people agree there are two workflows (we ourselves had 
>>> talked about this a lot when writing the proposal, but didn't put it in), 
>>> we felt it makes sense to consider adding that as an explicit declaration 
>>> *somewhere*.
>>> 
>>> @Eloy, @Orta: Suppose we had a semantic notion of which packages were 
>>> intended to be "top-level" versus used as a dependency, and we chose our 
>>> defaults accordingly (in this case, we would orient workflows towards 
>>> pinning by default in the top-level case, in the used as a dependency case 
>>> we would orient away from it, e.g. warning you if you checked it in). What 
>>> would you think of such a design?
>> 
>> Oooh, I like it.
>> 
>> Even though I would probably still pin/lock personally, I think this default 
>> combined with the explicit declaration strikes a perfect balance in 
>> trade-offs 
>> ___
>> swift-build-dev mailing list
>> swift-build-...@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-build-dev
> 

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[orta therox via swift-evolution]

Same, yeah 

> On 14 Oct 2016, at 21:21, Eloy Durán via swift-build-dev 
>  wrote:
> 
>> 5. Given that many people agree there are two workflows (we ourselves had 
>> talked about this a lot when writing the proposal, but didn't put it in), we 
>> felt it makes sense to consider adding that as an explicit declaration 
>> *somewhere*.
>> 
>> @Eloy, @Orta: Suppose we had a semantic notion of which packages were 
>> intended to be "top-level" versus used as a dependency, and we chose our 
>> defaults accordingly (in this case, we would orient workflows towards 
>> pinning by default in the top-level case, in the used as a dependency case 
>> we would orient away from it, e.g. warning you if you checked it in). What 
>> would you think of such a design?
> 
> Oooh, I like it.
> 
> Even though I would probably still pin/lock personally, I think this default 
> combined with the explicit declaration strikes a perfect balance in 
> trade-offs 
> ___
> swift-build-dev mailing list
> swift-build-...@swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Eloy Durán via swift-evolution]

> 5. Given that many people agree there are two workflows (we ourselves had 
> talked about this a lot when writing the proposal, but didn't put it in), we 
> felt it makes sense to consider adding that as an explicit declaration 
> *somewhere*.
> 
> @Eloy, @Orta: Suppose we had a semantic notion of which packages were 
> intended to be "top-level" versus used as a dependency, and we chose our 
> defaults accordingly (in this case, we would orient workflows towards pinning 
> by default in the top-level case, in the used as a dependency case we would 
> orient away from it, e.g. warning you if you checked it in). What would you 
> think of such a design?

Oooh, I like it.

Even though I would probably still pin/lock personally, I think this default 
combined with the explicit declaration strikes a perfect balance in trade-offs 
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

I'm replying here as a proxy for a bunch of other messages, since it is the 
latest one when I started typing. :)

> On Oct 14, 2016, at 12:40 PM, Ryan Lovelett  
> wrote:
> 
>> I particularly like Yehuda Katz's explanation on why it works this way: 
>> https://github.com/yarnpkg/yarn/issues/838#issuecomment-253362537 
>>  
> 
> This was a powerful description for me. @Rob thank you for providing that!
> 
> The articulation of the purpose/philosophy behind the whole thing was really 
> clear and made sense to me. Even though I've never used Rust or Cargo. Nor 
> have I ever run into any problems with npm such that I would prefer yarn over 
> it. I've used Bundler for years and love it.
> Yehuda Katz has been involved in making 3 different package managers for 3 
> different communities/languages over at least a 6-year span. It feels trite 
> but it seems likely he has knowledge/experience we might not. It might be 
> wise to seriously consider the wisdom of others with experience.
> 
> I will say that like others on this thread prefer determinism in my builds.
> 
> Whatever path leads me there is the path I want to walk. I'm not smart enough 
> to say if this proposal will preclude that or not. I am moved by the notion 
> that there are successful package managers out there as examples which today 
> achieve that goal. 

Alexis, Huon and I just had an in person conversation about this thread, and I 
wanted to post a summary:

1. I think there is some confusion in the thread based on us not describing in 
the proposal exactly how the package manager *behaves today*, and the parts 
which we expected to be unchanged.

We effectively "pin locally" today, in the sense that we *never* update a 
dependency without the users involvement, and we all agree we should never ever 
ever ever do this. I don't personally understand how any tool would ever not do 
this, so I may have skipped over clarifying this. I'm not sure if there was 
confusion on this point, but if so, hopefully that helps.

We also give you a way to "see" what you are pinned at with `swift package 
show-dependencies`.

What this proposal about is in one sense being able to export and share those 
pins.

2. Huon, Alexis, and I all agree we should never *inherit* pins by default.

3. Huon pointed out using "shared" to mean "package intended to be used as a 
dependency" isn't correct, since one might often share a package intended to 
just be built and used as a command line tool. OTOH, I personally don't care 
for library vs executable since executables can often be used as dependencies 
for server-side stuff, or as build tools.

4. We also all agree that there are effectively two very different workflows 
based on whether a package is intended to be a dependency.

We have three very interesting ecosystems:
A. Using packages as macOS/iOS/etc app dependencies.
B. Using packages for server-side Swift code.
C. Using packages to directly build command line tools and applications 
you can install.

In (A), all packages are dependencies, so our emphasis is on workflows oriented 
towards that.

In (B), we expect somewhat more sophistication around management of the pin 
file (w.r.t. PaaS deployment, continuous delivery, etc.), and so Huon and I at 
least agreed it was somewhat irrelevant. Also, we expect that more people 
developing in this space will simultaneously be factoring out packages which 
are intended to be dependencies.

In (C), by definition the most interesting and heavily used (as in installed) 
packages are *not* generally dependencies.

Today, (A) is a huge focus, and (B) is the main current consumer of SwiftPM. 
(C) is very interesting, and many of us would love to us it, but the reality is 
I don't see many people doing development in this space versus simply 
publishing via Homebrew, etc. Given that, my position is that *most* of the 
ecosystem is oriented towards packages which are intended to be dependencies, 
and we should choose our defaults accordingly.

5. Given that many people agree there are two workflows (we ourselves had 
talked about this a lot when writing the proposal, but didn't put it in), we 
felt it makes sense to consider adding that as an explicit declaration 
*somewhere*.

@Eloy, @Orta: Suppose we had a semantic notion of which packages were intended 
to be "top-level" versus used as a dependency, and we chose our defaults 
accordingly (in this case, we would orient workflows towards pinning by default 
in the top-level case, in the used as a dependency case we would orient away 
from it, e.g. warning you if you checked it in). What would you think of such a 
design?

 - Daniel

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Ryan Lovelett via swift-evolution]

> I particularly like Yehuda Katz's explanation on why it works this
> way: https://github.com/yarnpkg/yarn/issues/838#issuecomment-253362537

This was a powerful description for me. @Rob thank you for
providing that!

 1. The articulation of the purpose/philosophy behind the whole thing
was really clear and made sense to me. Even though I've never
used Rust or Cargo. Nor have I ever run into any problems with
npm such that I would prefer yarn over it. I've used Bundler for
years and love it.
 2. Yehuda Katz has been involved in making 3 different package managers
for 3 different communities/languages over at least a 6-year span.
It feels trite but it seems likely he has knowledge/experience we
might not. It might be wise to seriously consider the wisdom of
others with experience.

I will say that like others on this thread prefer determinism in
my builds.

Whatever path leads me there is the path I want to walk. I'm not smart
enough to say if this proposal will preclude that or not. I am moved by
the notion that there are successful package managers out there as
examples which today achieve that goal.
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Eloy Durán via swift-evolution]

>> Totally 
> 
> I'm not sure what this is to. :)

You and me both… Guess I must have lost track of my thought here while trying 
to respond from my phone and playing with the baby in between.

> Pin by default in the sense you are using it just means we automatically 
> would create "Package.pins", that is the only behavior change (i.e., you 
> don't inherit it from your dependencies).
> 
> Can you explain why simply needing to run one more command when you want to 
> do this is a big deal?

It’s clearly not a big deal in amount of work, what it comes down to is setting 
an example. If you, as an authority, make it the default option to not pin, 
then to many people that will signal that that’s The Right Thing to do.

> Is it possible that we have a different expectation about what the tools do 
> when you don't create this file?

No, it’s a philosophical difference in thinking about dependencies.

I see it as my responsibility to know exactly what code I’m pulling into my 
package. In my view, it’s absolutely unsafe to trust other people’s code. Even 
when they mean no harm, trusting them to properly apply SemVer is the same 
issue.

My worst nightmare is people updating dependencies and getting lazy in vetting 
them, a mindset that is in my observation the status quo in e.g. npm, a 
dependency manager which makes it the default to not pin and trust SemVer. On 
the other hand, the laziness might also be compounded by the micro-lib norm, 
it’s not unusual for npm packages to have a dependency graph of dozens, if not 
hundreds of packages.

This also leads me to the following point you bring up:

> One argument I haven't heard anyone refute is that we can always back down 
> from this position, but the converse isn't true (because its impact will have 
> permeated the ecosystem). I consider that an important point.

The reason I would not try to refute it is A) simply because it’s true and B) 
it’s, to me, more of a technicality that’s moot in contrast to my philosophical 
thoughts, as explained above.

One final note, playing the devil’s advocate against my own opinion, seeing as 
you are at a place where you can try it and later change, I’d say do it. 
However, it might be a good idea to document the possibility of this changing 
in the future, just so expectations are set correct and the thought to keep an 
eye on this is kept alive in the community.___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

> On Oct 14, 2016, at 10:39 AM, Max Desiatov  wrote:
> 
> Unfortunately, I'm not convinced that semantic versioning would help here. It 
> could have been different if any of the tools in Apple's ecosystem and other 
> ecosystems emphasized semantic versioning from the start.
> 
> I also strongly agree with issues highlighted by Orta, specifically with:
> 
> > Updates to code ships bugs. Updates you don’t expect gives you bugs you 
> > didn’t expect.
> 
> I would prefer the code that compiles on CI to be exactly the same that I run 
> locally without having a need to commit all the dependencies. I also think 
> that it makes sense for this to be the default behavior.

I agree this is true for your app (actually, what I think is ideal for an app 
is that you have two branches of CI, one for your pinned dependencies and one 
with them lifted, so you are detecting incoming future regressions ASAP).

This argument doesn't work for packages being shared, because you don't control 
what your dependencies do (unless we choose to inherit pins, which has larger 
problems).

 - Daniel

> 
> With best regards, Max.
> 
> 
>> On 14 Oct 2016, at 17:43, Daniel Dunbar > > wrote:
>> 
>> Can you check my reply to Eloy and see how it weighs with you?
>> 
>>  - Daniel
>> 
>>> On Oct 14, 2016, at 9:33 AM, Max Desiatov via swift-build-dev 
>>> > wrote:
>>> 
>>> I also strongly agree with this, I'd prefer version pinning to happen by 
>>> default, rather than with explicit command as it will make builds 
>>> reproducible by default. 
>>> 
>>> I totally agree that we can rely on past experience with other package 
>>> managers (npm being the case), where pinning with a separate command caused 
>>> more harm than good.
>>> 
>>> Overall, I think that it would be great if Package.lock was created by 
>>> default when it's not present and be updated only with an explicit command 
>>> for updating.
>>> 
>>> With best regards, Max.
>>> 
>>> 
 On 14 Oct 2016, at 08:29, orta therox via swift-build-dev 
 > wrote:
 
 Please don’t make this a separate command, it should ideally be created at 
 the end of an build (when there isn’t one already) or an update of your 
 dependencies - most people will be expecting to get the same set of 
 dependencies as the rest of their team. This pattern makes that harder.
 
 NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a 
 lot of time trying to keep that up to date for our npm projects. Facebook 
 made a replacement for NPM with mainly the  feature of “always locking” in 
 yarn  and I’d expect that to take a lot of the JS 
 mindshare on this one feature alone.
 
 -- 
 
 [A.]   Orta Therox
 
> w/ Artsy CocoaPods  / CocoaDocs 
>  / GIFs.app 
> 
> @orta  / orta.github.com 
> 
> Artsy is totally hiring iOS Devs  
> ATM
 
> On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev 
> > wrote:
> 
> Hi,
> 
> We're proposing version pinning feature in Swift Package Manager. The 
> proposal is available here 
> 
>  and also in this email:
> 
> Feedback welcomed!
> 
> Thanks,
> Ankit
> 
> 
> 
> Package Manager Version Pinning
> Proposal: SE-
> Author: Daniel Dunbar , Ankit Aggarwal 
> 
> Review Manager: TBD
> Status: Discussion
> Introduction
> This is a proposal for adding package manager features to "pin" or "lock" 
> package dependencies to particular versions.
> 
> Motivation
> As used in this proposal, version pinning refers to the practice of 
> controlling exactly which specific version of a dependency is selected by 
> the dependency resolution algorithm, independent from the semantic 
> versioning specification. Thus, it is a way of instructing the package 
> manager to select a particular version from among all of the versions of 
> a package which could be chosen while honoring the dependency constraints.
> 
> Terminology
> 
> We have chosen to use "pinning" to refer to this feature, over 
> "lockfiles", since the term "lock" is already overloaded between POSIX 
> file locks and locks in concurrent programming.
> 
> Philosophy
> 
> 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

> On Oct 14, 2016, at 10:37 AM, Eloy Duran  wrote:
> 
> Hey Daniel,
> 
> Totally 

I'm not sure what this is to. :)

> I think you should pin by default, I wouldn't even provide some option to 
> disable it.

Pin by default in the sense you are using it just means we automatically would 
create "Package.pins", that is the only behavior change (i.e., you don't 
inherit it from your dependencies).

Can you explain why simply needing to run one more command when you want to do 
this is a big deal? If you don't check it in, it doesn't change anything when 
you run `swift build`, that always kept a consistent set of versions. It 
changes `swift update`, but you were already running it at that point, so I'm 
not sure why that is desired.

Is it possible that we have a different expectation about what the tools do 
when you don't create this file? Our expectation is that you only ever get 
updated packages in a local build when you explicitly run `swift package 
update`. We expect to add features to notice when new dependencies are 
available, and let you choose to update, but for any one local build we are 
never talking about changing your dependencies without your interaction.

> As others have touched on, which I forgot to include, is that a library can 
> choose to not include the lock file in SCM. Especially if the lib uses a CI 
> for testing, that should bring up any issues with the latest versions of 
> dependencies while keeping a paper trail (local lock file) of dependency 
> versions that did work.
> 
> The key here is that it remains an explicit choice. The default should imo 
> always be consistent builds.

Consistent across what axis? If you don't inherit it, then you still don't get 
consistency between yourself and your clients. If you don't check it in, you 
don't get much of a behavior change. However, as is touched on in other places, 
choosing to inherit has big downsides.

> 
> 
> With regards to being in a different position, I doubt it. Swift developers 
> are, like in other languages, just humans, we all make mistakes. I think 
> we've all accepted that we'll always introduce bugs any time we have to make 
> decisions, the same applies to determining semantic versions.
> 
> You could automate some of that (e.g. check interfaces), but you cannot 
> guarantee behavioral intent and thus there's room left for human mistakes.

You can, however, in theory, apply a lot of automation for testing downstream 
dependencies (see earlier response).

> Please note, though, that I'm kinda pessimistic (realistic) about this and 
> it's solely based on experience with established ecosystems, where people 
> generally don't like being told how to do things. It may well be that your 
> situation will turn out to be different.

This is the trickiest part of this design question. We don't really know how 
the ecosystem will behave, we don't know how resilient Swift packages will be 
w.r.t. bugs and semantic versioning, etc. At the end of the day, we all just 
are sticking our fingers to the wind and hoping to do the best thing for 
everyone.

One argument I haven't heard anyone refute is that we can always back down from 
this position, but the converse isn't true (because its impact will have 
permeated the ecosystem). I consider that an important point.

> As the Queen would say, Godspeed!

:)

 - Daniel

> 
> On 14 Oct 2016, at 18:42, Daniel Dunbar  > wrote:
> 
>> Hey Eloy,
>> 
>> Same question as I sent to Orta, can you detail exactly what you would 
>> prefer to change?
>> 
>>> On Oct 14, 2016, at 4:06 AM, Eloy Durán via swift-evolution 
>>> > wrote:
>>> 
>>> I cannot agree more with Orta.
>>> 
>>> > It drives the user away from taking full advantage of semantic versioning.
>>> 
>>> While I ideally subscribe to this thought, the truth of the matter is that 
>>> this has proven unreliable on multiple occasions. Promoting the idea that 
>>> it works is going to inevitably lead users into problems when they did not 
>>> take explicit user action to update dependencies.
>> 
>> This makes a lot of sense to me. The open question in my mind is, given our 
>> other goals, are we in a position to make this model work, or to make it 
>> "the right model for us". Some of the other things I am considering (and 
>> maybe should go into the proposal):
>> 
>> 1. Swift is still evolving in a much greater way than other languages. That 
>> means we continue to need to have a very strong push towards forward 
>> versions. This is also true for the package manager features, we need the 
>> ecosystem to move forward aggressively so that we can move our own features 
>> forward aggressively.
>> 
>> 2. We have some lofty goals around semantic versioning, like trying to have 
>> a much deeper integration with the compiler, API, and ABI, and testing in 
>> order to help manage 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Rob Allen via swift-evolution]

> On 14 Oct 2016, at 18:55, Daniel Dunbar via swift-build-dev 
>  wrote:
> 
>> On Oct 14, 2016, at 9:43 AM, Alexis via swift-evolution 
>> > wrote:
>> 
>>> On Oct 14, 2016, at 2:01 AM, Ankit Aggarwal via swift-evolution 
>>> > wrote:
>>> 
>>> We considered making the pinning behavior default on running swift build, 
>>> however we think that pinning by default is likely to make the package 
>>> graph more constrained than it should be. It drives the user away from 
>>> taking full advantage of semantic versioning. We think it will be good for 
>>> the package ecosystem if such a restriction is not the default behavior and 
>>> that this design will lead to faster discovery of bugs and fixes in the 
>>> upstream.
>> 
>> I agree with the others that this is the better solution.
>> 
>> With regards to the constraining problem, the key insight adopted by 
>> Cargo/Yarn/Bundler is to distinguish libraries from applications. A library 
>> shouldn’t pin its dependencies, while an application should. This ensures 
>> that the ecosystem itself is maximally unconstrained, while ensuring actual 
>> applications continue to reliably build, regardless of ecosystem changes and 
>> the computer that it was built on. If a version of a library has trouble 
>> building with different versions, it should ideally specify that with its 
>> dependency constraints, not a lock file.
> 
> This I think is one big crux of the current discussion, and one of the things 
> not in the proposal was my expectation that most packages right now are going 
> to be shared (I personally wouldn't say libraries vs applications, since 
> applications can also be shared as build tools, I think it comes down to 
> whether it is shared or not).
> 
> If we agree that a library shouldn't pin its dependencies, how would you 
> surface this feature? What exact behavior would you prefer?
> 

Bundler, Composer, Cargo & Yarn all ignore the lock file on dependencies and 
only use the one on the main build target so that they can solve the 
constraining problem. The final build target lock file does have the list of 
all exact version required for all the dependencies and sub-dependencies though 
for deterministic building of the final product. 

This only works because they all take full advantage of semantic versioning 
anyway.

I particularly like Yehuda Katz's explanation on why it works this way: 
https://github.com/yarnpkg/yarn/issues/838#issuecomment-253362537 

Regards,

Rob...


___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Eloy Duran via swift-evolution]

> The big struggle I have is that if we go the other direction, and as a result 
> people's semantic versions become poorly specified, we will never be able to 
> recover. The converse is not true, if we start with this direction and 
> realize it doesn't work, we can relax our behavior.

Forgot to touch on this. One interesting thing we can observe from the current 
npm/yarn split is that because npm has set the tone about trusting semver 
plenty of people will still follow this, but in my opinion it's mostly a matter 
of them not being burned *yet*.

Whatever you decide is the right thing to do now will probably cement in users 
minds for years to come, regardless of how easy it technically is to make a 
change.

(Did I say yet that I'm pessimistic about these things?)

> On 14 Oct 2016, at 18:42, Daniel Dunbar  wrote:
> 
> Hey Eloy,
> 
> Same question as I sent to Orta, can you detail exactly what you would prefer 
> to change?
> 
>> On Oct 14, 2016, at 4:06 AM, Eloy Durán via swift-evolution 
>>  wrote:
>> 
>> I cannot agree more with Orta.
>> 
>> > It drives the user away from taking full advantage of semantic versioning.
>> 
>> While I ideally subscribe to this thought, the truth of the matter is that 
>> this has proven unreliable on multiple occasions. Promoting the idea that it 
>> works is going to inevitably lead users into problems when they did not take 
>> explicit user action to update dependencies.
> 
> This makes a lot of sense to me. The open question in my mind is, given our 
> other goals, are we in a position to make this model work, or to make it "the 
> right model for us". Some of the other things I am considering (and maybe 
> should go into the proposal):
> 
> 1. Swift is still evolving in a much greater way than other languages. That 
> means we continue to need to have a very strong push towards forward 
> versions. This is also true for the package manager features, we need the 
> ecosystem to move forward aggressively so that we can move our own features 
> forward aggressively.
> 
> 2. We have some lofty goals around semantic versioning, like trying to have a 
> much deeper integration with the compiler, API, and ABI, and testing in order 
> to help manage this more effectively.
> 
>> Semantic versioning is a great tool when you decide to sit down and 
>> explicitly update dependencies.
> 
> The problem is they go hand in hand. The more people pin versus following the 
> semantic versioning, the more and more likely it is that those specifications 
> are wrong. That leads to more pinning, which leads to more wrong 
> specifications.
> 
> My preference is that we very aggressively commit to using semantic 
> versioning, and then follow up with the entire cross functional design (an 
> index, if we ever do one, the compiler features, any IDE integration) to make 
> this work. I think we are in a better position than other tools which didn't 
> have the ability to make as many cross functional changes (e.g., integration 
> with the compiler to assist in semantic versioning).
> 
> On the other hand, I am also very pragmatic, and I respect your experience 
> here... if this model simply isn't going to work, then we shouldn't try to go 
> that way.
> 
> The big struggle I have is that if we go the other direction, and as a result 
> people's semantic versions become poorly specified, we will never be able to 
> recover. The converse is not true, if we start with this direction and 
> realize it doesn't work, we can relax our behavior.
> 
>> > We think it will be good for the package ecosystem if such a restriction 
>> > is not the default behavior and that this design will lead to faster 
>> > discovery of bugs and fixes in the upstream.
>> 
>> Again, ideally I would subscribe to this, but the truth is that when people 
>> are working on their projects they *really really really* do *not* like 
>> their builds breaking, especially not when it’s seemingly caused outside of 
>> their own fault.
> 
> One discussion we had a lot was that there are very different workflows 
> between when you are largely the consumer of packages, versus when you are 
> working on a package that is shared.
> 
> We generally agree that when you are simply the consumer of packages, pinning 
> makes sense.
> 
> However, when you are primarily a distributor of packages (and this is 
> expected to *most* of the momentum behind SwiftPM in the near term), I 
> believe that it is very important to the ecosystem that the semver specs be 
> correct, and so even if the team *wants* to pin, doing so would be actively 
> harmful.
> 
>> In the end, it comes down to whether you prioritise surfacing bugs (long 
>> term happiness) over user-experience (short term happiness). As a dependency 
>> manager author, I can tell you that I agree with your ideals and would want 
>> to choose the former, but as a user of dependency managers I would choose UX 
>> and Getting 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Max Desiatov via swift-evolution]

Unfortunately, I'm not convinced that semantic versioning would help here. It 
could have been different if any of the tools in Apple's ecosystem and other 
ecosystems emphasized semantic versioning from the start.

I also strongly agree with issues highlighted by Orta, specifically with:

> Updates to code ships bugs. Updates you don’t expect gives you bugs you 
> didn’t expect.

I would prefer the code that compiles on CI to be exactly the same that I run 
locally without having a need to commit all the dependencies. I also think that 
it makes sense for this to be the default behaviour.

With best regards, Max.


> On 14 Oct 2016, at 17:43, Daniel Dunbar  wrote:
> 
> Can you check my reply to Eloy and see how it weighs with you?
> 
>  - Daniel
> 
>> On Oct 14, 2016, at 9:33 AM, Max Desiatov via swift-build-dev 
>> > wrote:
>> 
>> I also strongly agree with this, I'd prefer version pinning to happen by 
>> default, rather than with explicit command as it will make builds 
>> reproducible by default. 
>> 
>> I totally agree that we can rely on past experience with other package 
>> managers (npm being the case), where pinning with a separate command caused 
>> more harm than good.
>> 
>> Overall, I think that it would be great if Package.lock was created by 
>> default when it's not present and be updated only with an explicit command 
>> for updating.
>> 
>> With best regards, Max.
>> 
>> 
>>> On 14 Oct 2016, at 08:29, orta therox via swift-build-dev 
>>> > wrote:
>>> 
>>> Please don’t make this a separate command, it should ideally be created at 
>>> the end of an build (when there isn’t one already) or an update of your 
>>> dependencies - most people will be expecting to get the same set of 
>>> dependencies as the rest of their team. This pattern makes that harder.
>>> 
>>> NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a 
>>> lot of time trying to keep that up to date for our npm projects. Facebook 
>>> made a replacement for NPM with mainly the  feature of “always locking” in 
>>> yarn  and I’d expect that to take a lot of the JS 
>>> mindshare on this one feature alone.
>>> 
>>> -- 
>>> 
>>> [A.]Orta Therox
>>> 
 w/ Artsy CocoaPods  / CocoaDocs 
  / GIFs.app 
 
 @orta  / orta.github.com 
 Artsy is totally hiring iOS Devs  
 ATM
>>> 
 On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev 
 > wrote:
 
 Hi,
 
 We're proposing version pinning feature in Swift Package Manager. The 
 proposal is available here 
 
  and also in this email:
 
 Feedback welcomed!
 
 Thanks,
 Ankit
 
 
 
 Package Manager Version Pinning
 Proposal: SE-
 Author: Daniel Dunbar , Ankit Aggarwal 
 
 Review Manager: TBD
 Status: Discussion
 Introduction
 This is a proposal for adding package manager features to "pin" or "lock" 
 package dependencies to particular versions.
 
 Motivation
 As used in this proposal, version pinning refers to the practice of 
 controlling exactly which specific version of a dependency is selected by 
 the dependency resolution algorithm, independent from the semantic 
 versioning specification. Thus, it is a way of instructing the package 
 manager to select a particular version from among all of the versions of a 
 package which could be chosen while honoring the dependency constraints.
 
 Terminology
 
 We have chosen to use "pinning" to refer to this feature, over 
 "lockfiles", since the term "lock" is already overloaded between POSIX 
 file locks and locks in concurrent programming.
 
 Philosophy
 
 Our philosophy with regard to pinning is that we actively want to 
 encourage packages to develop against the latest semantically appropriate 
 versions of their dependencies, in order to foster rapid development 
 amongst the ecosystem and strong reliance on the semantic versioning 
 concept. Our design for version pinning is thus intended to be a feature 
 for package authors and users to use in crafting specific workflows, not 
 be a mechanism by which most of the packages in the ecosystem pin 
 themselves to specific versions of each other.
 
 Use Cases
 
 Our proposal is designed to satisfy several different 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Eloy Duran via swift-evolution]

Hey Daniel,

Totally 

I think you should pin by default, I wouldn't even provide some option to 
disable it.

As others have touched on, which I forgot to include, is that a library can 
choose to not include the lock file in SCM. Especially if the lib uses a CI for 
testing, that should bring up any issues with the latest versions of 
dependencies while keeping a paper trail (local lock file) of dependency 
versions that did work.

The key here is that it remains an explicit choice. The default should imo 
always be consistent builds.



With regards to being in a different position, I doubt it. Swift developers 
are, like in other languages, just humans, we all make mistakes. I think we've 
all accepted that we'll always introduce bugs any time we have to make 
decisions, the same applies to determining semantic versions.

You could automate some of that (e.g. check interfaces), but you cannot 
guarantee behavioral intent and thus there's room left for human mistakes.

Please note, though, that I'm kinda pessimistic (realistic) about this and it's 
solely based on experience with established ecosystems, where people generally 
don't like being told how to do things. It may well be that your situation will 
turn out to be different.

As the Queen would say, Godspeed!

> On 14 Oct 2016, at 18:42, Daniel Dunbar  wrote:
> 
> Hey Eloy,
> 
> Same question as I sent to Orta, can you detail exactly what you would prefer 
> to change?
> 
>> On Oct 14, 2016, at 4:06 AM, Eloy Durán via swift-evolution 
>>  wrote:
>> 
>> I cannot agree more with Orta.
>> 
>> > It drives the user away from taking full advantage of semantic versioning.
>> 
>> While I ideally subscribe to this thought, the truth of the matter is that 
>> this has proven unreliable on multiple occasions. Promoting the idea that it 
>> works is going to inevitably lead users into problems when they did not take 
>> explicit user action to update dependencies.
> 
> This makes a lot of sense to me. The open question in my mind is, given our 
> other goals, are we in a position to make this model work, or to make it "the 
> right model for us". Some of the other things I am considering (and maybe 
> should go into the proposal):
> 
> 1. Swift is still evolving in a much greater way than other languages. That 
> means we continue to need to have a very strong push towards forward 
> versions. This is also true for the package manager features, we need the 
> ecosystem to move forward aggressively so that we can move our own features 
> forward aggressively.
> 
> 2. We have some lofty goals around semantic versioning, like trying to have a 
> much deeper integration with the compiler, API, and ABI, and testing in order 
> to help manage this more effectively.
> 
>> Semantic versioning is a great tool when you decide to sit down and 
>> explicitly update dependencies.
> 
> The problem is they go hand in hand. The more people pin versus following the 
> semantic versioning, the more and more likely it is that those specifications 
> are wrong. That leads to more pinning, which leads to more wrong 
> specifications.
> 
> My preference is that we very aggressively commit to using semantic 
> versioning, and then follow up with the entire cross functional design (an 
> index, if we ever do one, the compiler features, any IDE integration) to make 
> this work. I think we are in a better position than other tools which didn't 
> have the ability to make as many cross functional changes (e.g., integration 
> with the compiler to assist in semantic versioning).
> 
> On the other hand, I am also very pragmatic, and I respect your experience 
> here... if this model simply isn't going to work, then we shouldn't try to go 
> that way.
> 
> The big struggle I have is that if we go the other direction, and as a result 
> people's semantic versions become poorly specified, we will never be able to 
> recover. The converse is not true, if we start with this direction and 
> realize it doesn't work, we can relax our behavior.
> 
>> > We think it will be good for the package ecosystem if such a restriction 
>> > is not the default behavior and that this design will lead to faster 
>> > discovery of bugs and fixes in the upstream.
>> 
>> Again, ideally I would subscribe to this, but the truth is that when people 
>> are working on their projects they *really really really* do *not* like 
>> their builds breaking, especially not when it’s seemingly caused outside of 
>> their own fault.
> 
> One discussion we had a lot was that there are very different workflows 
> between when you are largely the consumer of packages, versus when you are 
> working on a package that is shared.
> 
> We generally agree that when you are simply the consumer of packages, pinning 
> makes sense.
> 
> However, when you are primarily a distributor of packages (and this is 
> expected to *most* of the momentum behind SwiftPM in the near term), 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

> On Oct 14, 2016, at 10:13 AM, orta therox  wrote:
> 
> Afraid it doesn’t convince me. Even if you have an index that has strict 
> semver adherence, the idea that you can trust people / machines to actually 
> understand whether something will break other people's build seems 
> unreasonable. Updates to code ships bugs. Updates you don’t expect gives you 
> bugs you didn’t expect

Updates you don't expect can also fix bugs you didn't find (or found but hadn't 
diagnosed). I'm not sure it is a one way street.

I completely agree with you here though, you can't trust the tooling. However, 
we also have the ability to test things via running actual tests against 
downstream dependencies. The combination of those two may be quite powerful, 
even if the latter is naturally resource constrained.

> Alexis hits it on the head with the differentiation between a library and an 
> app - libraries should define their own supported ranges, and they require 
> far less locking than applications. An application should always have a lock 
> so you have determinate builds and can update dependencies on purpose.

Yup, we agree here. I'll reply to this on Alexis's post.

> > We generally agree that when you are simply the consumer of packages, 
> > pinning makes sense.
> 
> Isn’t this going to be the majority of the behaviour for the tool on the long 
> run? So I would expect that to be where locking happens.

I don't think so.

Consider the Xcode use case, for example: we are not actively planning on 
adding support for all the myriad kinds of top-level targets and app needs, so 
the packages used in that context will by nature be a package which was shared.

> Sidenote: FWIW not a fan of the name change from lockfile - it feels 
> arbitrary, but I figured the change to a separate command was a bigger 
> elephant in the room.

I agree keeping them in separate threads is handy. I'll reply to this on Max's 
post.

 - Daniel

> 
> -- 
> 
> [A.]  Orta Therox
> 
>> w/ Artsy CocoaPods  / CocoaDocs 
>>  / GIFs.app 
>> 
>> @orta  / orta.github.com 
>> On 14 Oct 2016, at 17:43, Daniel Dunbar > > wrote:
>> 
>> Can you check my reply to Eloy and see how it weighs with you?
>> 
>>  - Daniel
>> 
>>> On Oct 14, 2016, at 9:33 AM, Max Desiatov via swift-build-dev 
>>> > wrote:
>>> 
>>> I also strongly agree with this, I'd prefer version pinning to happen by 
>>> default, rather than with explicit command as it will make builds 
>>> reproducible by default. 
>>> 
>>> I totally agree that we can rely on past experience with other package 
>>> managers (npm being the case), where pinning with a separate command caused 
>>> more harm than good.
>>> 
>>> Overall, I think that it would be great if Package.lock was created by 
>>> default when it's not present and be updated only with an explicit command 
>>> for updating.
>>> 
>>> With best regards, Max.
>>> 
>>> 
 On 14 Oct 2016, at 08:29, orta therox via swift-build-dev 
 > wrote:
 
 Please don’t make this a separate command, it should ideally be created at 
 the end of an build (when there isn’t one already) or an update of your 
 dependencies - most people will be expecting to get the same set of 
 dependencies as the rest of their team. This pattern makes that harder.
 
 NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a 
 lot of time trying to keep that up to date for our npm projects. Facebook 
 made a replacement for NPM with mainly the  feature of “always locking” in 
 yarn  and I’d expect that to take a lot of the JS 
 mindshare on this one feature alone.
 
 -- 
 
 [A.]   Orta Therox
 
> w/ Artsy CocoaPods  / CocoaDocs 
>  / GIFs.app 
> 
> @orta  / orta.github.com 
> 
> Artsy is totally hiring iOS Devs  
> ATM
 
> On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev 
> > wrote:
> 
> Hi,
> 
> We're proposing version pinning feature in Swift Package Manager. The 
> proposal is available here 
> 
>  and also in this email:
> 
> Feedback welcomed!
> 
> Thanks,
> Ankit
> 
> 
> 
> Package Manager Version 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[orta therox via swift-evolution]

Afraid it doesn’t convince me. Even if you have an index that has strict semver 
adherence, the idea that you can trust people / machines to actually understand 
whether something will break other people's build seems unreasonable. Updates 
to code ships bugs. Updates you don’t expect gives you bugs you didn’t expect.

Alexis hits it on the head with the differentiation between a library and an 
app - libraries should define their own supported ranges, and they require far 
less locking than applications. An application should always have a lock so you 
have determinate builds and can update dependencies on purpose.

> We generally agree that when you are simply the consumer of packages, pinning 
> makes sense.

Isn’t this going to be the majority of the behaviour for the tool on the long 
run? So I would expect that to be where locking happens.

Sidenote: FWIW not a fan of the name change from lockfile - it feels arbitrary, 
but I figured the change to a separate command was a bigger elephant in the 
room.

-- 

[A.]Orta Therox

> w/ Artsy CocoaPods  / CocoaDocs 
>  / GIFs.app 
> 
> @orta  / orta.github.com 
> On 14 Oct 2016, at 17:43, Daniel Dunbar  wrote:
> 
> Can you check my reply to Eloy and see how it weighs with you?
> 
>  - Daniel
> 
>> On Oct 14, 2016, at 9:33 AM, Max Desiatov via swift-build-dev 
>> > wrote:
>> 
>> I also strongly agree with this, I'd prefer version pinning to happen by 
>> default, rather than with explicit command as it will make builds 
>> reproducible by default. 
>> 
>> I totally agree that we can rely on past experience with other package 
>> managers (npm being the case), where pinning with a separate command caused 
>> more harm than good.
>> 
>> Overall, I think that it would be great if Package.lock was created by 
>> default when it's not present and be updated only with an explicit command 
>> for updating.
>> 
>> With best regards, Max.
>> 
>> 
>>> On 14 Oct 2016, at 08:29, orta therox via swift-build-dev 
>>> > wrote:
>>> 
>>> Please don’t make this a separate command, it should ideally be created at 
>>> the end of an build (when there isn’t one already) or an update of your 
>>> dependencies - most people will be expecting to get the same set of 
>>> dependencies as the rest of their team. This pattern makes that harder.
>>> 
>>> NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a 
>>> lot of time trying to keep that up to date for our npm projects. Facebook 
>>> made a replacement for NPM with mainly the  feature of “always locking” in 
>>> yarn  and I’d expect that to take a lot of the JS 
>>> mindshare on this one feature alone.
>>> 
>>> -- 
>>> 
>>> [A.]Orta Therox
>>> 
 w/ Artsy CocoaPods  / CocoaDocs 
  / GIFs.app 
 
 @orta  / orta.github.com 
 Artsy is totally hiring iOS Devs  
 ATM
>>> 
 On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev 
 > wrote:
 
 Hi,
 
 We're proposing version pinning feature in Swift Package Manager. The 
 proposal is available here 
 
  and also in this email:
 
 Feedback welcomed!
 
 Thanks,
 Ankit
 
 
 
 Package Manager Version Pinning
 Proposal: SE-
 Author: Daniel Dunbar , Ankit Aggarwal 
 
 Review Manager: TBD
 Status: Discussion
 Introduction
 This is a proposal for adding package manager features to "pin" or "lock" 
 package dependencies to particular versions.
 
 Motivation
 As used in this proposal, version pinning refers to the practice of 
 controlling exactly which specific version of a dependency is selected by 
 the dependency resolution algorithm, independent from the semantic 
 versioning specification. Thus, it is a way of instructing the package 
 manager to select a particular version from among all of the versions of a 
 package which could be chosen while honoring the dependency constraints.
 
 Terminology
 
 We have chosen to use "pinning" to refer to this feature, over 
 "lockfiles", since the term "lock" is already overloaded between POSIX 
 file locks and locks in concurrent programming.
 
 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Rob Allen via swift-evolution]

Hi,

As noted by others, the standard extension for this file .lock, so I would 
prefer to keep that convention.

As I would never want my CI or any random team member to build a different 
package to what I've developed and tested already, I will always want 
Package.lock to exist. It's much easier if the system always creates 
Package.lock on swift package update and always uses it (if it exists) with 
swift build. This way, I get to choose when to go from 1.3.1 to 1.3.2 just in 
case I happen to depend on that bug that's now been fixed…

I can't see any downside to not creating it automatically and it results in so 
much more consistency across a team.

Regards,

Rob...


> On 14 Oct 2016, at 17:29, Daniel Dunbar via swift-build-dev 
>  wrote:
> 
> Hey Orta,
> 
> Thanks for this feedback, this was one of the hotly debated points as you 
> might imagine.
> 
> Can you elaborate on exactly how you would prefer this to work?
> 
> The way I see it, the only thing that this changes is that the initial 
> package **author** who wants to be in this situation has to, one time, run 
> "package pin --all", and then "git add Package.pins". The latter is always 
> going to be necessary, so are you arguing the first should just be done by 
> default, or is there something deeper here?
> 
>  - Daniel
> 
>> On Oct 14, 2016, at 12:29 AM, orta therox via swift-evolution 
>> > wrote:
>> 
>> Please don’t make this a separate command, it should ideally be created at 
>> the end of an build (when there isn’t one already) or an update of your 
>> dependencies - most people will be expecting to get the same set of 
>> dependencies as the rest of their team. This pattern makes that harder.
>> 
>> NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot 
>> of time trying to keep that up to date for our npm projects. Facebook made a 
>> replacement for NPM with mainly the  feature of “always locking” in yarn 
>>  and I’d expect that to take a lot of the JS mindshare 
>> on this one feature alone.

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

> On Oct 14, 2016, at 9:35 AM, Daniel Duan  wrote:
> 
> The spelling nitpick should have been on GitHub in retrospect. I wrongly 
> assumed everyone know of our preference for dialect :)
> 
> Point 1-7 are all concrete descriptions of features. That's probably why the 
> SHA point feels out-of-place. Perhaps it deserves its own section.

Good point, let's move it to a future directions section or something.

 - Daniel

> 
> On Oct 14, 2016, at 9:26 AM, Daniel Dunbar  > wrote:
> 
>> 
>>> On Oct 13, 2016, at 11:16 PM, Daniel Duan via swift-build-dev 
>>> > wrote:
>>> 
>>> This is a familiar feature among package managers and has been proven 
>>> useful in practice. Two points regarding the proposed text:
>>> 
>>> 1. “honor” is mis-spelled in “weird queen’s dialect”.
>>> 2. SHA/man-in-the-middle attack section needs either more detailed 
>>> explanation/reference materials, or shouldn’t be included at all (assuming 
>>> when the hash feature gets introduced, there’d be another proposal for it).
>> 
>> I agree this needs more discussion, but given that we aren't proposing 
>> anything here, I feel like it is just a shoutout to an area for future 
>> development. What is the harm of having it there?
>> 
>> One of the possible values of having it in is jumpstarting the actual 
>> conversation on the topic!
>> 
>>  - Daniel
>> 
>>> 
>>> Overall strong +1.
 On Oct 13, 2016, at 11:01 PM, Ankit Aggarwal via swift-evolution 
 > wrote:
 
 Hi,
 
 We're proposing version pinning feature in Swift Package Manager. The 
 proposal is available here 
 
  and also in this email:
 
 Feedback welcomed!
 
 Thanks,
 Ankit
 
 
 
 Package Manager Version Pinning
 Proposal: SE-
 Author: Daniel Dunbar , Ankit Aggarwal 
 
 Review Manager: TBD
 Status: Discussion
 Introduction
 This is a proposal for adding package manager features to "pin" or "lock" 
 package dependencies to particular versions.
 
 Motivation
 As used in this proposal, version pinning refers to the practice of 
 controlling exactly which specific version of a dependency is selected by 
 the dependency resolution algorithm, independent from the semantic 
 versioning specification. Thus, it is a way of instructing the package 
 manager to select a particular version from among all of the versions of a 
 package which could be chosen while honoring the dependency constraints.
 
 Terminology
 
 We have chosen to use "pinning" to refer to this feature, over 
 "lockfiles", since the term "lock" is already overloaded between POSIX 
 file locks and locks in concurrent programming.
 
 Philosophy
 
 Our philosophy with regard to pinning is that we actively want to 
 encourage packages to develop against the latest semantically appropriate 
 versions of their dependencies, in order to foster rapid development 
 amongst the ecosystem and strong reliance on the semantic versioning 
 concept. Our design for version pinning is thus intended to be a feature 
 for package authors and users to use in crafting specific workflows, not 
 be a mechanism by which most of the packages in the ecosystem pin 
 themselves to specific versions of each other.
 
 Use Cases
 
 Our proposal is designed to satisfy several different use cases for such a 
 behavior:
 
 Standardizing team workflows
 
 When collaborating on a package, it can be valuable for team members (and 
 continuous integration) to all know they are using the same exact version 
 of dependencies, to avoid "works for me" situations.
 
 This can be particularly important for certain kinds of open source 
 projects which are actively being cloned by new users, and which want to 
 have some measure of control around exactly which available version of a 
 dependency is selected.
 
 Difficult to test packages or dependencies
 
 Complex packages which have dependencies which may be hard to test, or 
 hard to analyze when they break, may choose to maintain careful control 
 over what versions of their upstream dependencies they recommend -- even 
 if conceptually they regularly update those recommendations following the 
 true semantic version specification of the dependency.
 
 Dependency locking w.r.t. deployment
 
 When stabilizing a release for deployment, or building a version of a 
 package for deployment, it is important to be able to lock down the 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

Can you check my reply to Eloy and see how it weighs with you?

 - Daniel

> On Oct 14, 2016, at 9:33 AM, Max Desiatov via swift-build-dev 
>  wrote:
> 
> I also strongly agree with this, I'd prefer version pinning to happen by 
> default, rather than with explicit command as it will make builds 
> reproducible by default. 
> 
> I totally agree that we can rely on past experience with other package 
> managers (npm being the case), where pinning with a separate command caused 
> more harm than good.
> 
> Overall, I think that it would be great if Package.lock was created by 
> default when it's not present and be updated only with an explicit command 
> for updating.
> 
> With best regards, Max.
> 
> 
>> On 14 Oct 2016, at 08:29, orta therox via swift-build-dev 
>> > wrote:
>> 
>> Please don’t make this a separate command, it should ideally be created at 
>> the end of an build (when there isn’t one already) or an update of your 
>> dependencies - most people will be expecting to get the same set of 
>> dependencies as the rest of their team. This pattern makes that harder.
>> 
>> NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot 
>> of time trying to keep that up to date for our npm projects. Facebook made a 
>> replacement for NPM with mainly the  feature of “always locking” in yarn 
>>  and I’d expect that to take a lot of the JS mindshare 
>> on this one feature alone.
>> 
>> -- 
>> 
>> [A.] Orta Therox
>> 
>>> w/ Artsy CocoaPods  / CocoaDocs 
>>>  / GIFs.app 
>>> 
>>> @orta  / orta.github.com 
>>> Artsy is totally hiring iOS Devs  ATM
>> 
>>> On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev 
>>> > wrote:
>>> 
>>> Hi,
>>> 
>>> We're proposing version pinning feature in Swift Package Manager. The 
>>> proposal is available here 
>>> 
>>>  and also in this email:
>>> 
>>> Feedback welcomed!
>>> 
>>> Thanks,
>>> Ankit
>>> 
>>> 
>>> 
>>> Package Manager Version Pinning
>>> Proposal: SE-
>>> Author: Daniel Dunbar , Ankit Aggarwal 
>>> 
>>> Review Manager: TBD
>>> Status: Discussion
>>> Introduction
>>> This is a proposal for adding package manager features to "pin" or "lock" 
>>> package dependencies to particular versions.
>>> 
>>> Motivation
>>> As used in this proposal, version pinning refers to the practice of 
>>> controlling exactly which specific version of a dependency is selected by 
>>> the dependency resolution algorithm, independent from the semantic 
>>> versioning specification. Thus, it is a way of instructing the package 
>>> manager to select a particular version from among all of the versions of a 
>>> package which could be chosen while honoring the dependency constraints.
>>> 
>>> Terminology
>>> 
>>> We have chosen to use "pinning" to refer to this feature, over "lockfiles", 
>>> since the term "lock" is already overloaded between POSIX file locks and 
>>> locks in concurrent programming.
>>> 
>>> Philosophy
>>> 
>>> Our philosophy with regard to pinning is that we actively want to encourage 
>>> packages to develop against the latest semantically appropriate versions of 
>>> their dependencies, in order to foster rapid development amongst the 
>>> ecosystem and strong reliance on the semantic versioning concept. Our 
>>> design for version pinning is thus intended to be a feature for package 
>>> authors and users to use in crafting specific workflows, not be a mechanism 
>>> by which most of the packages in the ecosystem pin themselves to specific 
>>> versions of each other.
>>> 
>>> Use Cases
>>> 
>>> Our proposal is designed to satisfy several different use cases for such a 
>>> behavior:
>>> 
>>> Standardizing team workflows
>>> 
>>> When collaborating on a package, it can be valuable for team members (and 
>>> continuous integration) to all know they are using the same exact version 
>>> of dependencies, to avoid "works for me" situations.
>>> 
>>> This can be particularly important for certain kinds of open source 
>>> projects which are actively being cloned by new users, and which want to 
>>> have some measure of control around exactly which available version of a 
>>> dependency is selected.
>>> 
>>> Difficult to test packages or dependencies
>>> 
>>> Complex packages which have dependencies which may be hard to test, or hard 
>>> to analyze when they break, may choose to maintain careful control over 
>>> what versions of their upstream dependencies they recommend -- even if 
>>> 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

Hey Eloy,

Same question as I sent to Orta, can you detail exactly what you would prefer 
to change?

> On Oct 14, 2016, at 4:06 AM, Eloy Durán via swift-evolution 
>  wrote:
> 
> I cannot agree more with Orta.
> 
> > It drives the user away from taking full advantage of semantic versioning.
> 
> While I ideally subscribe to this thought, the truth of the matter is that 
> this has proven unreliable on multiple occasions. Promoting the idea that it 
> works is going to inevitably lead users into problems when they did not take 
> explicit user action to update dependencies.

This makes a lot of sense to me. The open question in my mind is, given our 
other goals, are we in a position to make this model work, or to make it "the 
right model for us". Some of the other things I am considering (and maybe 
should go into the proposal):

1. Swift is still evolving in a much greater way than other languages. That 
means we continue to need to have a very strong push towards forward versions. 
This is also true for the package manager features, we need the ecosystem to 
move forward aggressively so that we can move our own features forward 
aggressively.

2. We have some lofty goals around semantic versioning, like trying to have a 
much deeper integration with the compiler, API, and ABI, and testing in order 
to help manage this more effectively.

> Semantic versioning is a great tool when you decide to sit down and 
> explicitly update dependencies.

The problem is they go hand in hand. The more people pin versus following the 
semantic versioning, the more and more likely it is that those specifications 
are wrong. That leads to more pinning, which leads to more wrong specifications.

My preference is that we very aggressively commit to using semantic versioning, 
and then follow up with the entire cross functional design (an index, if we 
ever do one, the compiler features, any IDE integration) to make this work. I 
think we are in a better position than other tools which didn't have the 
ability to make as many cross functional changes (e.g., integration with the 
compiler to assist in semantic versioning).

On the other hand, I am also very pragmatic, and I respect your experience 
here... if this model simply isn't going to work, then we shouldn't try to go 
that way.

The big struggle I have is that if we go the other direction, and as a result 
people's semantic versions become poorly specified, we will never be able to 
recover. The converse is not true, if we start with this direction and realize 
it doesn't work, we can relax our behavior.

> > We think it will be good for the package ecosystem if such a restriction is 
> > not the default behavior and that this design will lead to faster discovery 
> > of bugs and fixes in the upstream.
> 
> Again, ideally I would subscribe to this, but the truth is that when people 
> are working on their projects they *really really really* do *not* like their 
> builds breaking, especially not when it’s seemingly caused outside of their 
> own fault.

One discussion we had a lot was that there are very different workflows between 
when you are largely the consumer of packages, versus when you are working on a 
package that is shared.

We generally agree that when you are simply the consumer of packages, pinning 
makes sense.

However, when you are primarily a distributor of packages (and this is expected 
to *most* of the momentum behind SwiftPM in the near term), I believe that it 
is very important to the ecosystem that the semver specs be correct, and so 
even if the team *wants* to pin, doing so would be actively harmful.

> In the end, it comes down to whether you prioritise surfacing bugs (long term 
> happiness) over user-experience (short term happiness). As a dependency 
> manager author, I can tell you that I agree with your ideals and would want 
> to choose the former, but as a user of dependency managers I would choose UX 
> and Getting (The) Things (I Really Wanted To Do) Done over surfacing bugs 
> *any* day.

I really appreciate the feedback. Do my arguments that we might be in a 
position to do better here have any weight with you?

 - Daniel

> 
> [A.]  Eloy Durán
>   Artsy 
>   
>   e...@artsy.net   Twitter 
> 
>   GitHub 
>> On 14 Oct 2016, at 09:29, orta therox via swift-build-dev 
>> > wrote:
>> 
>> Please don’t make this a separate command, it should ideally be created at 
>> the end of an build (when there isn’t one already) or an update of your 
>> dependencies - most people will be expecting to get the same set of 
>> dependencies as the rest of their team. This pattern makes that harder.
>> 
>> NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot 
>> of time trying to 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Duan via swift-evolution]

The spelling nitpick should have been on GitHub in retrospect. I wrongly 
assumed everyone know of our preference for dialect :)

Point 1-7 are all concrete descriptions of features. That's probably why the 
SHA point feels out-of-place. Perhaps it deserves its own section.

> On Oct 14, 2016, at 9:26 AM, Daniel Dunbar  wrote:
> 
> 
>> On Oct 13, 2016, at 11:16 PM, Daniel Duan via swift-build-dev 
>>  wrote:
>> 
>> This is a familiar feature among package managers and has been proven useful 
>> in practice. Two points regarding the proposed text:
>> 
>> 1. “honor” is mis-spelled in “weird queen’s dialect”.
>> 2. SHA/man-in-the-middle attack section needs either more detailed 
>> explanation/reference materials, or shouldn’t be included at all (assuming 
>> when the hash feature gets introduced, there’d be another proposal for it).
> 
> I agree this needs more discussion, but given that we aren't proposing 
> anything here, I feel like it is just a shoutout to an area for future 
> development. What is the harm of having it there?
> 
> One of the possible values of having it in is jumpstarting the actual 
> conversation on the topic!
> 
>  - Daniel
> 
>> 
>> Overall strong +1.
>>> On Oct 13, 2016, at 11:01 PM, Ankit Aggarwal via swift-evolution 
>>>  wrote:
>>> 
>>> Hi,
>>> 
>>> We're proposing version pinning feature in Swift Package Manager. The 
>>> proposal is available here and also in this email:
>>> 
>>> Feedback welcomed!
>>> 
>>> Thanks,
>>> Ankit
>>> 
>>> 
>>> 
>>> Package Manager Version Pinning
>>> Proposal: SE-
>>> Author: Daniel Dunbar, Ankit Aggarwal
>>> Review Manager: TBD
>>> Status: Discussion
>>> Introduction
>>> This is a proposal for adding package manager features to "pin" or "lock" 
>>> package dependencies to particular versions.
>>> 
>>> Motivation
>>> As used in this proposal, version pinning refers to the practice of 
>>> controlling exactly which specific version of a dependency is selected by 
>>> the dependency resolution algorithm, independent from the semantic 
>>> versioning specification. Thus, it is a way of instructing the package 
>>> manager to select a particular version from among all of the versions of a 
>>> package which could be chosen while honoring the dependency constraints.
>>> 
>>> Terminology
>>> 
>>> We have chosen to use "pinning" to refer to this feature, over "lockfiles", 
>>> since the term "lock" is already overloaded between POSIX file locks and 
>>> locks in concurrent programming.
>>> 
>>> Philosophy
>>> 
>>> Our philosophy with regard to pinning is that we actively want to encourage 
>>> packages to develop against the latest semantically appropriate versions of 
>>> their dependencies, in order to foster rapid development amongst the 
>>> ecosystem and strong reliance on the semantic versioning concept. Our 
>>> design for version pinning is thus intended to be a feature for package 
>>> authors and users to use in crafting specific workflows, not be a mechanism 
>>> by which most of the packages in the ecosystem pin themselves to specific 
>>> versions of each other.
>>> 
>>> Use Cases
>>> 
>>> Our proposal is designed to satisfy several different use cases for such a 
>>> behavior:
>>> 
>>> Standardizing team workflows
>>> 
>>> When collaborating on a package, it can be valuable for team members (and 
>>> continuous integration) to all know they are using the same exact version 
>>> of dependencies, to avoid "works for me" situations.
>>> 
>>> This can be particularly important for certain kinds of open source 
>>> projects which are actively being cloned by new users, and which want to 
>>> have some measure of control around exactly which available version of a 
>>> dependency is selected.
>>> 
>>> Difficult to test packages or dependencies
>>> 
>>> Complex packages which have dependencies which may be hard to test, or hard 
>>> to analyze when they break, may choose to maintain careful control over 
>>> what versions of their upstream dependencies they recommend -- even if 
>>> conceptually they regularly update those recommendations following the true 
>>> semantic version specification of the dependency.
>>> 
>>> Dependency locking w.r.t. deployment
>>> 
>>> When stabilizing a release for deployment, or building a version of a 
>>> package for deployment, it is important to be able to lock down the exact 
>>> versions of dependencies in use, so that the resulting product can be 
>>> exactly recreated later if necessary.
>>> 
>>> Proposed solution
>>> We will introduce support for an optional new file Package.pins adjacent to 
>>> the Package.swift manifest, called the "pins file". We will also introduce 
>>> a number of new commands (see below) for maintaining the pins file.
>>> 
>>> This file will record the active version pin information for the package, 
>>> including data such as the package identifier, the pinned version, and 
>>> explicit 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Max Desiatov via swift-evolution]

I also strongly agree with this, I'd prefer version pinning to happen by 
default, rather than with explicit command as it will make builds reproducible 
by default. 

I totally agree that we can rely on past experience with other package managers 
(npm being the case), where pinning with a separate command caused more harm 
than good.

Overall, I think that it would be great if Package.lock was created by default 
when it's not present and be updated only with an explicit command for updating.

With best regards, Max.


> On 14 Oct 2016, at 08:29, orta therox via swift-build-dev 
>  wrote:
> 
> Please don’t make this a separate command, it should ideally be created at 
> the end of an build (when there isn’t one already) or an update of your 
> dependencies - most people will be expecting to get the same set of 
> dependencies as the rest of their team. This pattern makes that harder.
> 
> NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot 
> of time trying to keep that up to date for our npm projects. Facebook made a 
> replacement for NPM with mainly the  feature of “always locking” in yarn 
>  and I’d expect that to take a lot of the JS mindshare 
> on this one feature alone.
> 
> -- 
> 
> [A.]  Orta Therox
> 
>> w/ Artsy CocoaPods  / CocoaDocs 
>>  / GIFs.app 
>> 
>> @orta  / orta.github.com 
>> Artsy is totally hiring iOS Devs  ATM
> 
>> On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev 
>> > wrote:
>> 
>> Hi,
>> 
>> We're proposing version pinning feature in Swift Package Manager. The 
>> proposal is available here 
>> 
>>  and also in this email:
>> 
>> Feedback welcomed!
>> 
>> Thanks,
>> Ankit
>> 
>> 
>> 
>> Package Manager Version Pinning
>> Proposal: SE-
>> Author: Daniel Dunbar , Ankit Aggarwal 
>> 
>> Review Manager: TBD
>> Status: Discussion
>> Introduction
>> This is a proposal for adding package manager features to "pin" or "lock" 
>> package dependencies to particular versions.
>> 
>> Motivation
>> As used in this proposal, version pinning refers to the practice of 
>> controlling exactly which specific version of a dependency is selected by 
>> the dependency resolution algorithm, independent from the semantic 
>> versioning specification. Thus, it is a way of instructing the package 
>> manager to select a particular version from among all of the versions of a 
>> package which could be chosen while honoring the dependency constraints.
>> 
>> Terminology
>> 
>> We have chosen to use "pinning" to refer to this feature, over "lockfiles", 
>> since the term "lock" is already overloaded between POSIX file locks and 
>> locks in concurrent programming.
>> 
>> Philosophy
>> 
>> Our philosophy with regard to pinning is that we actively want to encourage 
>> packages to develop against the latest semantically appropriate versions of 
>> their dependencies, in order to foster rapid development amongst the 
>> ecosystem and strong reliance on the semantic versioning concept. Our design 
>> for version pinning is thus intended to be a feature for package authors and 
>> users to use in crafting specific workflows, not be a mechanism by which 
>> most of the packages in the ecosystem pin themselves to specific versions of 
>> each other.
>> 
>> Use Cases
>> 
>> Our proposal is designed to satisfy several different use cases for such a 
>> behavior:
>> 
>> Standardizing team workflows
>> 
>> When collaborating on a package, it can be valuable for team members (and 
>> continuous integration) to all know they are using the same exact version of 
>> dependencies, to avoid "works for me" situations.
>> 
>> This can be particularly important for certain kinds of open source projects 
>> which are actively being cloned by new users, and which want to have some 
>> measure of control around exactly which available version of a dependency is 
>> selected.
>> 
>> Difficult to test packages or dependencies
>> 
>> Complex packages which have dependencies which may be hard to test, or hard 
>> to analyze when they break, may choose to maintain careful control over what 
>> versions of their upstream dependencies they recommend -- even if 
>> conceptually they regularly update those recommendations following the true 
>> semantic version specification of the dependency.
>> 
>> Dependency locking w.r.t. deployment
>> 
>> When stabilizing a release for deployment, or building a version of a 
>> package for deployment, it is important to be able to lock down the exact 
>> versions of 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

Hey Orta,

Thanks for this feedback, this was one of the hotly debated points as you might 
imagine.

Can you elaborate on exactly how you would prefer this to work?

The way I see it, the only thing that this changes is that the initial package 
**author** who wants to be in this situation has to, one time, run "package pin 
--all", and then "git add Package.pins". The latter is always going to be 
necessary, so are you arguing the first should just be done by default, or is 
there something deeper here?

 - Daniel

> On Oct 14, 2016, at 12:29 AM, orta therox via swift-evolution 
>  wrote:
> 
> Please don’t make this a separate command, it should ideally be created at 
> the end of an build (when there isn’t one already) or an update of your 
> dependencies - most people will be expecting to get the same set of 
> dependencies as the rest of their team. This pattern makes that harder.
> 
> NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot 
> of time trying to keep that up to date for our npm projects. Facebook made a 
> replacement for NPM with mainly the  feature of “always locking” in yarn 
>  and I’d expect that to take a lot of the JS mindshare 
> on this one feature alone.
> 
> -- 
> 
> [A.]  Orta Therox
> 
>> w/ Artsy CocoaPods  / CocoaDocs 
>>  / GIFs.app 
>> 
>> @orta  / orta.github.com 
>> Artsy is totally hiring iOS Devs  ATM
> 
>> On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev 
>> > wrote:
>> 
>> Hi,
>> 
>> We're proposing version pinning feature in Swift Package Manager. The 
>> proposal is available here 
>> 
>>  and also in this email:
>> 
>> Feedback welcomed!
>> 
>> Thanks,
>> Ankit
>> 
>> 
>> 
>> Package Manager Version Pinning
>> Proposal: SE-
>> Author: Daniel Dunbar , Ankit Aggarwal 
>> 
>> Review Manager: TBD
>> Status: Discussion
>> Introduction
>> This is a proposal for adding package manager features to "pin" or "lock" 
>> package dependencies to particular versions.
>> 
>> Motivation
>> As used in this proposal, version pinning refers to the practice of 
>> controlling exactly which specific version of a dependency is selected by 
>> the dependency resolution algorithm, independent from the semantic 
>> versioning specification. Thus, it is a way of instructing the package 
>> manager to select a particular version from among all of the versions of a 
>> package which could be chosen while honoring the dependency constraints.
>> 
>> Terminology
>> 
>> We have chosen to use "pinning" to refer to this feature, over "lockfiles", 
>> since the term "lock" is already overloaded between POSIX file locks and 
>> locks in concurrent programming.
>> 
>> Philosophy
>> 
>> Our philosophy with regard to pinning is that we actively want to encourage 
>> packages to develop against the latest semantically appropriate versions of 
>> their dependencies, in order to foster rapid development amongst the 
>> ecosystem and strong reliance on the semantic versioning concept. Our design 
>> for version pinning is thus intended to be a feature for package authors and 
>> users to use in crafting specific workflows, not be a mechanism by which 
>> most of the packages in the ecosystem pin themselves to specific versions of 
>> each other.
>> 
>> Use Cases
>> 
>> Our proposal is designed to satisfy several different use cases for such a 
>> behavior:
>> 
>> Standardizing team workflows
>> 
>> When collaborating on a package, it can be valuable for team members (and 
>> continuous integration) to all know they are using the same exact version of 
>> dependencies, to avoid "works for me" situations.
>> 
>> This can be particularly important for certain kinds of open source projects 
>> which are actively being cloned by new users, and which want to have some 
>> measure of control around exactly which available version of a dependency is 
>> selected.
>> 
>> Difficult to test packages or dependencies
>> 
>> Complex packages which have dependencies which may be hard to test, or hard 
>> to analyze when they break, may choose to maintain careful control over what 
>> versions of their upstream dependencies they recommend -- even if 
>> conceptually they regularly update those recommendations following the true 
>> semantic version specification of the dependency.
>> 
>> Dependency locking w.r.t. deployment
>> 
>> When stabilizing a release for deployment, or building a version of a 
>> package for deployment, it is important to be able to lock down the exact 
>> versions of 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Daniel Dunbar via swift-evolution]

> On Oct 13, 2016, at 11:16 PM, Daniel Duan via swift-build-dev 
>  wrote:
> 
> This is a familiar feature among package managers and has been proven useful 
> in practice. Two points regarding the proposed text:
> 
> 1. “honor” is mis-spelled in “weird queen’s dialect”.
> 2. SHA/man-in-the-middle attack section needs either more detailed 
> explanation/reference materials, or shouldn’t be included at all (assuming 
> when the hash feature gets introduced, there’d be another proposal for it).

I agree this needs more discussion, but given that we aren't proposing anything 
here, I feel like it is just a shoutout to an area for future development. What 
is the harm of having it there?

One of the possible values of having it in is jumpstarting the actual 
conversation on the topic!

 - Daniel

> 
> Overall strong +1.
>> On Oct 13, 2016, at 11:01 PM, Ankit Aggarwal via swift-evolution 
>> > wrote:
>> 
>> Hi,
>> 
>> We're proposing version pinning feature in Swift Package Manager. The 
>> proposal is available here 
>> 
>>  and also in this email:
>> 
>> Feedback welcomed!
>> 
>> Thanks,
>> Ankit
>> 
>> 
>> 
>> Package Manager Version Pinning
>> Proposal: SE-
>> Author: Daniel Dunbar , Ankit Aggarwal 
>> 
>> Review Manager: TBD
>> Status: Discussion
>> Introduction
>> This is a proposal for adding package manager features to "pin" or "lock" 
>> package dependencies to particular versions.
>> 
>> Motivation
>> As used in this proposal, version pinning refers to the practice of 
>> controlling exactly which specific version of a dependency is selected by 
>> the dependency resolution algorithm, independent from the semantic 
>> versioning specification. Thus, it is a way of instructing the package 
>> manager to select a particular version from among all of the versions of a 
>> package which could be chosen while honoring the dependency constraints.
>> 
>> Terminology
>> 
>> We have chosen to use "pinning" to refer to this feature, over "lockfiles", 
>> since the term "lock" is already overloaded between POSIX file locks and 
>> locks in concurrent programming.
>> 
>> Philosophy
>> 
>> Our philosophy with regard to pinning is that we actively want to encourage 
>> packages to develop against the latest semantically appropriate versions of 
>> their dependencies, in order to foster rapid development amongst the 
>> ecosystem and strong reliance on the semantic versioning concept. Our design 
>> for version pinning is thus intended to be a feature for package authors and 
>> users to use in crafting specific workflows, not be a mechanism by which 
>> most of the packages in the ecosystem pin themselves to specific versions of 
>> each other.
>> 
>> Use Cases
>> 
>> Our proposal is designed to satisfy several different use cases for such a 
>> behavior:
>> 
>> Standardizing team workflows
>> 
>> When collaborating on a package, it can be valuable for team members (and 
>> continuous integration) to all know they are using the same exact version of 
>> dependencies, to avoid "works for me" situations.
>> 
>> This can be particularly important for certain kinds of open source projects 
>> which are actively being cloned by new users, and which want to have some 
>> measure of control around exactly which available version of a dependency is 
>> selected.
>> 
>> Difficult to test packages or dependencies
>> 
>> Complex packages which have dependencies which may be hard to test, or hard 
>> to analyze when they break, may choose to maintain careful control over what 
>> versions of their upstream dependencies they recommend -- even if 
>> conceptually they regularly update those recommendations following the true 
>> semantic version specification of the dependency.
>> 
>> Dependency locking w.r.t. deployment
>> 
>> When stabilizing a release for deployment, or building a version of a 
>> package for deployment, it is important to be able to lock down the exact 
>> versions of dependencies in use, so that the resulting product can be 
>> exactly recreated later if necessary.
>> 
>> Proposed solution
>> We will introduce support for an optional new file Package.pins adjacent to 
>> the Package.swift manifest, called the "pins file". We will also introduce a 
>> number of new commands (see below) for maintaining the pins file.
>> 
>> This file will record the active version pin information for the package, 
>> including data such as the package identifier, the pinned version, and 
>> explicit information on the pinned version (e.g., the commit hash/SHA for 
>> the resolved tag).
>> 
>> The exact file format is unspecified/implementation defined, however, in 
>> practice it will be a JSON data file.
>> 
>> This file may be checked into SCM by the user, so that 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[Eloy Durán via swift-evolution]

I cannot agree more with Orta.

> It drives the user away from taking full advantage of semantic versioning.

While I ideally subscribe to this thought, the truth of the matter is that this 
has proven unreliable on multiple occasions. Promoting the idea that it works 
is going to inevitably lead users into problems when they did not take explicit 
user action to update dependencies.

Semantic versioning is a great tool when you decide to sit down and explicitly 
update dependencies.

> We think it will be good for the package ecosystem if such a restriction is 
> not the default behavior and that this design will lead to faster discovery 
> of bugs and fixes in the upstream.

Again, ideally I would subscribe to this, but the truth is that when people are 
working on their projects they *really really really* do *not* like their 
builds breaking, especially not when it’s seemingly caused outside of their own 
fault.

In the end, it comes down to whether you prioritise surfacing bugs (long term 
happiness) over user-experience (short term happiness). As a dependency manager 
author, I can tell you that I agree with your ideals and would want to choose 
the former, but as a user of dependency managers I would choose UX and Getting 
(The) Things (I Really Wanted To Do) Done over surfacing bugs *any* day.

[A.]Eloy Durán
Artsy 

e...@artsy.net   Twitter 

GitHub 
> On 14 Oct 2016, at 09:29, orta therox via swift-build-dev 
>  wrote:
> 
> Please don’t make this a separate command, it should ideally be created at 
> the end of an build (when there isn’t one already) or an update of your 
> dependencies - most people will be expecting to get the same set of 
> dependencies as the rest of their team. This pattern makes that harder.
> 
> NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot 
> of time trying to keep that up to date for our npm projects. Facebook made a 
> replacement for NPM with mainly the  feature of “always locking” in yarn 
>  and I’d expect that to take a lot of the JS mindshare 
> on this one feature alone.
> 
> -- 
> 
> [A.]  Orta Therox
> 
>> w/ Artsy CocoaPods  / CocoaDocs 
>>  / GIFs.app 
>> 
>> @orta  / orta.github.com 
>> Artsy is totally hiring iOS Devs  ATM
> 
>> On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev 
>> > wrote:
>> 
>> Hi,
>> 
>> We're proposing version pinning feature in Swift Package Manager. The 
>> proposal is available here 
>> 
>>  and also in this email:
>> 
>> Feedback welcomed!
>> 
>> Thanks,
>> Ankit
>> 
>> 
>> 
>> Package Manager Version Pinning
>> Proposal: SE-
>> Author: Daniel Dunbar , Ankit Aggarwal 
>> 
>> Review Manager: TBD
>> Status: Discussion
>> Introduction
>> This is a proposal for adding package manager features to "pin" or "lock" 
>> package dependencies to particular versions.
>> 
>> Motivation
>> As used in this proposal, version pinning refers to the practice of 
>> controlling exactly which specific version of a dependency is selected by 
>> the dependency resolution algorithm, independent from the semantic 
>> versioning specification. Thus, it is a way of instructing the package 
>> manager to select a particular version from among all of the versions of a 
>> package which could be chosen while honoring the dependency constraints.
>> 
>> Terminology
>> 
>> We have chosen to use "pinning" to refer to this feature, over "lockfiles", 
>> since the term "lock" is already overloaded between POSIX file locks and 
>> locks in concurrent programming.
>> 
>> Philosophy
>> 
>> Our philosophy with regard to pinning is that we actively want to encourage 
>> packages to develop against the latest semantically appropriate versions of 
>> their dependencies, in order to foster rapid development amongst the 
>> ecosystem and strong reliance on the semantic versioning concept. Our design 
>> for version pinning is thus intended to be a feature for package authors and 
>> users to use in crafting specific workflows, not be a mechanism by which 
>> most of the packages in the ecosystem pin themselves to specific versions of 
>> each other.
>> 
>> Use Cases
>> 
>> Our proposal is designed to satisfy several different use cases for such a 
>> behavior:
>> 
>> Standardizing team workflows
>> 
>> When collaborating on a package, it can be valuable for team 

Re: [swift-evolution] [swift-build-dev] Proposal: Package Manager Version Pinning

[orta therox via swift-evolution]

Please don’t make this a separate command, it should ideally be created at the 
end of an build (when there isn’t one already) or an update of your 
dependencies - most people will be expecting to get the same set of 
dependencies as the rest of their team. This pattern makes that harder.

NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot of 
time trying to keep that up to date for our npm projects. Facebook made a 
replacement for NPM with mainly the  feature of “always locking” in yarn 
 and I’d expect that to take a lot of the JS mindshare on 
this one feature alone.

-- 

[A.]Orta Therox

> w/ Artsy CocoaPods  / CocoaDocs 
>  / GIFs.app 
> 
> @orta  / orta.github.com 
> Artsy is totally hiring iOS Devs  ATM

> On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev 
>  wrote:
> 
> Hi,
> 
> We're proposing version pinning feature in Swift Package Manager. The 
> proposal is available here 
> 
>  and also in this email:
> 
> Feedback welcomed!
> 
> Thanks,
> Ankit
> 
> 
> 
> Package Manager Version Pinning
> Proposal: SE-
> Author: Daniel Dunbar , Ankit Aggarwal 
> 
> Review Manager: TBD
> Status: Discussion
> Introduction
> This is a proposal for adding package manager features to "pin" or "lock" 
> package dependencies to particular versions.
> 
> Motivation
> As used in this proposal, version pinning refers to the practice of 
> controlling exactly which specific version of a dependency is selected by the 
> dependency resolution algorithm, independent from the semantic versioning 
> specification. Thus, it is a way of instructing the package manager to select 
> a particular version from among all of the versions of a package which could 
> be chosen while honoring the dependency constraints.
> 
> Terminology
> 
> We have chosen to use "pinning" to refer to this feature, over "lockfiles", 
> since the term "lock" is already overloaded between POSIX file locks and 
> locks in concurrent programming.
> 
> Philosophy
> 
> Our philosophy with regard to pinning is that we actively want to encourage 
> packages to develop against the latest semantically appropriate versions of 
> their dependencies, in order to foster rapid development amongst the 
> ecosystem and strong reliance on the semantic versioning concept. Our design 
> for version pinning is thus intended to be a feature for package authors and 
> users to use in crafting specific workflows, not be a mechanism by which most 
> of the packages in the ecosystem pin themselves to specific versions of each 
> other.
> 
> Use Cases
> 
> Our proposal is designed to satisfy several different use cases for such a 
> behavior:
> 
> Standardizing team workflows
> 
> When collaborating on a package, it can be valuable for team members (and 
> continuous integration) to all know they are using the same exact version of 
> dependencies, to avoid "works for me" situations.
> 
> This can be particularly important for certain kinds of open source projects 
> which are actively being cloned by new users, and which want to have some 
> measure of control around exactly which available version of a dependency is 
> selected.
> 
> Difficult to test packages or dependencies
> 
> Complex packages which have dependencies which may be hard to test, or hard 
> to analyze when they break, may choose to maintain careful control over what 
> versions of their upstream dependencies they recommend -- even if 
> conceptually they regularly update those recommendations following the true 
> semantic version specification of the dependency.
> 
> Dependency locking w.r.t. deployment
> 
> When stabilizing a release for deployment, or building a version of a package 
> for deployment, it is important to be able to lock down the exact versions of 
> dependencies in use, so that the resulting product can be exactly recreated 
> later if necessary.
> 
> Proposed solution
> We will introduce support for an optional new file Package.pins adjacent to 
> the Package.swift manifest, called the "pins file". We will also introduce a 
> number of new commands (see below) for maintaining the pins file.
> 
> This file will record the active version pin information for the package, 
> including data such as the package identifier, the pinned version, and 
> explicit information on the pinned version (e.g., the commit hash/SHA for the 
> resolved tag).
> 
> The exact file format is unspecified/implementation defined, however, in 
> practice it will be a JSON data file.
> 
> This file may be checked into SCM by the user, so that