I like the proposal. I have 1 concern. As a package author, like PromiseKit <https://github.com/mxcl/PromiseKit> I often have a need to fix some of my dependencies and publish it to my package users.
With the current proposal I can do it: - Advise it in README, which is not really a solution done by swift package manager. - update Manifest.swift file. There is no convent way to do it now. I would love to have some functionality to make it simpler for me to specify (modify Manifest.swift or smt else) that Dependency X should: - use specific tag X - use commit X But maybe that should be part of another proposal. The lock file would perfectly solve the problems for developing projects that uses packages. - Kostiantyn > On 20 Mar 2016, at 06:07, Daniel Dunbar via swift-build-dev > <[email protected]> wrote: > > My thoughts on this proposal: > > 1. I agree with some of the other comments that I would prefer the version > file be adjacent to "Package.swift". When the Packages directory isn't being > checked in, I really would like to think of it as an "implementation detail" > and not embed functionality inside it that would make it hard to remove or > change in the future. > > 2. I like VersionLocks.json well enough, but would like to see a discussion > about possible alternatives. My personal proposal (in line with #1) is to use > "PackageVersions.json" which has a nice agreement with Package.swift and > would mean two common metadata files show up adjacent. I don't really want to > bike shed on the name, but I suspect whatever we pick first will last for a > while so I would at least like to review the various alternatives. I also > will throw out that my personal opinion is we don't need to pick a name that > bears much resemblance with existing terminology, whatever we pick will > eventually become "the standard" for the SwiftPM ecosystem so I would prefer > to pick the most-descriptive-possible name up front, not one that alludes to > the same concept in other systems. > > 3. I like the terminology section here, I almost feel like we should adopt > that as official terminology in our documentation (which I don't think we > have yet, correct me if I am wrong). > > 4. I would like it if the lock file recorded the exact SHA it received, and > validate that when retrieving. This helps protect users against MITM attacks > or unexpected changes if an upstream modifies a tag. It also can be used as > part of safety checks when migrating to an alternate repository host which is > expected to have the same content. > > 5. The "workflow - build" sections #2,3,4 are rather complicated. Is this > because the proposal is trying to work with existing Packages layouts, or > because the proposal is trying to handle the various variations of what the > user may have checked in inside the Packages subdirectory? > > 6. I wonder if we should be defining, as Eloy alludes to, two different > things: > - The version lock file, which defines the expected versions for the package > manager to use when it is doing package resolution. > - The package state file (in Packages.swift), which is used by the package > manager to track information on the Packages/ subdir state in order to > provide useful features primarily focused at the scenarios when the user is > modifying those files. > Currently it seems like a lot of the behaviors in the proposal are focused at > the latter case, but they feel like they should be decoupled problems to me. > > - Daniel > >> On Mar 17, 2016, at 11:23 AM, Max Howell via swift-evolution >> <[email protected] <mailto:[email protected]>> wrote: >> >> The following is a draft proposal, feedback welcome. >> >> ____________ >> SwiftPM Dependency Version Locking >> Proposal: SE-NNNN >> <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-dependency-lockfiles.md> >> Author(s): Ankit Agarwal <https://github.com/aciidb0mb3r>, Max Howell >> <https://github.com/mxcl> >> Status: Discussion >> Review manager: Rick Ballard >> Introduction >> This proposal seeks to declare a new, generated file >> Packages/VersionLocks.json that describes the exact state of a package’s >> dependency graph and then by default will be respected when executing most >> package manager commands. Thus it is considered a “version lock” for a >> package’s dependency sources. >> >> Swift-evolution thread >> <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html> >> Terminology >> A package refers to a published, versioned git repository designed to be >> consumed as a dependency by SwiftPM. >> A project refers to an end-user workspace that uses SwiftPM (via a >> Package.swift and swift build) fetching and building packages as part of its >> build >> Describing this distinction is required because both the above have the same >> form, but are used differently by an end-user. An end-user may publish >> packages, but will eventually consume those packages in a project. >> >> As justification for this confusion, it is considered a feature that >> projects can easily and trivially become packages when using SwiftPM. >> Encouraging a vibrant packaging ecosystem is one of our goals. >> >> Motivation >> In a vibrant packaging ecosystem, dependencies update continuously with >> bug-fixes and new features. A development team needs: >> >> To ensure they are all using the same versions of their dependencies for any >> given version-control commit. >> Ensure they are all using the same versions of the underlying Swift toolchain >> Be able to override or modify dependency specifications for the whole team >> for specific commits. >> Currently with SwiftPM it is possible to fulfill 1. by committing the >> sources of a package’s dependencies with the package itself, but this is not >> always desirable. There is no way to achieve 2. and 3. with SwiftPM alone. >> >> Additionally, there is not currently a way to know which version of Swift a >> package requires to build. At this time this situation is particularly >> precarious because Swift itself is not backwards compatible. As a Swift >> developer at the very least recording which Swift version a package was >> built with by the package developer is essential information in order to >> assess a package's suitability. Practically the package manager could in the >> future use this information to aid an end-user or even fix the problem when >> packages fail to compile. >> >> Proposed Solution >> A file: Packages/VersionLocks.json will be created alongside the >> Package.swift file. Its contents will describe: >> >> The URL and versions of cloned dependencies >> An inline diff of any local modifications made to those packages relative to >> their pristine cloned states >> The exact version of the Swift toolchain used as part of the last successful >> build of the package >> This file is generated by SwiftPM. >> >> This file should be checked-in with projects. >> >> This file is generated and should not be edited by users. If the file is >> edited by users the behavior is undefined. >> >> This file should be checked-in with packages designed for consumption in >> projects, however SwiftPM will not use the checkout files of dependencies >> when determining a project’s dependency graph (this would make dependency >> graphs much less likely to resolve due to overly strict versioning >> requirements). In the future we may choose to make it possible for end-users >> to attempt to build a package using all checkout files since in certain >> deployment scenarios where an exact graph has already been tested, this is a >> solid reliabiity feature. >> >> Any local, modifications made to the clones in Packages are recorded in >> Packages/VersionLocks.json as part of the flow described in the next >> section. Modifications here means: changes to git remotes and the git-ref of >> the checked-out HEAD. >> >> Detailed Design >> In a fresh clone that does not contain a Packages directory swift build will >> determine the dependency graph, clone the packages into Packages and >> generate a Packages/VersionLocks.json file. >> >> The user can now step into the Packages directory and modify package >> sources. If the user then runs swift build again the package manager will >> error out: >> >> error: dependency sources have been modified >> execute `swift build --lock` or `swift build --ignore-lock` >> It is an error to build against an unlocked dependency graph, but to >> facilitate fixing bugs etc. an ignore flag can be specified. >> >> When swift build --lock is specified the package manager regenerates the >> lockfile detailing the active git remote and the SHA that is checked-out. >> >> Every time swift build completes a build the lockfile is updated (if >> necessary) recording the current version of the Swift toolchain that >> achieved the build. >> >> Packages/VersionLocks.json >> >> The exact design of the contents of this file will be explored during >> iterative development, but here is a possible example: >> >> json { "packages": [ { "clone": "Packages/PromiseKit-3.0.3", "origin": >> "https://github.com/mxcl/PromiseKit <https://github.com/mxcl/PromiseKit>" >> "ref": "3.0.3" }, { "clone": "Packages/Alamofire-1.2.3", "origin": >> "https://github.com/a-fork-somewhere/Alamofire >> <https://github.com/a-fork-somewhere/Alamofire>" "ref": "crucial-fix" }, { >> "clone": "Packages/Quick-1.2.3", "origin": "https://github.com/Quick/Quick >> <https://github.com/Quick/Quick>" "ref": "1.2.3" } ] } >> >> Workflow — Regular Build >> >> User runs swift build >> If Packages/ contains clones and a VersionLocks.jsonSwiftPM skips to 7. >> If Packages/ contains clones and no VersionLocks.json the lockfile is >> generated from the clones >> If Packages/ contains checked out sources without git information and no >> VersionLocks.json SwiftPM fetches the git information and provided there is >> no diff, generates the Lockfile, if there is variation it is an error * >> If Packages/VersionLocks.json is present its dependency graph is used >> If Packages doesn't exist or is empty the dependency graph is resolved, >> packages are cloned and the Lockfile is generated >> Build, if Packages are missing because we skipped from 2. the build will >> error, it is the user's responsibility to instruct SwiftPM to --update or to >> fix their dependency graph some other way. >> >> This scenario is so users can check in their complete dependency sources to >> their tree instead of / as well as the VersionLocks.json file: a situation >> which sometimes is necessary if your dependencies are removed from their >> third party online location, etc. >> >> Workflow — Making Modifications >> >> User makes local modification to a dependency’s sources >> User runs swift build >> swift build errors out. >> User must either lock the graph or run with --ignore-lock >> The error-out is likely to be considered tedious by users, however we >> consider it important that users are made aware and forced to act when they >> modify their dependencies and thus are exposing their team/users to >> so-called “dependency hell”. >> >> Runing swift build --lock regenerates the lockfile, but does not build. >> >> Modifications must be committed. This means that if the modifications are >> not uploaded to a location accessible to the rest of the team they will fail >> to build when they update their checkouts. >> >> The package manager could check for this by asking git if the specified >> origin has the current locked ref and error out as appropriate. >> >> Workflow — Overriding Packages >> >> User steps into a Package directory eg. Packages/Foo-1.2.3 >> User changes the origin of Foo to their own fork >> User alters HEAD to point to a fix in their own fork >> swift build errors out. >> User must either lock the graph or run with --ignore-lock >> Running swift build --lock regenerates the lockfile, the new origin and tag >> is stored. Thus a fresh clone of this project would use these overrides. >> >> It is important to note that this workflow will not be respected for >> dependencies, only for projects. >> >> If a package author requires an override they have a few options: >> >> Change the Package.swift dependency specification. This should only be done >> as a last resort, for example, a critical bug must be fixed in a dependency >> and that dependency author is not being responsive. It is up to the Package >> author to ensure this scenario goes well. SwiftPM itself wants to guard >> against these conditions with our proposed “publish & lint” step that >> validates such decisions before signing a published package tag. But we are >> not there yet and thus package authors should be responsible. >> Advise end-users in a package README that they should override the >> dependency themselves. >> 2 is preferred, but 1 will happen. We consider it our responsibility to >> develop tooling that makes 1. safe or unnecessary, but we are not there yet. >> >> Workflow — Updating Packages >> >> SwiftPM has no update mechanism yet, but once it does running swift build >> --update will fetch the latest versions of all dependencies and update the >> lockfile. >> >> Impact on existing code >> This proposal will have no impact on existing code. >> >> Alternatives Considered >> One alternative is to allow mentioning refs in manifest file while declaring >> a dependency but as discussed in this >> <http://markdownlivepreview.com/%22https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/> >> thread it might not be the best idea. >> >> Using Git submodules for this feature was considered. However something >> additionally would be required to specify swift version and record local >> diffs. Also this would lock us into git, and despite the fact that currently >> we only use git, we have not yet ruled out supporting other version control >> systems. >> >> _______________________________________________ >> swift-evolution mailing list >> [email protected] <mailto:[email protected]> >> https://lists.swift.org/mailman/listinfo/swift-evolution > > _______________________________________________ > swift-build-dev mailing list > [email protected] > https://lists.swift.org/mailman/listinfo/swift-build-dev
_______________________________________________ swift-evolution mailing list [email protected] https://lists.swift.org/mailman/listinfo/swift-evolution
