> On Apr 25, 2016, at 8:15 PM, Samantha John <s...@gethopscotch.com> wrote: > > On Mon, Apr 25, 2016 at 12:11 AM, Daniel Dunbar <daniel_dun...@apple.com > <mailto:daniel_dun...@apple.com>> wrote: > >> On Apr 24, 2016, at 3:19 PM, Samantha John via swift-dev >> <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote: >> >> Hello List (cc/Jordan), >> >> At a high level: Brian and I are looking into contributing to incremental >> compilation in Swift. Right now we're trying to do an incremental compile by >> invoking swiftc. >> >> Our understanding so far: >> - swiftc takes a list of input files. >> - If swiftc is invoked with '-incremental', it requires an >> '-output-file-map' option to also be passed. We assume this is used to >> determine where intermediate dependency files go. >> - Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of >> files Xcode generates when building Swift projects, we determined >> '-output-file-map' to be a JSON file (which is parsed by the llvm yaml >> parser? >> https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100 >> >> <https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100>). >> We tried writing our own, this is the shortest version that the compiler >> didn't error on: >> >> ``` >> { >> "": { >> "swift-dependencies": "MyModule-main.swiftdeps" >> } >> } >> ``` >> >> - When compiling with the above file ('swiftc -incremental -output-file-map >> OurFileMap.json *.swift'), swiftc writes to MyModule-main.swiftdeps. The end >> result looks like this: >> >> ``` >> version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift >> a2cf18ba2f)" >> options: "9277a78155e85019ce36a3c52e9f3f02" >> build_time: [514847765, 412105000] >> inputs: >> "Class1.swift": [514841531, 0] >> "Class2.swift": [514844635, 0] >> "main.swift": [514841821, 0] >> ``` >> >> - Where the [xxx, yyy] are timestamps with the first number representing >> seconds, the second nanoseconds. >> (https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221 >> >> <https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221>) >> >> - We invoked 'swiftc -incremental -output-file-map OurFileMap.json *.swift >> -parseable-output -save-temps' to show us the paths to the generated >> .swiftdeps files. We assume that to get incremental compiles to work for us, >> we'd need to pass these generated .swiftdeps files' paths to the compiler >> somehow. We can't figure out how to do this, so this is the point where our >> incremental compile fails: >> https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348 >> >> <https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348> > You don't need to point the compiler at the swiftdeps files, it reuses the > ones passed in the output file map if -incremental is passed. > > You didn't mention it (I don't think) but you also need to pass > -emit-dependencies, however it sounds like you are doing this. > >> This is as far as we got. Would super appreciate if anyone on the list could >> point us in the right direction from here, especially for the following >> questions: >> >> 1. This would probably be easier if we could find the right test case. We >> poked around in test/Driver/Dependencies which had a bunch of tests around >> reading the .swiftdeps files, but we couldn't find tests that demonstrated >> how incremental compilation worked at a high level. >> >> 2. '-output-file-map': Is this file meant to be written by hand, or is there >> a part of the swift compiler that writes this for you? > > It is meant to be written by the build system which invokes Swift. See also: > > https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206 > > <https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206> > which is what the Swift package manager uses (which supports incremental > compiles). > > This sounds perfect. I tried to install the llbuild but got stuck on > filecheck (and saw that you are trying to add it to the toolchain: > https://github.com/apple/swift-llbuild/pull/22/files#r61007286 > <https://github.com/apple/swift-llbuild/pull/22/files#r61007286>) Is there an > executable of llbuild I could download, or do you have any basic instructions > for what to do about file check?
You shouldn't need to download llbuild, you probably want to go the xcodebuild route... > > > Your best bet is to take the exact command line used by xcodebuild, and then > invoke swiftc with that to replicate the incremental build. If you run with > -v you should be able to see exactly what files get built > > I tried running xcodebuild with the verbose flag but wasn't getting much > useful. Is that what you meant here? -v didn't seem to be an option. I wasn't very clear here. What you need to do is run xcodebuild from the command line, and then find the top-level line which is compiling the module you are about (this will be the line with `swiftc ... -incremental ...` in it but not including `-frontend). That is the command you can run with `-v`. - Daniel > > Thanks so much, > Sam > > > HTH! > - Daniel > >> >> 3. Specifying paths for .swiftdeps: We had assumed this was done based on >> the '-output-file-map', but writing the paths we wanted manually did not >> seem to work. Any tips? >> >> Thanks so much! >> Sam and Brian >> >> On Wed, Apr 13, 2016 at 5:18 PM, Samantha John <s...@gethopscotch.com >> <mailto:s...@gethopscotch.com>> wrote: >> Hi Jordan, >> >> The thing that sticks out in the dependency analysis is the treatment of >> external dependencies. The entire module has the same list of external >> dependencies which causes a lot of needless recompiles- especially if you >> start with a large swift project and slowly start to move things into >> modules. >> >> So to me the lowest hanging fruit would be to only mark files for >> recompilation that explicitly import the external dependency. This seems >> pretty safe since you can't compile unless the dependency is explicitly >> imported. Has anyone on the list tried this before? >> >> A second idea would be to consider a file as changed only if its build >> artifact actually changes. Obviously, we'd have to actually build the file >> to figure this out, so we wouldn't have the same level of parallelism >> initially. Perhaps if it was an optional compiler flag this would be more >> palatable? Also wondering if anyone has tried something along these lines. >> >> Thanks! >> >> >> George- The bridge between objective c and swift between is definitely a >> choke point. We've been able to mitigate objective-c recompiles somewhat by >> limiting our imports of swift into objective-c. We've even gone so far as to >> make wrapper classes in objective c around some of our most commonly used >> swift classes so as not to import swift. >> >> It's also very true that changing an objective c .h file that is imported >> into the bridging header will trigger massive recompiles. As more of our app >> has transitioned to swift this has been less of an issue. Most of the >> problems at this point have to do with recompiling a large portion of our >> swift code due to small changes in unrelated parts of our other swift code. >> >> >> >> >> >> Get the latest from Hopscotch! >> >> Sign-up for our newsletter <http://eepurl.com/Ui0eX> >> >> On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.li...@gmail.com >> <mailto:gwk.li...@gmail.com>> wrote: >> Hey Sam, >> >> One thought: if you have an app with mixed objc and swift code, then the >> app-bridge.h and app-swift.h files might be creating massive choke points in >> your dependency graph. I have no idea how optimized the bridging >> functionality is but it has always seemed like a potentially weak part of >> the dependency management. I imagine that with an objc half and a swift half >> of the code base, every time you make a change in a swift file you trigger >> recompilation of the objc half, and vice versa. I'd love to hear from the >> core team to what extent this is true! >> >> George >> >> >> >>> On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev >>> <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote: >>> >>> Thank you Jordan! This is a great starting off point. >>> >>> I'm thinking about proposing a "strict import" mode in swift: A compile >>> flag that when turned on would require you to explicitly import any file >>> that contained a dependency you needed (like in objective-c). >>> >>> I'm going to spend more time looking over the docs and the output logs to >>> see if this would be a feasible. If anyone has opinions or insights into >>> this I would love to hear from you. >>> >>> Sam >>> >>> On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_r...@apple.com >>> <mailto:jordan_r...@apple.com>> wrote: >>> Hi, Sam. I don't think we currently have a good answer for this built into >>> xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds would >>> be fast enough that it wouldn't matter! That's obviously not where we are.) >>> >>> Since '-debug-time-function-bodies' is now public knowledge, I'll share >>> another one of our debugging flags, '-driver-show-incremental'. You can add >>> this to your "Other Swift Flags". The output isn't very detailed, though: >>> >>> Queuing Tree.swift (initial) >>> Queuing AdventureScene.swift (initial) >>> Queuing AdventureScene.swift because of dependencies discovered later >>> Queuing AppDelegate.swift because of dependencies discovered later >>> Queuing ChaseArtificialIntelligence.swift because of dependencies >>> discovered later >>> Queuing Character.swift because of dependencies discovered later >>> Queuing SpawnArtificialIntelligence.swift because of dependencies >>> discovered later >>> Queuing Goblin.swift because of dependencies discovered later >>> Queuing Cave.swift because of dependencies discovered later >>> Queuing AdventureSceneOSXEvents.swift because of dependencies discovered >>> later >>> Queuing HeroCharacter.swift because of dependencies discovered later >>> Queuing EnemyCharacter.swift because of dependencies discovered later >>> Queuing Boss.swift because of dependencies discovered later >>> Queuing SharedAssetManagement.swift because of dependencies discovered later >>> Queuing Warrior.swift because of dependencies discovered later >>> Queuing Archer.swift because of dependencies discovered later >>> Queuing Player.swift because of dependencies discovered later >>> Queuing ArtificialIntelligence.swift because of dependencies discovered >>> later >>> >>> In this case, I took a version of the Adventure sample project and modified >>> "Tree.swift"; that triggered recompilation of several other files. >>> Unfortunately this view doesn't tell you how they're related, only which >>> ones are actually getting rebuilt. >>> >>> The next step (and moving into the territory of "working on Swift" rather >>> than just "trying to figure out why it's repeating work") would be to look >>> at the "swiftdeps" files stored in your DerivedData folder. These are >>> currently just YAML files describing what Swift thinks the file depends on, >>> as well as what will trigger rebuilding of other files. This is intended to >>> be a conservative estimate, since not recompiling something would result in >>> an invalid binary. (Unfortunately I say "intended" because there are known >>> bugs; fortunately, archive builds are always clean builds anyway.) >>> >>> There's a document in the Swift repo describing the logic behind Swift's >>> dependency analysis: >>> https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst >>> <https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst>. >>> The one thing that's not in there is the notion of changes that don't >>> affect other files at all. This is accomplished by computing a hash of all >>> the tokens that could affect other files, and seeing if that hash has >>> changed. >>> >>> We definitely have room for improvement here. >>> >>> Jordan >>> >>> >>>> On Mar 31, 2016, at 11:24 , Samantha John via swift-dev >>>> <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote: >>>> >>>> I have a large project (308 swift files, 441 objective c, 66k lines of >>>> code) where incremental builds can be extremely slow. I'm trying to do >>>> some profiling to figure out what type of things cause large scale >>>> recompiles. The problem is that I can't find a good way of telling which >>>> files get recompiled on an incremental build and which do not. It seems >>>> like files that are not recompiled still get listed in xcode, but the >>>> compiler just passes over them really fast. >>>> >>>> Does anyone know if xctool or xcodebuild has this type of functionality? >>>> Or is there some other way to get this info? >>>> >>>> Thank you, >>>> Sam >>>> _______________________________________________ >>>> swift-dev mailing list >>>> swift-dev@swift.org <mailto:swift-dev@swift.org> >>>> https://lists.swift.org/mailman/listinfo/swift-dev >>>> <https://lists.swift.org/mailman/listinfo/swift-dev> >>> >>> >>> _______________________________________________ >>> swift-dev mailing list >>> swift-dev@swift.org <mailto:swift-dev@swift.org> >>> https://lists.swift.org/mailman/listinfo/swift-dev >>> <https://lists.swift.org/mailman/listinfo/swift-dev> >> >> >> >> _______________________________________________ >> swift-dev mailing list >> swift-dev@swift.org <mailto:swift-dev@swift.org> >> https://lists.swift.org/mailman/listinfo/swift-dev >> <https://lists.swift.org/mailman/listinfo/swift-dev>
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
