On Mon, Apr 25, 2016 at 12:11 AM, Daniel Dunbar <daniel_dun...@apple.com> wrote:
> > On Apr 24, 2016, at 3:19 PM, Samantha John via swift-dev < > swift-dev@swift.org> wrote: > > Hello List (cc/Jordan), > > At a high level: Brian and I are looking into contributing to incremental > compilation in Swift. Right now we're trying to do an incremental compile > by invoking swiftc. > > Our understanding so far: > - swiftc takes a list of input files. > - If swiftc is invoked with '-incremental', it requires an > '-output-file-map' option to also be passed. We assume this is used to > determine where intermediate dependency files go. > - Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of > files Xcode generates when building Swift projects, we determined > '-output-file-map' to be a JSON file (which is parsed by the llvm yaml > parser? > https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100). > We tried writing our own, this is the shortest version that the compiler > didn't error on: > > ``` > { > "": { > "swift-dependencies": "MyModule-main.swiftdeps" > } > } > ``` > > - When compiling with the above file ('swiftc -incremental > -output-file-map OurFileMap.json *.swift'), swiftc writes to > MyModule-main.swiftdeps. The end result looks like this: > > ``` > version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift > a2cf18ba2f)" > options: "9277a78155e85019ce36a3c52e9f3f02" > build_time: [514847765, 412105000] > inputs: > "Class1.swift": [514841531, 0] > "Class2.swift": [514844635, 0] > "main.swift": [514841821, 0] > ``` > > - Where the [xxx, yyy] are timestamps with the first number representing > seconds, the second nanoseconds. ( > https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221 > ) > > - We invoked 'swiftc -incremental -output-file-map OurFileMap.json *.swift > -parseable-output -save-temps' to show us the paths to the generated > .swiftdeps files. We assume that to get incremental compiles to work for > us, we'd need to pass these generated .swiftdeps files' paths to the > compiler somehow. We can't figure out how to do this, so this is the point > where our incremental compile fails: > https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348 > > > You don't need to point the compiler at the swiftdeps files, it reuses the > ones passed in the output file map if -incremental is passed. > > You didn't mention it (I don't think) but you also need to pass > -emit-dependencies, however it sounds like you are doing this. > > This is as far as we got. Would super appreciate if anyone on the list > could point us in the right direction from here, especially for the > following questions: > > 1. This would probably be easier if we could find the right test case. We > poked around in test/Driver/Dependencies which had a bunch of tests around > reading the .swiftdeps files, but we couldn't find tests that demonstrated > how incremental compilation worked at a high level. > > 2. '-output-file-map': Is this file meant to be written by hand, or is > there a part of the swift compiler that writes this for you? > > > It is meant to be written by the build system which invokes Swift. See > also: > > https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206 > which is what the Swift package manager uses (which supports incremental > compiles). > This sounds perfect. I tried to install the llbuild but got stuck on filecheck (and saw that you are trying to add it to the toolchain: https://github.com/apple/swift-llbuild/pull/22/files#r61007286) Is there an executable of llbuild I could download, or do you have any basic instructions for what to do about filecheck? > Your best bet is to take the exact command line used by xcodebuild, and > then invoke swiftc with that to replicate the incremental build. If you run > with -v you should be able to see exactly what files get built > I tried running xcodebuild with the verbose flag but wasn't getting much useful. Is that what you meant here? -v didn't seem to be an option. Thanks so much, Sam > HTH! > - Daniel > > > 3. Specifying paths for .swiftdeps: We had assumed this was done based on > the '-output-file-map', but writing the paths we wanted manually did not > seem to work. Any tips? > > Thanks so much! > Sam and Brian > > On Wed, Apr 13, 2016 at 5:18 PM, Samantha John <s...@gethopscotch.com> > wrote: > >> Hi Jordan, >> >> The thing that sticks out in the dependency analysis is the treatment of >> external dependencies. The entire module has the same list of external >> dependencies which causes a lot of needless recompiles- especially if you >> start with a large swift project and slowly start to move things into >> modules. >> >> So to me the lowest hanging fruit would be to only mark files for >> recompilation that explicitly import the external dependency. This seems >> pretty safe since you can't compile unless the dependency is explicitly >> imported. Has anyone on the list tried this before? >> >> A second idea would be to consider a file as changed only if its build >> artifact actually changes. Obviously, we'd have to actually build the file >> to figure this out, so we wouldn't have the same level of parallelism >> initially. Perhaps if it was an optional compiler flag this would be more >> palatable? Also wondering if anyone has tried something along these lines. >> >> Thanks! >> >> >> George- The bridge between objective c and swift between is definitely a >> choke point. We've been able to mitigate objective-c recompiles somewhat by >> limiting our imports of swift into objective-c. We've even gone so far as >> to make wrapper classes in objective c around some of our most commonly >> used swift classes so as not to import swift. >> >> It's also very true that changing an objective c .h file that is imported >> into the bridging header will trigger massive recompiles. As more of our >> app has transitioned to swift this has been less of an issue. Most of the >> problems at this point have to do with recompiling a large portion of our >> swift code due to small changes in unrelated parts of our other swift code. >> >> >> >> >> >> Get the latest from Hopscotch! >> >> Sign-up for our newsletter <http://eepurl.com/Ui0eX> >> >> On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.li...@gmail.com> wrote: >> >>> Hey Sam, >>> >>> One thought: if you have an app with mixed objc and swift code, then the >>> app-bridge.h and app-swift.h files might be creating massive choke points >>> in your dependency graph. I have no idea how optimized the bridging >>> functionality is but it has always seemed like a potentially weak part of >>> the dependency management. I imagine that with an objc half and a swift >>> half of the code base, every time you make a change in a swift file you >>> trigger recompilation of the objc half, and vice versa. I'd love to hear >>> from the core team to what extent this is true! >>> >>> George >>> >>> >>> >>> On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev < >>> swift-dev@swift.org> wrote: >>> >>> Thank you Jordan! This is a great starting off point. >>> >>> I'm thinking about proposing a "strict import" mode in swift: A compile >>> flag that when turned on would require you to explicitly import any file >>> that contained a dependency you needed (like in objective-c). >>> >>> I'm going to spend more time looking over the docs and the output logs >>> to see if this would be a feasible. If anyone has opinions or insights into >>> this I would love to hear from you. >>> >>> Sam >>> >>> On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_r...@apple.com> >>> wrote: >>> >>>> Hi, Sam. I don't think we currently have a good answer for this built >>>> into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds >>>> would be fast enough that it wouldn't matter! That's obviously not where we >>>> are.) >>>> >>>> Since '-debug-time-function-bodies' is now public knowledge, I'll share >>>> another one of our debugging flags, '-driver-show-incremental'. You can add >>>> this to your "Other Swift Flags". The output isn't very detailed, though: >>>> >>>> Queuing Tree.swift (initial) >>>> Queuing AdventureScene.swift (initial) >>>> Queuing AdventureScene.swift because of dependencies discovered later >>>> Queuing AppDelegate.swift because of dependencies discovered later >>>> Queuing ChaseArtificialIntelligence.swift because of dependencies >>>> discovered later >>>> Queuing Character.swift because of dependencies discovered later >>>> Queuing SpawnArtificialIntelligence.swift because of dependencies >>>> discovered later >>>> Queuing Goblin.swift because of dependencies discovered later >>>> Queuing Cave.swift because of dependencies discovered later >>>> Queuing AdventureSceneOSXEvents.swift because of dependencies >>>> discovered later >>>> Queuing HeroCharacter.swift because of dependencies discovered later >>>> Queuing EnemyCharacter.swift because of dependencies discovered later >>>> Queuing Boss.swift because of dependencies discovered later >>>> Queuing SharedAssetManagement.swift because of dependencies discovered >>>> later >>>> Queuing Warrior.swift because of dependencies discovered later >>>> Queuing Archer.swift because of dependencies discovered later >>>> Queuing Player.swift because of dependencies discovered later >>>> Queuing ArtificialIntelligence.swift because of dependencies discovered >>>> later >>>> >>>> In this case, I took a version of the Adventure sample project and >>>> modified "Tree.swift"; that triggered recompilation of several other files. >>>> Unfortunately this view doesn't tell you how they're related, only which >>>> ones are actually getting rebuilt. >>>> >>>> The next step (and moving into the territory of "working on Swift" >>>> rather than just "trying to figure out why it's repeating work") would be >>>> to look at the "swiftdeps" files stored in your DerivedData folder. These >>>> are currently just YAML files describing what Swift thinks the file depends >>>> on, as well as what will trigger rebuilding of other files. This is >>>> intended to be a conservative estimate, since *not* recompiling >>>> something would result in an invalid binary. (Unfortunately I say >>>> "intended" because there are known bugs; fortunately, archive builds are >>>> always clean builds anyway.) >>>> >>>> There's a document in the Swift repo describing the logic behind >>>> Swift's dependency analysis: >>>> https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst. >>>> The one thing that's *not* in there is the notion of changes that >>>> don't affect other files at all. This is accomplished by computing a hash >>>> of all the tokens that *could* affect other files, and seeing if that >>>> hash has changed. >>>> >>>> We definitely have room for improvement here. >>>> >>>> Jordan >>>> >>>> >>>> On Mar 31, 2016, at 11:24 , Samantha John via swift-dev < >>>> swift-dev@swift.org> wrote: >>>> >>>> I have a large project (308 swift files, 441 objective c, 66k lines of >>>> code) where incremental builds can be extremely slow. I'm trying to do some >>>> profiling to figure out what type of things cause large scale recompiles. >>>> The problem is that I can't find a good way of telling which files get >>>> recompiled on an incremental build and which do not. It seems like files >>>> that are not recompiled still get listed in xcode, but the compiler just >>>> passes over them really fast. >>>> >>>> Does anyone know if xctool or xcodebuild has this type of >>>> functionality? Or is there some other way to get this info? >>>> >>>> Thank you, >>>> Sam >>>> _______________________________________________ >>>> swift-dev mailing list >>>> swift-dev@swift.org >>>> https://lists.swift.org/mailman/listinfo/swift-dev >>>> >>>> >>>> >>> _______________________________________________ >>> swift-dev mailing list >>> swift-dev@swift.org >>> https://lists.swift.org/mailman/listinfo/swift-dev >>> >>> >>> >> > _______________________________________________ > swift-dev mailing list > swift-dev@swift.org > https://lists.swift.org/mailman/listinfo/swift-dev > > >
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
