> On Apr 25, 2016, at 8:15 PM, Samantha John <s...@gethopscotch.com> wrote:
> 
> On Mon, Apr 25, 2016 at 12:11 AM, Daniel Dunbar <daniel_dun...@apple.com 
> <mailto:daniel_dun...@apple.com>> wrote:
> 
>> On Apr 24, 2016, at 3:19 PM, Samantha John via swift-dev 
>> <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
>> 
>> Hello List (cc/Jordan),  
>> 
>> At a high level: Brian and I are looking into contributing to incremental 
>> compilation in Swift. Right now we're trying to do an incremental compile by 
>> invoking swiftc. 
>> 
>> Our understanding so far:
>> - swiftc takes a list of input files. 
>> - If swiftc is invoked with '-incremental', it requires an 
>> '-output-file-map' option to also be passed. We assume this is used to 
>> determine where intermediate dependency files go.
>> - Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of 
>> files Xcode generates when building Swift projects, we determined 
>> '-output-file-map' to be a JSON file (which is parsed by the llvm yaml 
>> parser? 
>> https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100
>>  
>> <https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100>).
>>  We tried writing our own, this is the shortest version that the compiler 
>> didn't error on:
>> 
>> ```
>> {
>>   "": {
>>     "swift-dependencies": "MyModule-main.swiftdeps"
>>   }
>> }
>> ```
>> 
>> - When compiling with the above file ('swiftc -incremental -output-file-map 
>> OurFileMap.json *.swift'), swiftc writes to MyModule-main.swiftdeps. The end 
>> result looks like this:
>> 
>> ```
>> version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift 
>> a2cf18ba2f)"
>> options: "9277a78155e85019ce36a3c52e9f3f02"
>> build_time: [514847765, 412105000]
>> inputs:
>>   "Class1.swift": [514841531, 0]
>>   "Class2.swift": [514844635, 0]
>>   "main.swift": [514841821, 0]
>> ```
>> 
>> - Where the [xxx, yyy] are timestamps with the first number representing 
>> seconds, the second nanoseconds. 
>> (https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221
>>  
>> <https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221>)
>> 
>> - We invoked 'swiftc -incremental -output-file-map OurFileMap.json *.swift 
>> -parseable-output -save-temps' to show us the paths to the generated 
>> .swiftdeps files. We assume that to get incremental compiles to work for us, 
>> we'd need to pass these generated .swiftdeps files' paths to the compiler 
>> somehow. We can't figure out how to do this, so this is the point where our 
>> incremental compile fails: 
>> https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348
>>  
>> <https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348>
> You don't need to point the compiler at the swiftdeps files, it reuses the 
> ones passed in the output file map if -incremental is passed.
> 
> You didn't mention it (I don't think) but you also need to pass 
> -emit-dependencies, however it sounds like you are doing this.
> 
>> This is as far as we got. Would super appreciate if anyone on the list could 
>> point us in the right direction from here, especially for the following 
>> questions:
>> 
>> 1. This would probably be easier if we could find the right test case. We 
>> poked around in test/Driver/Dependencies which had a bunch of tests around 
>> reading the .swiftdeps files, but we couldn't find tests that demonstrated 
>> how incremental compilation worked at a high level.
>> 
>> 2. '-output-file-map': Is this file meant to be written by hand, or is there 
>> a part of the swift compiler that writes this for you?
> 
> It is meant to be written by the build system which invokes Swift. See also:
>   
> https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206
>  
> <https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206>
> which is what the Swift package manager uses (which supports incremental 
> compiles).
> 
> This sounds perfect. I tried to install the llbuild but got stuck on 
> filecheck (and saw that you are trying to add it to the toolchain: 
> https://github.com/apple/swift-llbuild/pull/22/files#r61007286 
> <https://github.com/apple/swift-llbuild/pull/22/files#r61007286>) Is there an 
> executable of llbuild I could download, or do you have any basic instructions 
> for what to do about file check? 

You shouldn't need to download llbuild, you probably want to go the xcodebuild 
route...

> 
> 
> Your best bet is to take the exact command line used by xcodebuild, and then 
> invoke swiftc with that to replicate the incremental build. If you run with 
> -v you should be able to see exactly what files get built 
> 
>  I tried running xcodebuild with the verbose flag but wasn't getting much 
> useful. Is that what you meant here? -v didn't seem to be an option. 

I wasn't very clear here. What you need to do is run xcodebuild from the 
command line, and then find the top-level line which is compiling the module 
you are about (this will be the line with `swiftc ... -incremental ...` in it 
but not including `-frontend). That is the command you can run with `-v`.

 - Daniel

>  
> Thanks so much, 
> Sam
> 
> 
> HTH!
>  - Daniel
> 
>> 
>> 3. Specifying paths for .swiftdeps: We had assumed this was done based on 
>> the '-output-file-map', but writing the paths we wanted manually did not 
>> seem to work. Any tips?
>> 
>> Thanks so much!
>> Sam and Brian
>> 
>> On Wed, Apr 13, 2016 at 5:18 PM, Samantha John <s...@gethopscotch.com 
>> <mailto:s...@gethopscotch.com>> wrote:
>> Hi Jordan, 
>> 
>> The thing that sticks out in the dependency analysis is the treatment of 
>> external dependencies. The entire module has the same list of external 
>> dependencies which causes a lot of needless recompiles- especially if you 
>> start with a large swift project and slowly start to move things into 
>> modules. 
>> 
>> So to me the lowest hanging fruit would be to only mark files for 
>> recompilation that explicitly import the external dependency. This seems 
>> pretty safe since you can't compile unless the dependency is explicitly 
>> imported. Has anyone on the list tried this before? 
>> 
>> A second idea would be to consider a file as changed only if its build 
>> artifact actually changes. Obviously, we'd have to actually build the file 
>> to figure this out, so we wouldn't have the same level of parallelism 
>> initially. Perhaps if it was an optional compiler flag this would be more 
>> palatable? Also wondering if anyone has tried something along these lines.
>> 
>> Thanks! 
>> 
>> 
>> George- The bridge between objective c and swift between is definitely a 
>> choke point. We've been able to mitigate objective-c recompiles somewhat by 
>> limiting our imports of swift into objective-c. We've even gone so far as to 
>> make wrapper classes in objective c around some of our most commonly used 
>> swift classes so as not to import swift. 
>> 
>> It's also very true that changing an objective c .h file that is imported 
>> into the bridging header will trigger massive recompiles. As more of our app 
>> has transitioned to swift this has been less of an issue. Most of the 
>> problems at this point have to do with recompiling a large portion of our 
>> swift code due to small changes in unrelated parts of our other swift code. 
>> 
>> 
>> 
>> 
>> 
>> Get the latest from Hopscotch!
>> 
>> Sign-up for our newsletter <http://eepurl.com/Ui0eX>
>> 
>> On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.li...@gmail.com 
>> <mailto:gwk.li...@gmail.com>> wrote:
>> Hey Sam,
>> 
>> One thought: if you have an app with mixed objc and swift code, then the 
>> app-bridge.h and app-swift.h files might be creating massive choke points in 
>> your dependency graph. I have no idea how optimized the bridging 
>> functionality is but it has always seemed like a potentially weak part of 
>> the dependency management. I imagine that with an objc half and a swift half 
>> of the code base, every time you make a change in a swift file you trigger 
>> recompilation of the objc half, and vice versa. I'd love to hear from the 
>> core team to what extent this is true!
>> 
>> George
>> 
>> 
>> 
>>> On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev 
>>> <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
>>> 
>>> Thank you Jordan! This is a great starting off point.
>>> 
>>> I'm thinking about proposing a "strict import" mode in swift: A compile 
>>> flag that when turned on would require you to explicitly import any file 
>>> that contained a dependency you needed (like in objective-c).
>>> 
>>> I'm going to spend more time looking over the docs and the output logs to 
>>> see if this would be a feasible. If anyone has opinions or insights into 
>>> this I would love to hear from you. 
>>> 
>>> Sam
>>> 
>>> On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_r...@apple.com 
>>> <mailto:jordan_r...@apple.com>> wrote:
>>> Hi, Sam. I don't think we currently have a good answer for this built into 
>>> xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds would 
>>> be fast enough that it wouldn't matter! That's obviously not where we are.)
>>> 
>>> Since '-debug-time-function-bodies' is now public knowledge, I'll share 
>>> another one of our debugging flags, '-driver-show-incremental'. You can add 
>>> this to your "Other Swift Flags". The output isn't very detailed, though:
>>> 
>>> Queuing Tree.swift (initial)
>>> Queuing AdventureScene.swift (initial)
>>> Queuing AdventureScene.swift because of dependencies discovered later
>>> Queuing AppDelegate.swift because of dependencies discovered later
>>> Queuing ChaseArtificialIntelligence.swift because of dependencies 
>>> discovered later
>>> Queuing Character.swift because of dependencies discovered later
>>> Queuing SpawnArtificialIntelligence.swift because of dependencies 
>>> discovered later
>>> Queuing Goblin.swift because of dependencies discovered later
>>> Queuing Cave.swift because of dependencies discovered later
>>> Queuing AdventureSceneOSXEvents.swift because of dependencies discovered 
>>> later
>>> Queuing HeroCharacter.swift because of dependencies discovered later
>>> Queuing EnemyCharacter.swift because of dependencies discovered later
>>> Queuing Boss.swift because of dependencies discovered later
>>> Queuing SharedAssetManagement.swift because of dependencies discovered later
>>> Queuing Warrior.swift because of dependencies discovered later
>>> Queuing Archer.swift because of dependencies discovered later
>>> Queuing Player.swift because of dependencies discovered later
>>> Queuing ArtificialIntelligence.swift because of dependencies discovered 
>>> later
>>> 
>>> In this case, I took a version of the Adventure sample project and modified 
>>> "Tree.swift"; that triggered recompilation of several other files. 
>>> Unfortunately this view doesn't tell you how they're related, only which 
>>> ones are actually getting rebuilt.
>>> 
>>> The next step (and moving into the territory of "working on Swift" rather 
>>> than just "trying to figure out why it's repeating work") would be to look 
>>> at the "swiftdeps" files stored in your DerivedData folder. These are 
>>> currently just YAML files describing what Swift thinks the file depends on, 
>>> as well as what will trigger rebuilding of other files. This is intended to 
>>> be a conservative estimate, since not recompiling something would result in 
>>> an invalid binary. (Unfortunately I say "intended" because there are known 
>>> bugs; fortunately, archive builds are always clean builds anyway.)
>>> 
>>> There's a document in the Swift repo describing the logic behind Swift's 
>>> dependency analysis: 
>>> https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst 
>>> <https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst>. 
>>> The one thing that's not in there is the notion of changes that don't 
>>> affect other files at all. This is accomplished by computing a hash of all 
>>> the tokens that could affect other files, and seeing if that hash has 
>>> changed.
>>> 
>>> We definitely have room for improvement here.
>>> 
>>> Jordan
>>> 
>>> 
>>>> On Mar 31, 2016, at 11:24 , Samantha John via swift-dev 
>>>> <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
>>>> 
>>>> I have a large project (308 swift files, 441 objective c, 66k lines of 
>>>> code) where incremental builds can be extremely slow. I'm trying to do 
>>>> some profiling to figure out what type of things cause large scale 
>>>> recompiles. The problem is that I can't find a good way of telling which 
>>>> files get recompiled on an incremental build and which do not. It seems 
>>>> like files that are not recompiled still get listed in xcode, but the 
>>>> compiler just passes over them really fast. 
>>>> 
>>>> Does anyone know if xctool or xcodebuild has this type of functionality? 
>>>> Or is there some other way to get this info?
>>>> 
>>>> Thank you,
>>>> Sam
>>>> _______________________________________________
>>>> swift-dev mailing list
>>>> swift-dev@swift.org <mailto:swift-dev@swift.org>
>>>> https://lists.swift.org/mailman/listinfo/swift-dev 
>>>> <https://lists.swift.org/mailman/listinfo/swift-dev>
>>> 
>>> 
>>> _______________________________________________
>>> swift-dev mailing list
>>> swift-dev@swift.org <mailto:swift-dev@swift.org>
>>> https://lists.swift.org/mailman/listinfo/swift-dev 
>>> <https://lists.swift.org/mailman/listinfo/swift-dev>
>> 
>> 
>> 
>> _______________________________________________
>> swift-dev mailing list
>> swift-dev@swift.org <mailto:swift-dev@swift.org>
>> https://lists.swift.org/mailman/listinfo/swift-dev 
>> <https://lists.swift.org/mailman/listinfo/swift-dev>
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Reply via email to