I think we almost never do clean builds locally. We really noticed the
regression on the CI as they more than doubled in time.
I added the following for now:
dexOptions {
preDexLibraries = Boolean.valueOf(System.getProperty("pre-dex", "true"))
}
I hesitate to agree on adding more configuration options but for something
like this I certainly wouldn’t complain.
---
Jake Wharton
http://about.me/jakewharton
On Wed, Jan 8, 2014 at 8:38 PM, Xavier Ducrohet <x...@android.com> wrote:
> I implemented and checked in parallelization of pre-dexing but even with
> plenty of cores I don't think you're going to beat a build with predexing
> off when running a clean build.
>
> I think it's still a good idea to disable it on CI. I'd like to see if I
> can add task specific command line attributes (similar to the tasks --all
> flag for instance), so that maybe you could just do "./gradlew build
> --no-pre-dexing" instead of hacking the value of
> android.dexOptions.preDexLibraries through an env var or something.
>
>
> On Wed, Jan 8, 2014 at 1:25 PM, Jake Wharton <jakewhar...@gmail.com>wrote:
>
>> General info:
>>
>> - All apps compile with Java 7
>> - All apps minSdkVersion=14, targetSdkVersion=19, compileSdkVersion=19
>> - All apps build tools 19.0.0 (will ensure 19.0.1 doesn’t change
>> behavior)
>>
>> Project structure:
>>
>> $ tree -d -L 2
>> .
>> ├── one
>> │ ├── one-app
>> │ └── one-protos
>> ├── two
>> │ ├── two-protos
>> │ ├── two-v0
>> │ └── two-v1
>> ├── common
>> │ ├── truststore
>> │ ├── typeface-square-market
>> │ └── zxing-qrcode
>> └── three
>> ├── three-app
>> └── three-protos
>>
>> App “One”:
>>
>> apk
>> +--- com.jakewharton.hugo:hugo-annotations:1.0.0
>> +--- com.android.support:support-v4:19.0.+ -> 19.0.0
>> +--- com.google.android.gms:play-services:4.0.+ -> 4.0.30
>> | \--- com.android.support:support-v4:13.0.0 -> 19.0.0
>> +--- com.squareup.dagger:dagger:1.2.0
>> | \--- javax.inject:javax.inject:1
>> +--- com.squareup.okhttp:okhttp:1.2.1
>> | \--- com.squareup.okhttp:okhttp-protocols:1.2.1
>> +--- com.squareup.picasso:picasso:2.1.1
>> +--- com.squareup:pollexor:1.2.0
>> +--- com.squareup:otto:1.3.4
>> +--- com.squareup.retrofit:retrofit:1.3.0
>> | \--- com.google.code.gson:gson:2.2.4
>> +--- com.squareup.retrofit:converter-wire:1.3.0
>> | +--- com.squareup.retrofit:retrofit:1.3.0 (*)
>> | \--- com.squareup.wire:wire-runtime:1.2.0 -> 1.3.0-SNAPSHOT
>> +--- com.jakewharton:butterknife:4.0.1
>> +--- com.jakewharton.timber:timber:2.1.0
>> +--- com.netflix.rxjava:rxjava-core:0.16.0
>> +--- com.netflix.rxjava:rxjava-android:0.16.0
>> | \--- com.netflix.rxjava:rxjava-core:0.16.0
>> +--- com.crashlytics.android:crashlytics:1.+ -> 1.1.5
>> +--- sterling.common:truststore:unspecified
>> +--- sterling.common:typeface-square-market:unspecified
>> \--- sterling.one:one-protos:unspecified
>> \--- com.squareup.wire:wire-runtime:1.3.0-SNAPSHOT
>>
>> debugApk
>> +--- com.jakewharton.hugo:hugo-runtime:1.0.0
>> | +--- com.jakewharton.hugo:hugo-annotations:1.0.0
>> | \--- org.aspectj:aspectjrt:1.7.4
>> +--- com.squareup.retrofit:retrofit-mock:1.3.0
>> | \--- com.squareup.retrofit:retrofit:1.3.0
>> | \--- com.google.code.gson:gson:2.2.4
>> +--- com.github.kevinsawicki:http-request:5.5
>> +--- com.jakewharton.madge:madge:1.1.1
>> \--- com.jakewharton.scalpel:scalpel:1.0.0
>>
>> apt
>> \--- com.squareup.dagger:dagger-compiler:1.2.0
>> +--- com.squareup.dagger:dagger:1.2.0
>> | \--- javax.inject:javax.inject:1
>> +--- com.squareup:javawriter:2.3.0
>> \--- com.google.guava:guava:15.0
>>
>> App “Two”:
>>
>> apk
>> +--- com.android.support:support-v4:19.0.+ -> 19.0.0
>> +--- com.google.android.gms:play-services:4.0.+ -> 4.0.30
>> | \--- com.android.support:support-v4:13.0.0 -> 19.0.0
>> +--- com.squareup.retrofit:retrofit:1.2.1
>> | \--- com.google.code.gson:gson:2.2.4
>> +--- com.squareup.retrofit:converter-wire:1.2.1
>> | +--- com.squareup.retrofit:retrofit:1.2.1 (*)
>> | \--- com.squareup.wire:wire-runtime:1.0.1 -> 1.0.2-SNAPSHOT
>> +--- com.squareup.dagger:dagger:1.2.+ -> 1.2.0
>> | \--- javax.inject:javax.inject:1
>> +--- com.squareup.okhttp:okhttp:1.2.1
>> | \--- com.squareup.okhttp:okhttp-protocols:1.2.1
>> +--- com.squareup.picasso:picasso:2.0.2
>> +--- com.squareup.wire:wire-runtime:1.0.1 -> 1.0.2-SNAPSHOT
>> +--- com.crashlytics.android:crashlytics:1.0.2
>> +--- com.flurry:agent:2.2.5
>> +--- com.jakewharton:butterknife:4.0.+ -> 4.0.1
>> +--- com.jakewharton.timber:timber:2.0.+ -> 2.0.0
>> +--- sterling.common:typeface-square-market:unspecified
>> \--- sterling.two:two-protos:unspecified
>> \--- com.squareup.wire:wire-runtime:1.0.2-SNAPSHOT
>>
>> apt
>> \--- com.squareup.dagger:dagger-compiler:1.2.+ -> 1.2.0
>> +--- com.squareup.dagger:dagger:1.2.0
>> | \--- javax.inject:javax.inject:1
>> +--- com.squareup:javawriter:2.3.0
>> \--- com.google.guava:guava:15.0
>>
>> App “Three”:
>>
>> apk
>> +--- com.android.support:support-v4:19.0.+ -> 19.0.0
>> +--- com.google.android.gms:play-services:3.2.+ -> 3.2.65
>> | \--- com.android.support:support-v4:13.0.0 -> 19.0.0
>> +--- com.squareup.dagger:dagger:1.0.1
>> | \--- javax.inject:javax.inject:1
>> +--- com.squareup.okhttp:okhttp:1.2.1
>> | \--- com.squareup.okhttp:okhttp-protocols:1.2.1
>> +--- com.squareup.picasso:picasso:2.1.1
>> +--- com.squareup.phrase:phrase:1.0-SNAPSHOT
>> +--- com.squareup.retrofit:retrofit:1.2.3-SNAPSHOT
>> | \--- com.google.code.gson:gson:2.2.4
>> +--- com.squareup.retrofit:converter-wire:1.2.3-SNAPSHOT
>> | +--- com.squareup.retrofit:retrofit:1.2.3-SNAPSHOT (*)
>> | \--- com.squareup.wire:wire-runtime:1.2.0 -> 1.3.0-SNAPSHOT
>> +--- com.jakewharton:butterknife:4.0.+ -> 4.0.1
>> +--- com.jakewharton.timber:timber:2.0.0
>> +--- com.netflix.rxjava:rxjava-core:0.15.+ -> 0.15.1
>> +--- com.netflix.rxjava:rxjava-android:0.15.+ -> 0.15.1
>> | \--- com.netflix.rxjava:rxjava-core:0.15.1
>> +--- com.crashlytics.android:crashlytics:1.0.+ -> 1.0.4
>> +--- sterling.common:truststore:unspecified
>> +--- sterling.common:typeface-square-market:unspecified
>> +--- sterling.common:zxing-qrcode:unspecified
>> \--- sterling.three:three-protos:unspecified
>> \--- com.squareup.wire:wire-runtime:1.3.0-SNAPSHOT
>>
>> debugApk
>> \--- com.squareup.retrofit:retrofit-mock:1.2.3-SNAPSHOT
>> \--- com.squareup.retrofit:retrofit:1.2.3-SNAPSHOT
>> \--- com.google.code.gson:gson:2.2.4
>>
>>
>>
>> ---
>> Jake Wharton
>> http://about.me/jakewharton
>>
>>
>> On Wed, Jan 8, 2014 at 8:39 AM, Xavier Ducrohet <x...@android.com> wrote:
>>
>>> wow, the difference is crazy. If you can provide the dependency tree
>>> that would be nice.
>>>
>>> I was looking at the code for the PreDex task and realize we don't
>>> parallelize the pre-dexing when there are more than one library which I
>>> thought we did. I think dx is making use of all the cores but I'm not sure
>>> how much it would help for this case.
>>>
>>>
>>> On Wed, Jan 8, 2014 at 1:27 AM, Jake Wharton <jakewhar...@gmail.com>wrote:
>>>
>>>> All of these benchmarks are done with nothing in gradle.properties.
>>>>
>>>> With pre-dexing off, ./gradlew clean assemble --profile --offline:
>>>>
>>>> :one:one-app 1m13.07s
>>>> :one:one-app:dexDogfood 17.612s
>>>> :one:one-app:dexRelease 17.261s
>>>> :one:one-app:dexDebug 15.849s
>>>>
>>>> :two:two-v1 1m1.84s
>>>> :two:two-v1:dexDogfood 15.553s
>>>> :two:two-v1:dexDebug 15.066s
>>>> :two:two-v1:dexRelease 14.183s
>>>>
>>>> :three:three-app 36.907s
>>>> :three:three-app:dexDebug 9.278s
>>>> :three:three-app:dexDogfood 8.848s
>>>> :three:three-app:dexRelease 8.654s
>>>>
>>>> With pre-dexing on, ./gradlew clean assemble --profile --offline:
>>>>
>>>> :one:one-app 3m14.97s
>>>> :one:one-app:preDexDebug 58.122s
>>>> :one:one-app:preDexRelease 51.320s
>>>> :one:one-app:preDexDogfood 50.261s
>>>> :one:one-app:dexDebug 5.591s
>>>> :one:one-app:dexDogfood 4.257s
>>>> :one:one-app:dexRelease 4.037s
>>>>
>>>> :two:two-v1 2m29.22s
>>>> :two:two-v1:preDexDogfood 36.434s
>>>> :two:two-v1:preDexDebug 36.408s
>>>> :two:two-v1:preDexRelease 35.988s
>>>> :two:two-v1:dexDogfood 14.891s
>>>> :two:two-v1:dexDebug 4.893s
>>>> :two:two-v1:dexRelease 4.003s
>>>>
>>>> :three:three-app 2m14.54s
>>>> :three:three-app:preDexRelease 44.191s
>>>> :three:three-app:preDexDebug 35.970s
>>>> :three:three-app:preDexDogfood 33.390s
>>>> :three:three-app:dexDebug 3.956s
>>>> :three:three-app:dexRelease 3.575s
>>>> :three:three-app:dexDogfood 3.451s
>>>>
>>>> $ java -version
>>>> java version "1.8.0-ea"
>>>> Java(TM) SE Runtime Environment (build 1.8.0-ea-b116)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.0-b58, mixed mode)
>>>>
>>>> $ ./gradlew --version
>>>>
>>>> ------------------------------------------------------------
>>>> Gradle 1.9
>>>> ------------------------------------------------------------
>>>>
>>>> Build time: 2013-11-19 08:20:02 UTC
>>>> Build number: none
>>>> Revision: 7970ec3503b4f5767ee1c1c69f8b4186c4763e3d
>>>>
>>>> Groovy: 1.8.6
>>>> Ant: Apache Ant(TM) version 1.9.2 compiled on July 8 2013
>>>> Ivy: 2.2.0
>>>> JVM: 1.8.0-ea (Oracle Corporation 25.0-b58)
>>>> OS: Mac OS X 10.8.5 x86_64
>>>>
>>>> Let me know if there's anything else I can provide (e.g., dependency
>>>> tree?)
>>>>
>>>>
>>>>
>>>> ---
>>>> Jake Wharton
>>>> http://about.me/jakewharton
>>>>
>>>>
>>>> On Tue, Jan 7, 2014 at 9:39 PM, Xavier Ducrohet <x...@android.com>wrote:
>>>>
>>>>> BTW I'm interested in the time it takes to do predex + dex vs dex only
>>>>> (no pre-dexing) for clean builds
>>>>>
>>>>> It's worth it in incremental builds, but I'm suprised clean builds are
>>>>> so much longer. I'd like to send some stats to the VM team (who owns dx)
>>>>>
>>>>> thanks.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jan 7, 2014 at 11:36 PM, Xavier Ducrohet <x...@android.com>wrote:
>>>>>
>>>>>> android {
>>>>>> dexOptions {
>>>>>> preDexLibraries = false
>>>>>> }
>>>>>> }
>>>>>>
>>>>>>
>>>>>> On Tue, Jan 7, 2014 at 3:13 PM, Jake Wharton
>>>>>> <jakewhar...@gmail.com>wrote:
>>>>>>
>>>>>>> Well it's definitely pre-dexing. Can this be easily disabled for CI
>>>>>>> builds?
>>>>>>>
>>>>>>> :app1:app1-app 3m14.13s
>>>>>>> :app1:app1-app:preDexDebug 1m5.23s
>>>>>>> :app1:app1-app:preDexDogfood 51.836s
>>>>>>> :app1:app1-app:preDexRelease 41.499s
>>>>>>>
>>>>>>> 81% of time spent pre-dexing
>>>>>>>
>>>>>>>
>>>>>>> :app2:app2-v1 2m25.18s
>>>>>>> :app2:app2-v1:preDexRelease 39.274s
>>>>>>> :app2:app2-v1:preDexDogfood 38.637s
>>>>>>> :app2:app2-v1:preDexDebug 36.630s
>>>>>>>
>>>>>>> 78% of time spent pre-dexing
>>>>>>>
>>>>>>>
>>>>>>> :app3:app3-app 2m18.30s
>>>>>>> :app3:app3-app:preDexRelease 45.504s
>>>>>>> :app3:app3-app:preDexDebug 36.681s
>>>>>>> :app3:app3-app:preDexDogfood 34.769s
>>>>>>>
>>>>>>> 84% of time spent pre-dexing
>>>>>>>
>>>>>>>
>>>>>>> ---
>>>>>>> Jake Wharton
>>>>>>> http://about.me/jakewharton
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jan 6, 2014 at 10:37 PM, Jake Wharton <jakewhar...@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> They were clean builds, yes. I'll dig deeper tomorrow. It was the
>>>>>>>> end of the work day for me when we started the comparisons.
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Jake Wharton
>>>>>>>> http://about.me/jakewharton
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 6, 2014 at 6:07 PM, Xavier Ducrohet
>>>>>>>> <x...@android.com>wrote:
>>>>>>>>
>>>>>>>>> incremental dexing has been disabled. There are some issues with
>>>>>>>>> it. We have implemented pre-dexing though, but I'm not sure how the
>>>>>>>>> performance is impacted.
>>>>>>>>>
>>>>>>>>> Are those time for clean builds? (which should not be impacted by
>>>>>>>>> the removal of the incremental dex).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jan 6, 2014 at 4:50 PM, Jake Wharton <
>>>>>>>>> jakewhar...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I forgot to ask a question... Is anyone else seeing anything like
>>>>>>>>>> this?
>>>>>>>>>>
>>>>>>>>>> I see this on small pet projects too. I can perceive tasks taking
>>>>>>>>>> 10x longer than when on 0.6.x. Usually those towards the latter end
>>>>>>>>>> of the
>>>>>>>>>> processing (preDex, dex, assemble).
>>>>>>>>>>
>>>>>>>>>> Still digging for more info to pinpoint what is happening.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> Jake Wharton
>>>>>>>>>> http://about.me/jakewharton
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 6, 2014 at 4:44 PM, Jake Wharton <
>>>>>>>>>> jakewhar...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Our project builds have experienced a serious regression in
>>>>>>>>>>> terms of performance upon upgrading to 0.7.x.
>>>>>>>>>>>
>>>>>>>>>>> Without --parallel:
>>>>>>>>>>>
>>>>>>>>>>> - 0.6.x: 4m 21s
>>>>>>>>>>> - 0.7.x: 9m 04s
>>>>>>>>>>>
>>>>>>>>>>> With --parallel:
>>>>>>>>>>>
>>>>>>>>>>> - 0.6.x: 3m 36s
>>>>>>>>>>> - 0.7.x: 6m 58s
>>>>>>>>>>>
>>>>>>>>>>> This project has 3 apps (each with three build types, no
>>>>>>>>>>> flavors), 4 library modules, and 3 java modules.
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>> Jake Wharton
>>>>>>>>>>> http://about.me/jakewharton
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "adt-dev" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to adt-dev+unsubscr...@googlegroups.com.
>>>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Xavier Ducrohet
>>>>>>>>> Android SDK Tech Lead
>>>>>>>>> Google Inc.
>>>>>>>>> http://developer.android.com | http://tools.android.com
>>>>>>>>>
>>>>>>>>> Please do not send me questions directly. Thanks!
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "adt-dev" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to adt-dev+unsubscr...@googlegroups.com.
>>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "adt-dev" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to adt-dev+unsubscr...@googlegroups.com.
>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Xavier Ducrohet
>>>>>> Android SDK Tech Lead
>>>>>> Google Inc.
>>>>>> http://developer.android.com | http://tools.android.com
>>>>>>
>>>>>> Please do not send me questions directly. Thanks!
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Xavier Ducrohet
>>>>> Android SDK Tech Lead
>>>>> Google Inc.
>>>>> http://developer.android.com | http://tools.android.com
>>>>>
>>>>> Please do not send me questions directly. Thanks!
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "adt-dev" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to adt-dev+unsubscr...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "adt-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to adt-dev+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>
>>>
>>>
>>>
>>> --
>>> Xavier Ducrohet
>>> Android SDK Tech Lead
>>> Google Inc.
>>> http://developer.android.com | http://tools.android.com
>>>
>>> Please do not send me questions directly. Thanks!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "adt-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to adt-dev+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "adt-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to adt-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
>
> --
> Xavier Ducrohet
> Android SDK Tech Lead
> Google Inc.
> http://developer.android.com | http://tools.android.com
>
> Please do not send me questions directly. Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "adt-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to adt-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
You received this message because you are subscribed to the Google Groups
"adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to adt-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
