Yeah I figured this would look ugly like that. I'll see what we can do.
On Wed, Jan 8, 2014 at 11:27 PM, Jake Wharton <jakewhar...@gmail.com> wrote:
> I think we almost never do clean builds locally. We really noticed the
> regression on the CI as they more than doubled in time.
>
> I added the following for now:
>
> dexOptions {
> preDexLibraries = Boolean.valueOf(System.getProperty("pre-dex", "true"))
> }
>
> I hesitate to agree on adding more configuration options but for something
> like this I certainly wouldn’t complain.
>
>
> ---
> Jake Wharton
> http://about.me/jakewharton
>
>
> On Wed, Jan 8, 2014 at 8:38 PM, Xavier Ducrohet <x...@android.com> wrote:
>
>> I implemented and checked in parallelization of pre-dexing but even with
>> plenty of cores I don't think you're going to beat a build with predexing
>> off when running a clean build.
>>
>> I think it's still a good idea to disable it on CI. I'd like to see if I
>> can add task specific command line attributes (similar to the tasks --all
>> flag for instance), so that maybe you could just do "./gradlew build
>> --no-pre-dexing" instead of hacking the value of
>> android.dexOptions.preDexLibraries through an env var or something.
>>
>>
>> On Wed, Jan 8, 2014 at 1:25 PM, Jake Wharton <jakewhar...@gmail.com>wrote:
>>
>>> General info:
>>>
>>> - All apps compile with Java 7
>>> - All apps minSdkVersion=14, targetSdkVersion=19,
>>> compileSdkVersion=19
>>> - All apps build tools 19.0.0 (will ensure 19.0.1 doesn’t change
>>> behavior)
>>>
>>> Project structure:
>>>
>>> $ tree -d -L 2
>>> .
>>> ├── one
>>> │ ├── one-app
>>> │ └── one-protos
>>> ├── two
>>> │ ├── two-protos
>>> │ ├── two-v0
>>> │ └── two-v1
>>> ├── common
>>> │ ├── truststore
>>> │ ├── typeface-square-market
>>> │ └── zxing-qrcode
>>> └── three
>>> ├── three-app
>>> └── three-protos
>>>
>>> App “One”:
>>>
>>> apk
>>> +--- com.jakewharton.hugo:hugo-annotations:1.0.0
>>> +--- com.android.support:support-v4:19.0.+ -> 19.0.0
>>> +--- com.google.android.gms:play-services:4.0.+ -> 4.0.30
>>> | \--- com.android.support:support-v4:13.0.0 -> 19.0.0
>>> +--- com.squareup.dagger:dagger:1.2.0
>>> | \--- javax.inject:javax.inject:1
>>> +--- com.squareup.okhttp:okhttp:1.2.1
>>> | \--- com.squareup.okhttp:okhttp-protocols:1.2.1
>>> +--- com.squareup.picasso:picasso:2.1.1
>>> +--- com.squareup:pollexor:1.2.0
>>> +--- com.squareup:otto:1.3.4
>>> +--- com.squareup.retrofit:retrofit:1.3.0
>>> | \--- com.google.code.gson:gson:2.2.4
>>> +--- com.squareup.retrofit:converter-wire:1.3.0
>>> | +--- com.squareup.retrofit:retrofit:1.3.0 (*)
>>> | \--- com.squareup.wire:wire-runtime:1.2.0 -> 1.3.0-SNAPSHOT
>>> +--- com.jakewharton:butterknife:4.0.1
>>> +--- com.jakewharton.timber:timber:2.1.0
>>> +--- com.netflix.rxjava:rxjava-core:0.16.0
>>> +--- com.netflix.rxjava:rxjava-android:0.16.0
>>> | \--- com.netflix.rxjava:rxjava-core:0.16.0
>>> +--- com.crashlytics.android:crashlytics:1.+ -> 1.1.5
>>> +--- sterling.common:truststore:unspecified
>>> +--- sterling.common:typeface-square-market:unspecified
>>> \--- sterling.one:one-protos:unspecified
>>> \--- com.squareup.wire:wire-runtime:1.3.0-SNAPSHOT
>>>
>>> debugApk
>>> +--- com.jakewharton.hugo:hugo-runtime:1.0.0
>>> | +--- com.jakewharton.hugo:hugo-annotations:1.0.0
>>> | \--- org.aspectj:aspectjrt:1.7.4
>>> +--- com.squareup.retrofit:retrofit-mock:1.3.0
>>> | \--- com.squareup.retrofit:retrofit:1.3.0
>>> | \--- com.google.code.gson:gson:2.2.4
>>> +--- com.github.kevinsawicki:http-request:5.5
>>> +--- com.jakewharton.madge:madge:1.1.1
>>> \--- com.jakewharton.scalpel:scalpel:1.0.0
>>>
>>> apt
>>> \--- com.squareup.dagger:dagger-compiler:1.2.0
>>> +--- com.squareup.dagger:dagger:1.2.0
>>> | \--- javax.inject:javax.inject:1
>>> +--- com.squareup:javawriter:2.3.0
>>> \--- com.google.guava:guava:15.0
>>>
>>> App “Two”:
>>>
>>> apk
>>> +--- com.android.support:support-v4:19.0.+ -> 19.0.0
>>> +--- com.google.android.gms:play-services:4.0.+ -> 4.0.30
>>> | \--- com.android.support:support-v4:13.0.0 -> 19.0.0
>>> +--- com.squareup.retrofit:retrofit:1.2.1
>>> | \--- com.google.code.gson:gson:2.2.4
>>> +--- com.squareup.retrofit:converter-wire:1.2.1
>>> | +--- com.squareup.retrofit:retrofit:1.2.1 (*)
>>> | \--- com.squareup.wire:wire-runtime:1.0.1 -> 1.0.2-SNAPSHOT
>>> +--- com.squareup.dagger:dagger:1.2.+ -> 1.2.0
>>> | \--- javax.inject:javax.inject:1
>>> +--- com.squareup.okhttp:okhttp:1.2.1
>>> | \--- com.squareup.okhttp:okhttp-protocols:1.2.1
>>> +--- com.squareup.picasso:picasso:2.0.2
>>> +--- com.squareup.wire:wire-runtime:1.0.1 -> 1.0.2-SNAPSHOT
>>> +--- com.crashlytics.android:crashlytics:1.0.2
>>> +--- com.flurry:agent:2.2.5
>>> +--- com.jakewharton:butterknife:4.0.+ -> 4.0.1
>>> +--- com.jakewharton.timber:timber:2.0.+ -> 2.0.0
>>> +--- sterling.common:typeface-square-market:unspecified
>>> \--- sterling.two:two-protos:unspecified
>>> \--- com.squareup.wire:wire-runtime:1.0.2-SNAPSHOT
>>>
>>> apt
>>> \--- com.squareup.dagger:dagger-compiler:1.2.+ -> 1.2.0
>>> +--- com.squareup.dagger:dagger:1.2.0
>>> | \--- javax.inject:javax.inject:1
>>> +--- com.squareup:javawriter:2.3.0
>>> \--- com.google.guava:guava:15.0
>>>
>>> App “Three”:
>>>
>>> apk
>>> +--- com.android.support:support-v4:19.0.+ -> 19.0.0
>>> +--- com.google.android.gms:play-services:3.2.+ -> 3.2.65
>>> | \--- com.android.support:support-v4:13.0.0 -> 19.0.0
>>> +--- com.squareup.dagger:dagger:1.0.1
>>> | \--- javax.inject:javax.inject:1
>>> +--- com.squareup.okhttp:okhttp:1.2.1
>>> | \--- com.squareup.okhttp:okhttp-protocols:1.2.1
>>> +--- com.squareup.picasso:picasso:2.1.1
>>> +--- com.squareup.phrase:phrase:1.0-SNAPSHOT
>>> +--- com.squareup.retrofit:retrofit:1.2.3-SNAPSHOT
>>> | \--- com.google.code.gson:gson:2.2.4
>>> +--- com.squareup.retrofit:converter-wire:1.2.3-SNAPSHOT
>>> | +--- com.squareup.retrofit:retrofit:1.2.3-SNAPSHOT (*)
>>> | \--- com.squareup.wire:wire-runtime:1.2.0 -> 1.3.0-SNAPSHOT
>>> +--- com.jakewharton:butterknife:4.0.+ -> 4.0.1
>>> +--- com.jakewharton.timber:timber:2.0.0
>>> +--- com.netflix.rxjava:rxjava-core:0.15.+ -> 0.15.1
>>> +--- com.netflix.rxjava:rxjava-android:0.15.+ -> 0.15.1
>>> | \--- com.netflix.rxjava:rxjava-core:0.15.1
>>> +--- com.crashlytics.android:crashlytics:1.0.+ -> 1.0.4
>>> +--- sterling.common:truststore:unspecified
>>> +--- sterling.common:typeface-square-market:unspecified
>>> +--- sterling.common:zxing-qrcode:unspecified
>>> \--- sterling.three:three-protos:unspecified
>>> \--- com.squareup.wire:wire-runtime:1.3.0-SNAPSHOT
>>>
>>> debugApk
>>> \--- com.squareup.retrofit:retrofit-mock:1.2.3-SNAPSHOT
>>> \--- com.squareup.retrofit:retrofit:1.2.3-SNAPSHOT
>>> \--- com.google.code.gson:gson:2.2.4
>>>
>>>
>>>
>>> ---
>>> Jake Wharton
>>> http://about.me/jakewharton
>>>
>>>
>>> On Wed, Jan 8, 2014 at 8:39 AM, Xavier Ducrohet <x...@android.com> wrote:
>>>
>>>> wow, the difference is crazy. If you can provide the dependency tree
>>>> that would be nice.
>>>>
>>>> I was looking at the code for the PreDex task and realize we don't
>>>> parallelize the pre-dexing when there are more than one library which I
>>>> thought we did. I think dx is making use of all the cores but I'm not sure
>>>> how much it would help for this case.
>>>>
>>>>
>>>> On Wed, Jan 8, 2014 at 1:27 AM, Jake Wharton <jakewhar...@gmail.com>wrote:
>>>>
>>>>> All of these benchmarks are done with nothing in gradle.properties.
>>>>>
>>>>> With pre-dexing off, ./gradlew clean assemble --profile --offline:
>>>>>
>>>>> :one:one-app 1m13.07s
>>>>> :one:one-app:dexDogfood 17.612s
>>>>> :one:one-app:dexRelease 17.261s
>>>>> :one:one-app:dexDebug 15.849s
>>>>>
>>>>> :two:two-v1 1m1.84s
>>>>> :two:two-v1:dexDogfood 15.553s
>>>>> :two:two-v1:dexDebug 15.066s
>>>>> :two:two-v1:dexRelease 14.183s
>>>>>
>>>>> :three:three-app 36.907s
>>>>> :three:three-app:dexDebug 9.278s
>>>>> :three:three-app:dexDogfood 8.848s
>>>>> :three:three-app:dexRelease 8.654s
>>>>>
>>>>> With pre-dexing on, ./gradlew clean assemble --profile --offline:
>>>>>
>>>>> :one:one-app 3m14.97s
>>>>> :one:one-app:preDexDebug 58.122s
>>>>> :one:one-app:preDexRelease 51.320s
>>>>> :one:one-app:preDexDogfood 50.261s
>>>>> :one:one-app:dexDebug 5.591s
>>>>> :one:one-app:dexDogfood 4.257s
>>>>> :one:one-app:dexRelease 4.037s
>>>>>
>>>>> :two:two-v1 2m29.22s
>>>>> :two:two-v1:preDexDogfood 36.434s
>>>>> :two:two-v1:preDexDebug 36.408s
>>>>> :two:two-v1:preDexRelease 35.988s
>>>>> :two:two-v1:dexDogfood 14.891s
>>>>> :two:two-v1:dexDebug 4.893s
>>>>> :two:two-v1:dexRelease 4.003s
>>>>>
>>>>> :three:three-app 2m14.54s
>>>>> :three:three-app:preDexRelease 44.191s
>>>>> :three:three-app:preDexDebug 35.970s
>>>>> :three:three-app:preDexDogfood 33.390s
>>>>> :three:three-app:dexDebug 3.956s
>>>>> :three:three-app:dexRelease 3.575s
>>>>> :three:three-app:dexDogfood 3.451s
>>>>>
>>>>> $ java -version
>>>>> java version "1.8.0-ea"
>>>>> Java(TM) SE Runtime Environment (build 1.8.0-ea-b116)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.0-b58, mixed mode)
>>>>>
>>>>> $ ./gradlew --version
>>>>>
>>>>> ------------------------------------------------------------
>>>>> Gradle 1.9
>>>>> ------------------------------------------------------------
>>>>>
>>>>> Build time: 2013-11-19 08:20:02 UTC
>>>>> Build number: none
>>>>> Revision: 7970ec3503b4f5767ee1c1c69f8b4186c4763e3d
>>>>>
>>>>> Groovy: 1.8.6
>>>>> Ant: Apache Ant(TM) version 1.9.2 compiled on July 8 2013
>>>>> Ivy: 2.2.0
>>>>> JVM: 1.8.0-ea (Oracle Corporation 25.0-b58)
>>>>> OS: Mac OS X 10.8.5 x86_64
>>>>>
>>>>> Let me know if there's anything else I can provide (e.g., dependency
>>>>> tree?)
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> Jake Wharton
>>>>> http://about.me/jakewharton
>>>>>
>>>>>
>>>>> On Tue, Jan 7, 2014 at 9:39 PM, Xavier Ducrohet <x...@android.com>wrote:
>>>>>
>>>>>> BTW I'm interested in the time it takes to do predex + dex vs dex
>>>>>> only (no pre-dexing) for clean builds
>>>>>>
>>>>>> It's worth it in incremental builds, but I'm suprised clean builds
>>>>>> are so much longer. I'd like to send some stats to the VM team (who owns
>>>>>> dx)
>>>>>>
>>>>>> thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jan 7, 2014 at 11:36 PM, Xavier Ducrohet <x...@android.com>wrote:
>>>>>>
>>>>>>> android {
>>>>>>> dexOptions {
>>>>>>> preDexLibraries = false
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jan 7, 2014 at 3:13 PM, Jake Wharton
>>>>>>> <jakewhar...@gmail.com>wrote:
>>>>>>>
>>>>>>>> Well it's definitely pre-dexing. Can this be easily disabled for CI
>>>>>>>> builds?
>>>>>>>>
>>>>>>>> :app1:app1-app 3m14.13s
>>>>>>>> :app1:app1-app:preDexDebug 1m5.23s
>>>>>>>> :app1:app1-app:preDexDogfood 51.836s
>>>>>>>> :app1:app1-app:preDexRelease 41.499s
>>>>>>>>
>>>>>>>> 81% of time spent pre-dexing
>>>>>>>>
>>>>>>>>
>>>>>>>> :app2:app2-v1 2m25.18s
>>>>>>>> :app2:app2-v1:preDexRelease 39.274s
>>>>>>>> :app2:app2-v1:preDexDogfood 38.637s
>>>>>>>> :app2:app2-v1:preDexDebug 36.630s
>>>>>>>>
>>>>>>>> 78% of time spent pre-dexing
>>>>>>>>
>>>>>>>>
>>>>>>>> :app3:app3-app 2m18.30s
>>>>>>>> :app3:app3-app:preDexRelease 45.504s
>>>>>>>> :app3:app3-app:preDexDebug 36.681s
>>>>>>>> :app3:app3-app:preDexDogfood 34.769s
>>>>>>>>
>>>>>>>> 84% of time spent pre-dexing
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Jake Wharton
>>>>>>>> http://about.me/jakewharton
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 6, 2014 at 10:37 PM, Jake Wharton <
>>>>>>>> jakewhar...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> They were clean builds, yes. I'll dig deeper tomorrow. It was the
>>>>>>>>> end of the work day for me when we started the comparisons.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Jake Wharton
>>>>>>>>> http://about.me/jakewharton
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jan 6, 2014 at 6:07 PM, Xavier Ducrohet
>>>>>>>>> <x...@android.com>wrote:
>>>>>>>>>
>>>>>>>>>> incremental dexing has been disabled. There are some issues with
>>>>>>>>>> it. We have implemented pre-dexing though, but I'm not sure how the
>>>>>>>>>> performance is impacted.
>>>>>>>>>>
>>>>>>>>>> Are those time for clean builds? (which should not be impacted by
>>>>>>>>>> the removal of the incremental dex).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 6, 2014 at 4:50 PM, Jake Wharton <
>>>>>>>>>> jakewhar...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I forgot to ask a question... Is anyone else seeing anything
>>>>>>>>>>> like this?
>>>>>>>>>>>
>>>>>>>>>>> I see this on small pet projects too. I can perceive tasks
>>>>>>>>>>> taking 10x longer than when on 0.6.x. Usually those towards the
>>>>>>>>>>> latter end
>>>>>>>>>>> of the processing (preDex, dex, assemble).
>>>>>>>>>>>
>>>>>>>>>>> Still digging for more info to pinpoint what is happening.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>> Jake Wharton
>>>>>>>>>>> http://about.me/jakewharton
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 6, 2014 at 4:44 PM, Jake Wharton <
>>>>>>>>>>> jakewhar...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Our project builds have experienced a serious regression in
>>>>>>>>>>>> terms of performance upon upgrading to 0.7.x.
>>>>>>>>>>>>
>>>>>>>>>>>> Without --parallel:
>>>>>>>>>>>>
>>>>>>>>>>>> - 0.6.x: 4m 21s
>>>>>>>>>>>> - 0.7.x: 9m 04s
>>>>>>>>>>>>
>>>>>>>>>>>> With --parallel:
>>>>>>>>>>>>
>>>>>>>>>>>> - 0.6.x: 3m 36s
>>>>>>>>>>>> - 0.7.x: 6m 58s
>>>>>>>>>>>>
>>>>>>>>>>>> This project has 3 apps (each with three build types, no
>>>>>>>>>>>> flavors), 4 library modules, and 3 java modules.
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> Jake Wharton
>>>>>>>>>>>> http://about.me/jakewharton
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "adt-dev" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>> it, send an email to adt-dev+unsubscr...@googlegroups.com.
>>>>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Xavier Ducrohet
>>>>>>>>>> Android SDK Tech Lead
>>>>>>>>>> Google Inc.
>>>>>>>>>> http://developer.android.com | http://tools.android.com
>>>>>>>>>>
>>>>>>>>>> Please do not send me questions directly. Thanks!
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "adt-dev" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to adt-dev+unsubscr...@googlegroups.com.
>>>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "adt-dev" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to adt-dev+unsubscr...@googlegroups.com.
>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Xavier Ducrohet
>>>>>>> Android SDK Tech Lead
>>>>>>> Google Inc.
>>>>>>> http://developer.android.com | http://tools.android.com
>>>>>>>
>>>>>>> Please do not send me questions directly. Thanks!
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Xavier Ducrohet
>>>>>> Android SDK Tech Lead
>>>>>> Google Inc.
>>>>>> http://developer.android.com | http://tools.android.com
>>>>>>
>>>>>> Please do not send me questions directly. Thanks!
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "adt-dev" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to adt-dev+unsubscr...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "adt-dev" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to adt-dev+unsubscr...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Xavier Ducrohet
>>>> Android SDK Tech Lead
>>>> Google Inc.
>>>> http://developer.android.com | http://tools.android.com
>>>>
>>>> Please do not send me questions directly. Thanks!
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "adt-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to adt-dev+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "adt-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to adt-dev+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>>
>> --
>> Xavier Ducrohet
>> Android SDK Tech Lead
>> Google Inc.
>> http://developer.android.com | http://tools.android.com
>>
>> Please do not send me questions directly. Thanks!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "adt-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to adt-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "adt-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to adt-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
Xavier Ducrohet
Android SDK Tech Lead
Google Inc.
http://developer.android.com | http://tools.android.com
Please do not send me questions directly. Thanks!
--
You received this message because you are subscribed to the Google Groups
"adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to adt-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
