On Dec 24, 2017 1:15 PM, "Peter Warrington" <[email protected]> wrote:
> Thank you! It cuts about a quater off the time finder normally takes. > > On Friday, December 22, 2017 at 2:10:08 AM UTC, Colin Cross wrote: >> >> The patch stack at https://android-review.goog >> lesource.com/c/platform/build/soong/+/575773 removes >700k lstat calls >> from finder, I'm curious if it makes any difference for you. >> >> On Thu, Dec 21, 2017 at 12:41 PM, Peter Warrington <[email protected]> >> wrote: >> >>> Thank you for all your help and good luck with improving finder! With >>> regards to memory, I have a measly 4GB of ram so that is probally slowing >>> finder down and I do expect it to be slow (I do plan on upgrading in case >>> you are wondering). However, I do not believe what I am experiencing is the >>> same as what Alessandro is experiencing. Again, Thanks for all your help! >>> >>> On Thursday, December 21, 2017 at 8:14:39 PM UTC, Colin Cross wrote: >>>> >>>> Yeah, this is showing very slow lstat calls: >>>> [pid 4007] <... lstat resumed> {st_mode=S_IFDIR|0777, st_size=4096, >>>> ...}) = 0 <0.020417> >>>> vs. mine: >>>> [pid 219359] <... lstat resumed> {st_mode=S_IFREG|0640, st_size=323161, >>>> ...}) = 0 <0.000046> >>>> >>>> I don't know why yours would be so much slower, maybe just round trip >>>> latency to the external drive and not enough memory to keep them all >>>> in the inode cache? >>>> >>>> We can improve finder to drastically reduce the number of lstat calls, >>>> but ninja is going to have to stat most of the files to determine what >>>> needs building so I don't know how much improvement you will see. >>>> >>>> >>>> On Thu, Dec 21, 2017 at 11:56 AM, Peter Warrington <[email protected] >>>> > wrote: >>>> >>>>> Sorry about that. Is this attached strace any better? >>>>> >>>>> On Thursday, December 21, 2017 at 7:36:22 PM UTC, Colin Cross wrote: >>>>>> >>>>>> I don't see the data I'd expect with -T, it should have a time in >>>>>> angle brackets after each finished syscall: >>>>>> [pid 219359] <... lstat resumed> {st_mode=S_IFREG|0640, >>>>>> st_size=323161, ...}) = 0 <0.000046> >>>>>> >>>>>> >>>>>> On Thu, Dec 21, 2017 at 3:05 AM, Peter Warrington < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Attached is a much longer strace with the -T option used. >>>>>>> >>>>>>> On Wednesday, December 20, 2017 at 11:16:53 PM UTC, Colin Cross >>>>>>> wrote: >>>>>>>> >>>>>>>> The results of the finder are cached in out/.module_paths/files.db, >>>>>>>> so once the command has completed successfully once the next runs will >>>>>>>> be >>>>>>>> much faster. You could try wiping that file to reproduce it again, or >>>>>>>> run >>>>>>>> with a different value for OUT_DIR (OUT_DIR=out2 lunch ...). >>>>>>>> >>>>>>>> On Wed, Dec 20, 2017 at 1:58 PM, Peter Warrington < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hello, the command has now started to work like normal after >>>>>>>>> leaving it for a while, so my problem might not be the same as >>>>>>>>> Alessandro's. Unfortunately, I cannot upload a bigger stack trace >>>>>>>>> until >>>>>>>>> tomorrow because of the time in my timezone. Sorry! >>>>>>>>> >>>>>>>>> On Wednesday, December 20, 2017 at 9:48:34 PM UTC, Colin Cross >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Can you attach a longer strace with -T ? >>>>>>>>>> >>>>>>>>>> On Wed, Dec 20, 2017 at 1:28 PM, Peter Warrington < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Thank you! That is very interesting. In case it helps, I >>>>>>>>>>> benchmarked my external HDD and it has an average read rate of 92.0 >>>>>>>>>>> MB/s >>>>>>>>>>> and an average access time of 16.37 msec. However, I didn't have any >>>>>>>>>>> trouble with this previously. >>>>>>>>>>> >>>>>>>>>>> On Wednesday, December 20, 2017 at 9:06:16 PM UTC, Colin Cross >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Nothing looks wrong in that strace, it is making forward >>>>>>>>>>>> progress. It needs to read every directory and stat every file in >>>>>>>>>>>> the >>>>>>>>>>>> tree, is your filesystem very slow for some reason? >>>>>>>>>>>> >>>>>>>>>>>> One difference I see between the old implementation using the >>>>>>>>>>>> find tool and the new implementation is that find seems to be able >>>>>>>>>>>> to >>>>>>>>>>>> avoiding calling stat on every directory entry, because the >>>>>>>>>>>> relevant >>>>>>>>>>>> information (whether the entry is a subdirectory or a file) is >>>>>>>>>>>> contained in >>>>>>>>>>>> the return from the getdents64 syscall. The Go implementation is >>>>>>>>>>>> ignoring >>>>>>>>>>>> that information and has to call lstat to get it. >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Dec 20, 2017 at 11:32 AM, Peter Warrington < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Attached is a partial strace. >>>>>>>>>>>>> >>>>>>>>>>>>> On Wednesday, December 20, 2017 at 7:06:31 PM UTC, Colin Cross >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> From the stack trace after ctrl-c, lunch will be waiting for >>>>>>>>>>>>>> the child soong_ui process to complete. An strace of that >>>>>>>>>>>>>> process would be >>>>>>>>>>>>>> interesting. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Dec 18, 2017 at 4:12 PM, Alessandro Brucato < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is the output of strace attached to the process in >>>>>>>>>>>>>>> which I execute "lunch". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <https://lh3.googleusercontent.com/-Rk-T1kCPUnA/WjhYORJYCKI/AAAAAAAAGSY/LIzqOc58jvkTpj7dowKqjz1OsKoAy0dZQCLcBGAs/s1600/strace.PNG> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> When "lunch" is stuck, the output of strace ends with >>>>>>>>>>>>>>> "wait4(-1,". All the remaining part appears when I abort >>>>>>>>>>>>>>> "lunch" with >>>>>>>>>>>>>>> CTRL+C. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Maybe this issue is due to the fact that I'm using a guest >>>>>>>>>>>>>>> in Virtualbox? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Il giorno lunedì 18 dicembre 2017 23:45:48 UTC+1, Colin >>>>>>>>>>>>>>> Cross ha scritto: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can you capture the full output of all the stack traces? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It's getting stuck while trying to walk all the checked-out >>>>>>>>>>>>>>>> code looking for various build files. If it was completely >>>>>>>>>>>>>>>> deadlocked then >>>>>>>>>>>>>>>> it would have died on its own, so it's probably stuck in a >>>>>>>>>>>>>>>> loop somehow. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Dec 18, 2017 at 8:12 AM, Alessandro Brucato < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello, I'm trying to build the Android source code for my >>>>>>>>>>>>>>>>> Hikey 960 board. >>>>>>>>>>>>>>>>> I'm following this video tutorial https://www.youtube.c >>>>>>>>>>>>>>>>> om/watch?v=RD_NZRKiZBU in parallel to the official >>>>>>>>>>>>>>>>> Android guide https://source.android.com/set >>>>>>>>>>>>>>>>> up/devices#hikey-boards. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Everything went fine until I reached the command "lunch >>>>>>>>>>>>>>>>> hikey960-userdebug". It blocks, without returning any output. >>>>>>>>>>>>>>>>> I left it >>>>>>>>>>>>>>>>> "working" for 1 hour and then I aborted it (even if I know >>>>>>>>>>>>>>>>> that this >>>>>>>>>>>>>>>>> command should take few seconds). If I execute only "lunch" >>>>>>>>>>>>>>>>> it shows the >>>>>>>>>>>>>>>>> list of available build targets, and when I select >>>>>>>>>>>>>>>>> "hikey960-userdebug" it >>>>>>>>>>>>>>>>> blocks. When I abort it with CTRL+C, it says "Still alive, >>>>>>>>>>>>>>>>> killing >>>>>>>>>>>>>>>>> subprocesses" and then it shows a lot of messages referring >>>>>>>>>>>>>>>>> to threads. >>>>>>>>>>>>>>>>> Here is the screenshot https://ibb.co/k32KQR >>>>>>>>>>>>>>>>> I noticed that also the "make" command (inside the AOSP >>>>>>>>>>>>>>>>> directory) has the same behaviour. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This is the list of commands I've used during the process: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> sudo apt-get install openjdk-7-jdk >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> sudo apt-get install git-core gnupg flex bison gperf >>>>>>>>>>>>>>>>> build-essential zip curl zlib1g-dev gcc-multilib g++-multilib >>>>>>>>>>>>>>>>> libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev >>>>>>>>>>>>>>>>> lib32z-dev >>>>>>>>>>>>>>>>> ccache libgl1-mesa-dev libxml2-utils xsltproc unzip >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> mkdir output >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> export OUT_DIR_COMMON_BASE=/home/brucedh/output/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> mkdir ~/bin >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> PATH=~/bin:$PATH >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> curl https://storage.googleapis.com >>>>>>>>>>>>>>>>> /git-repo-downloads/repo > ~/bin/repo >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> chmod a+x ~/bin/repo >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> mkdir dev >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cd dev >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> mkdir hikey960 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cd hikey960 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> git config --global user.name "<REDACTED>" >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> git config --global user.email "<REDACTED>" >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> repo init -u https://android.googlesource.c >>>>>>>>>>>>>>>>> om/platform/manifest -b master >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> repo sync –j4 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> git clone https://android.googlesource.c >>>>>>>>>>>>>>>>> om/kernel/hikey-linaro >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cd hikey-linaro >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> git checkout -b android-hikey-linaro-4.9 >>>>>>>>>>>>>>>>> origin/android-hikey-linaro-4.9 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> make ARCH=arm64 menuconfig >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> make ARCH=arm64 hikey960_defconfig >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> PATH=/home/brucedh/dev/hikey960/prebuilts/gcc/linux-x86/ >>>>>>>>>>>>>>>>> aarch64/aarch64-linux-android-4.9/bin/:$PATH >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-android- -j4 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cp arch/arm64/boot/dts/hisilicon/hi3660-hikey960.dtb >>>>>>>>>>>>>>>>> ../device/linaro/hikey-kernel/hi3660-hikey960.dtb-4.9 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cp arch/arm64/boot/Image.gz ../device/linaro/hikey-kernel/ >>>>>>>>>>>>>>>>> Image.gz-hikey960-4.9 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cd ~/dev/hikey960 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> . ./build/envsetup.sh >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> lunch hikey960-userdebug >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'm using Ubuntu 16.04 LTS as guest on Virtualbox. I tried >>>>>>>>>>>>>>>>> to do all the process once again from the beginning with a >>>>>>>>>>>>>>>>> new virtualbox >>>>>>>>>>>>>>>>> image, but I found the same problem. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks in advance! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> You received this message because you are subscribed to >>>>>>>>>>>>>>>>> the "Android Building" mailing list. >>>>>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> For more options, visit this group at >>>>>>>>>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>> You received this message because you are subscribed to >>>>>>>>>>>>>>>>> the Google Groups "Android Building" group. >>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>>>> from it, send an email to android-buildi...@googlegroups >>>>>>>>>>>>>>>>> .com. >>>>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>> "Android Building" mailing list. >>>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> For more options, visit this group at >>>>>>>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>> Google Groups "Android Building" group. >>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>> from it, send an email to [email protected] >>>>>>>>>>>>>>> . >>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> "Android Building" mailing list. >>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> For more options, visit this group at >>>>>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>>>>> >>>>>>>>>>>>> --- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "Android Building" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> "Android Building" mailing list. >>>>>>>>>>> To post to this group, send email to >>>>>>>>>>> [email protected] >>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>> [email protected] >>>>>>>>>>> For more options, visit this group at >>>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>>> >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "Android Building" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the >>>>>>>>> "Android Building" mailing list. >>>>>>>>> To post to this group, send email to [email protected] >>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>> [email protected] >>>>>>>>> For more options, visit this group at >>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>> >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Android Building" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>> -- >>>>>>> You received this message because you are subscribed to the "Android >>>>>>> Building" mailing list. >>>>>>> To post to this group, send email to [email protected] >>>>>>> To unsubscribe from this group, send email to >>>>>>> [email protected] >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>> >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Android Building" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>> -- >>>>> You received this message because you are subscribed to the "Android >>>>> Building" mailing list. >>>>> To post to this group, send email to [email protected] >>>>> To unsubscribe from this group, send email to >>>>> [email protected] >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/android-building?hl=en >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Android Building" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> -- >>> You received this message because you are subscribed to the "Android >>> Building" mailing list. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/android-building?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Android Building" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > -- > You received this message because you are subscribed to the "Android > Building" mailing list. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/android-building?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "Android Building" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- -- You received this message because you are subscribed to the "Android Building" mailing list. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/android-building?hl=en --- You received this message because you are subscribed to the Google Groups "Android Building" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
