Thank you! It cuts about a quater off the time finder normally takes. On Friday, December 22, 2017 at 2:10:08 AM UTC, Colin Cross wrote: > > The patch stack at > https://android-review.googlesource.com/c/platform/build/soong/+/575773 > removes >700k lstat calls from finder, I'm curious if it makes any > difference for you. > > On Thu, Dec 21, 2017 at 12:41 PM, Peter Warrington <[email protected] > <javascript:>> wrote: > >> Thank you for all your help and good luck with improving finder! With >> regards to memory, I have a measly 4GB of ram so that is probally slowing >> finder down and I do expect it to be slow (I do plan on upgrading in case >> you are wondering). However, I do not believe what I am experiencing is the >> same as what Alessandro is experiencing. Again, Thanks for all your help! >> >> On Thursday, December 21, 2017 at 8:14:39 PM UTC, Colin Cross wrote: >>> >>> Yeah, this is showing very slow lstat calls: >>> [pid 4007] <... lstat resumed> {st_mode=S_IFDIR|0777, st_size=4096, >>> ...}) = 0 <0.020417> >>> vs. mine: >>> [pid 219359] <... lstat resumed> {st_mode=S_IFREG|0640, st_size=323161, >>> ...}) = 0 <0.000046> >>> >>> I don't know why yours would be so much slower, maybe just round trip >>> latency to the external drive and not enough memory to keep them all in >>> the inode cache? >>> >>> We can improve finder to drastically reduce the number of lstat calls, >>> but ninja is going to have to stat most of the files to determine what >>> needs building so I don't know how much improvement you will see. >>> >>> >>> On Thu, Dec 21, 2017 at 11:56 AM, Peter Warrington <[email protected]> >>> wrote: >>> >>>> Sorry about that. Is this attached strace any better? >>>> >>>> On Thursday, December 21, 2017 at 7:36:22 PM UTC, Colin Cross wrote: >>>>> >>>>> I don't see the data I'd expect with -T, it should have a time in >>>>> angle brackets after each finished syscall: >>>>> [pid 219359] <... lstat resumed> {st_mode=S_IFREG|0640, >>>>> st_size=323161, ...}) = 0 <0.000046> >>>>> >>>>> >>>>> On Thu, Dec 21, 2017 at 3:05 AM, Peter Warrington <[email protected] >>>>> > wrote: >>>>> >>>>>> Attached is a much longer strace with the -T option used. >>>>>> >>>>>> On Wednesday, December 20, 2017 at 11:16:53 PM UTC, Colin Cross wrote: >>>>>>> >>>>>>> The results of the finder are cached in out/.module_paths/files.db, >>>>>>> so once the command has completed successfully once the next runs will >>>>>>> be >>>>>>> much faster. You could try wiping that file to reproduce it again, or >>>>>>> run >>>>>>> with a different value for OUT_DIR (OUT_DIR=out2 lunch ...). >>>>>>> >>>>>>> On Wed, Dec 20, 2017 at 1:58 PM, Peter Warrington < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hello, the command has now started to work like normal after >>>>>>>> leaving it for a while, so my problem might not be the same as >>>>>>>> Alessandro's. Unfortunately, I cannot upload a bigger stack trace >>>>>>>> until >>>>>>>> tomorrow because of the time in my timezone. Sorry! >>>>>>>> >>>>>>>> On Wednesday, December 20, 2017 at 9:48:34 PM UTC, Colin Cross >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Can you attach a longer strace with -T ? >>>>>>>>> >>>>>>>>> On Wed, Dec 20, 2017 at 1:28 PM, Peter Warrington < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Thank you! That is very interesting. In case it helps, I >>>>>>>>>> benchmarked my external HDD and it has an average read rate of 92.0 >>>>>>>>>> MB/s >>>>>>>>>> and an average access time of 16.37 msec. However, I didn't have any >>>>>>>>>> trouble with this previously. >>>>>>>>>> >>>>>>>>>> On Wednesday, December 20, 2017 at 9:06:16 PM UTC, Colin Cross >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Nothing looks wrong in that strace, it is making forward >>>>>>>>>>> progress. It needs to read every directory and stat every file in >>>>>>>>>>> the >>>>>>>>>>> tree, is your filesystem very slow for some reason? >>>>>>>>>>> >>>>>>>>>>> One difference I see between the old implementation using the >>>>>>>>>>> find tool and the new implementation is that find seems to be able >>>>>>>>>>> to >>>>>>>>>>> avoiding calling stat on every directory entry, because the >>>>>>>>>>> relevant >>>>>>>>>>> information (whether the entry is a subdirectory or a file) is >>>>>>>>>>> contained in >>>>>>>>>>> the return from the getdents64 syscall. The Go implementation is >>>>>>>>>>> ignoring >>>>>>>>>>> that information and has to call lstat to get it. >>>>>>>>>>> >>>>>>>>>>> On Wed, Dec 20, 2017 at 11:32 AM, Peter Warrington < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Attached is a partial strace. >>>>>>>>>>>> >>>>>>>>>>>> On Wednesday, December 20, 2017 at 7:06:31 PM UTC, Colin Cross >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> From the stack trace after ctrl-c, lunch will be waiting for >>>>>>>>>>>>> the child soong_ui process to complete. An strace of that >>>>>>>>>>>>> process would be >>>>>>>>>>>>> interesting. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Dec 18, 2017 at 4:12 PM, Alessandro Brucato < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> This is the output of strace attached to the process in which >>>>>>>>>>>>>> I execute "lunch". >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> <https://lh3.googleusercontent.com/-Rk-T1kCPUnA/WjhYORJYCKI/AAAAAAAAGSY/LIzqOc58jvkTpj7dowKqjz1OsKoAy0dZQCLcBGAs/s1600/strace.PNG> >>>>>>>>>>>>>> >>>>>>>>>>>>>> When "lunch" is stuck, the output of strace ends with >>>>>>>>>>>>>> "wait4(-1,". All the remaining part appears when I abort "lunch" >>>>>>>>>>>>>> with >>>>>>>>>>>>>> CTRL+C. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Maybe this issue is due to the fact that I'm using a guest in >>>>>>>>>>>>>> Virtualbox? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Il giorno lunedì 18 dicembre 2017 23:45:48 UTC+1, Colin Cross >>>>>>>>>>>>>> ha scritto: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can you capture the full output of all the stack traces? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It's getting stuck while trying to walk all the checked-out >>>>>>>>>>>>>>> code looking for various build files. If it was completely >>>>>>>>>>>>>>> deadlocked then >>>>>>>>>>>>>>> it would have died on its own, so it's probably stuck in a loop >>>>>>>>>>>>>>> somehow. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Dec 18, 2017 at 8:12 AM, Alessandro Brucato < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hello, I'm trying to build the Android source code for my >>>>>>>>>>>>>>>> Hikey 960 board. >>>>>>>>>>>>>>>> I'm following this video tutorial >>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=RD_NZRKiZBU in parallel to >>>>>>>>>>>>>>>> the official Android guide >>>>>>>>>>>>>>>> https://source.android.com/setup/devices#hikey-boards. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Everything went fine until I reached the command "lunch >>>>>>>>>>>>>>>> hikey960-userdebug". It blocks, without returning any output. >>>>>>>>>>>>>>>> I left it >>>>>>>>>>>>>>>> "working" for 1 hour and then I aborted it (even if I know >>>>>>>>>>>>>>>> that this >>>>>>>>>>>>>>>> command should take few seconds). If I execute only "lunch" it >>>>>>>>>>>>>>>> shows the >>>>>>>>>>>>>>>> list of available build targets, and when I select >>>>>>>>>>>>>>>> "hikey960-userdebug" it >>>>>>>>>>>>>>>> blocks. When I abort it with CTRL+C, it says "Still alive, >>>>>>>>>>>>>>>> killing >>>>>>>>>>>>>>>> subprocesses" and then it shows a lot of messages referring to >>>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>>> Here is the screenshot https://ibb.co/k32KQR >>>>>>>>>>>>>>>> I noticed that also the "make" command (inside the AOSP >>>>>>>>>>>>>>>> directory) has the same behaviour. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This is the list of commands I've used during the process: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> sudo apt-get install openjdk-7-jdk >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> sudo apt-get install git-core gnupg flex bison gperf >>>>>>>>>>>>>>>> build-essential zip curl zlib1g-dev gcc-multilib g++-multilib >>>>>>>>>>>>>>>> libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev >>>>>>>>>>>>>>>> lib32z-dev >>>>>>>>>>>>>>>> ccache libgl1-mesa-dev libxml2-utils xsltproc unzip >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> mkdir output >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> export OUT_DIR_COMMON_BASE=/home/brucedh/output/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> mkdir ~/bin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> PATH=~/bin:$PATH >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> curl https://storage.googleapis.com/git-repo-downloads/repo >>>>>>>>>>>>>>>> > ~/bin/repo >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> chmod a+x ~/bin/repo >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> mkdir dev >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cd dev >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> mkdir hikey960 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cd hikey960 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> git config --global user.name "<REDACTED>" >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> git config --global user.email "<REDACTED>" >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> repo init -u >>>>>>>>>>>>>>>> https://android.googlesource.com/platform/manifest -b >>>>>>>>>>>>>>>> master >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> repo sync –j4 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> git clone >>>>>>>>>>>>>>>> https://android.googlesource.com/kernel/hikey-linaro >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cd hikey-linaro >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> git checkout -b android-hikey-linaro-4.9 >>>>>>>>>>>>>>>> origin/android-hikey-linaro-4.9 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> make ARCH=arm64 menuconfig >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> make ARCH=arm64 hikey960_defconfig >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> PATH=/home/brucedh/dev/hikey960/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/:$PATH >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-android- -j4 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cp arch/arm64/boot/dts/hisilicon/hi3660-hikey960.dtb >>>>>>>>>>>>>>>> ../device/linaro/hikey-kernel/hi3660-hikey960.dtb-4.9 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cp arch/arm64/boot/Image.gz >>>>>>>>>>>>>>>> ../device/linaro/hikey-kernel/Image.gz-hikey960-4.9 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cd ~/dev/hikey960 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> . ./build/envsetup.sh >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> lunch hikey960-userdebug >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'm using Ubuntu 16.04 LTS as guest on Virtualbox. I tried >>>>>>>>>>>>>>>> to do all the process once again from the beginning with a new >>>>>>>>>>>>>>>> virtualbox >>>>>>>>>>>>>>>> image, but I found the same problem. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks in advance! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>>> "Android Building" mailing list. >>>>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> For more options, visit this group at >>>>>>>>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>>> Google Groups "Android Building" group. >>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>>> from it, send an email to >>>>>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> "Android Building" mailing list. >>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> For more options, visit this group at >>>>>>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>>>>>> >>>>>>>>>>>>>> --- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> Google Groups "Android Building" group. >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> "Android Building" mailing list. >>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>> [email protected] >>>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>>> [email protected] >>>>>>>>>>>> For more options, visit this group at >>>>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "Android Building" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> "Android Building" mailing list. >>>>>>>>>> To post to this group, send email to [email protected] >>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>> [email protected] >>>>>>>>>> For more options, visit this group at >>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>> >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Android Building" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the >>>>>>>> "Android Building" mailing list. >>>>>>>> To post to this group, send email to [email protected] >>>>>>>> To unsubscribe from this group, send email to >>>>>>>> [email protected] >>>>>>>> For more options, visit this group at >>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>> >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Android Building" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>> -- >>>>>> You received this message because you are subscribed to the "Android >>>>>> Building" mailing list. >>>>>> To post to this group, send email to [email protected] >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected] >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/android-building?hl=en >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Android Building" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> -- >>>> You received this message because you are subscribed to the "Android >>>> Building" mailing list. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> [email protected] >>>> For more options, visit this group at >>>> http://groups.google.com/group/android-building?hl=en >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Android Building" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> -- >> You received this message because you are subscribed to the "Android >> Building" mailing list. >> To post to this group, send email to [email protected] >> <javascript:> >> To unsubscribe from this group, send email to >> [email protected] <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/android-building?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "Android Building" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > >
-- -- You received this message because you are subscribed to the "Android Building" mailing list. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/android-building?hl=en --- You received this message because you are subscribed to the Google Groups "Android Building" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
