The patch stack at https://android-review.googlesource.com/c/platform/build/soong/+/575773 removes >700k lstat calls from finder, I'm curious if it makes any difference for you.
On Thu, Dec 21, 2017 at 12:41 PM, Peter Warrington <[email protected]> wrote: > Thank you for all your help and good luck with improving finder! With > regards to memory, I have a measly 4GB of ram so that is probally slowing > finder down and I do expect it to be slow (I do plan on upgrading in case > you are wondering). However, I do not believe what I am experiencing is the > same as what Alessandro is experiencing. Again, Thanks for all your help! > > On Thursday, December 21, 2017 at 8:14:39 PM UTC, Colin Cross wrote: >> >> Yeah, this is showing very slow lstat calls: >> [pid 4007] <... lstat resumed> {st_mode=S_IFDIR|0777, st_size=4096, >> ...}) = 0 <0.020417> >> vs. mine: >> [pid 219359] <... lstat resumed> {st_mode=S_IFREG|0640, st_size=323161, >> ...}) = 0 <0.000046> >> >> I don't know why yours would be so much slower, maybe just round trip >> latency to the external drive and not enough memory to keep them all in >> the inode cache? >> >> We can improve finder to drastically reduce the number of lstat calls, >> but ninja is going to have to stat most of the files to determine what >> needs building so I don't know how much improvement you will see. >> >> >> On Thu, Dec 21, 2017 at 11:56 AM, Peter Warrington <[email protected]> >> wrote: >> >>> Sorry about that. Is this attached strace any better? >>> >>> On Thursday, December 21, 2017 at 7:36:22 PM UTC, Colin Cross wrote: >>>> >>>> I don't see the data I'd expect with -T, it should have a time in angle >>>> brackets after each finished syscall: >>>> [pid 219359] <... lstat resumed> {st_mode=S_IFREG|0640, st_size=323161, >>>> ...}) = 0 <0.000046> >>>> >>>> >>>> On Thu, Dec 21, 2017 at 3:05 AM, Peter Warrington <[email protected]> >>>> wrote: >>>> >>>>> Attached is a much longer strace with the -T option used. >>>>> >>>>> On Wednesday, December 20, 2017 at 11:16:53 PM UTC, Colin Cross wrote: >>>>>> >>>>>> The results of the finder are cached in out/.module_paths/files.db, >>>>>> so once the command has completed successfully once the next runs will be >>>>>> much faster. You could try wiping that file to reproduce it again, or >>>>>> run >>>>>> with a different value for OUT_DIR (OUT_DIR=out2 lunch ...). >>>>>> >>>>>> On Wed, Dec 20, 2017 at 1:58 PM, Peter Warrington < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hello, the command has now started to work like normal after leaving >>>>>>> it for a while, so my problem might not be the same as Alessandro's. >>>>>>> Unfortunately, I cannot upload a bigger stack trace until tomorrow >>>>>>> because >>>>>>> of the time in my timezone. Sorry! >>>>>>> >>>>>>> On Wednesday, December 20, 2017 at 9:48:34 PM UTC, Colin Cross wrote: >>>>>>>> >>>>>>>> Can you attach a longer strace with -T ? >>>>>>>> >>>>>>>> On Wed, Dec 20, 2017 at 1:28 PM, Peter Warrington < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Thank you! That is very interesting. In case it helps, I >>>>>>>>> benchmarked my external HDD and it has an average read rate of 92.0 >>>>>>>>> MB/s >>>>>>>>> and an average access time of 16.37 msec. However, I didn't have any >>>>>>>>> trouble with this previously. >>>>>>>>> >>>>>>>>> On Wednesday, December 20, 2017 at 9:06:16 PM UTC, Colin Cross >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Nothing looks wrong in that strace, it is making forward >>>>>>>>>> progress. It needs to read every directory and stat every file in >>>>>>>>>> the >>>>>>>>>> tree, is your filesystem very slow for some reason? >>>>>>>>>> >>>>>>>>>> One difference I see between the old implementation using the >>>>>>>>>> find tool and the new implementation is that find seems to be able to >>>>>>>>>> avoiding calling stat on every directory entry, because the relevant >>>>>>>>>> information (whether the entry is a subdirectory or a file) is >>>>>>>>>> contained in >>>>>>>>>> the return from the getdents64 syscall. The Go implementation is >>>>>>>>>> ignoring >>>>>>>>>> that information and has to call lstat to get it. >>>>>>>>>> >>>>>>>>>> On Wed, Dec 20, 2017 at 11:32 AM, Peter Warrington < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Attached is a partial strace. >>>>>>>>>>> >>>>>>>>>>> On Wednesday, December 20, 2017 at 7:06:31 PM UTC, Colin Cross >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> From the stack trace after ctrl-c, lunch will be waiting for >>>>>>>>>>>> the child soong_ui process to complete. An strace of that process >>>>>>>>>>>> would be >>>>>>>>>>>> interesting. >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Dec 18, 2017 at 4:12 PM, Alessandro Brucato < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> This is the output of strace attached to the process in which >>>>>>>>>>>>> I execute "lunch". >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> <https://lh3.googleusercontent.com/-Rk-T1kCPUnA/WjhYORJYCKI/AAAAAAAAGSY/LIzqOc58jvkTpj7dowKqjz1OsKoAy0dZQCLcBGAs/s1600/strace.PNG> >>>>>>>>>>>>> >>>>>>>>>>>>> When "lunch" is stuck, the output of strace ends with >>>>>>>>>>>>> "wait4(-1,". All the remaining part appears when I abort "lunch" >>>>>>>>>>>>> with >>>>>>>>>>>>> CTRL+C. >>>>>>>>>>>>> >>>>>>>>>>>>> Maybe this issue is due to the fact that I'm using a guest in >>>>>>>>>>>>> Virtualbox? >>>>>>>>>>>>> >>>>>>>>>>>>> Il giorno lunedì 18 dicembre 2017 23:45:48 UTC+1, Colin Cross >>>>>>>>>>>>> ha scritto: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you capture the full output of all the stack traces? >>>>>>>>>>>>>> >>>>>>>>>>>>>> It's getting stuck while trying to walk all the checked-out >>>>>>>>>>>>>> code looking for various build files. If it was completely >>>>>>>>>>>>>> deadlocked then >>>>>>>>>>>>>> it would have died on its own, so it's probably stuck in a loop >>>>>>>>>>>>>> somehow. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Dec 18, 2017 at 8:12 AM, Alessandro Brucato < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello, I'm trying to build the Android source code for my >>>>>>>>>>>>>>> Hikey 960 board. >>>>>>>>>>>>>>> I'm following this video tutorial https://www.youtube.c >>>>>>>>>>>>>>> om/watch?v=RD_NZRKiZBU in parallel to the official Android >>>>>>>>>>>>>>> guide https://source.android.com/setup/devices#hikey-boards. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Everything went fine until I reached the command "lunch >>>>>>>>>>>>>>> hikey960-userdebug". It blocks, without returning any output. I >>>>>>>>>>>>>>> left it >>>>>>>>>>>>>>> "working" for 1 hour and then I aborted it (even if I know that >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> command should take few seconds). If I execute only "lunch" it >>>>>>>>>>>>>>> shows the >>>>>>>>>>>>>>> list of available build targets, and when I select >>>>>>>>>>>>>>> "hikey960-userdebug" it >>>>>>>>>>>>>>> blocks. When I abort it with CTRL+C, it says "Still alive, >>>>>>>>>>>>>>> killing >>>>>>>>>>>>>>> subprocesses" and then it shows a lot of messages referring to >>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>> Here is the screenshot https://ibb.co/k32KQR >>>>>>>>>>>>>>> I noticed that also the "make" command (inside the AOSP >>>>>>>>>>>>>>> directory) has the same behaviour. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is the list of commands I've used during the process: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> sudo apt-get install openjdk-7-jdk >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> sudo apt-get install git-core gnupg flex bison gperf >>>>>>>>>>>>>>> build-essential zip curl zlib1g-dev gcc-multilib g++-multilib >>>>>>>>>>>>>>> libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev >>>>>>>>>>>>>>> lib32z-dev >>>>>>>>>>>>>>> ccache libgl1-mesa-dev libxml2-utils xsltproc unzip >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> mkdir output >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> export OUT_DIR_COMMON_BASE=/home/brucedh/output/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> mkdir ~/bin >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> PATH=~/bin:$PATH >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> curl https://storage.googleapis.com/git-repo-downloads/repo >>>>>>>>>>>>>>> > ~/bin/repo >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> chmod a+x ~/bin/repo >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> mkdir dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cd dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> mkdir hikey960 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cd hikey960 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> git config --global user.name "<REDACTED>" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> git config --global user.email "<REDACTED>" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> repo init -u https://android.googlesource.c >>>>>>>>>>>>>>> om/platform/manifest -b master >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> repo sync –j4 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> git clone https://android.googlesource.c >>>>>>>>>>>>>>> om/kernel/hikey-linaro >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cd hikey-linaro >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> git checkout -b android-hikey-linaro-4.9 >>>>>>>>>>>>>>> origin/android-hikey-linaro-4.9 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> make ARCH=arm64 menuconfig >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> make ARCH=arm64 hikey960_defconfig >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> PATH=/home/brucedh/dev/hikey960/prebuilts/gcc/linux-x86/ >>>>>>>>>>>>>>> aarch64/aarch64-linux-android-4.9/bin/:$PATH >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-android- -j4 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cp arch/arm64/boot/dts/hisilicon/hi3660-hikey960.dtb >>>>>>>>>>>>>>> ../device/linaro/hikey-kernel/hi3660-hikey960.dtb-4.9 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cp arch/arm64/boot/Image.gz ../device/linaro/hikey-kernel/ >>>>>>>>>>>>>>> Image.gz-hikey960-4.9 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cd ~/dev/hikey960 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> . ./build/envsetup.sh >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> lunch hikey960-userdebug >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm using Ubuntu 16.04 LTS as guest on Virtualbox. I tried >>>>>>>>>>>>>>> to do all the process once again from the beginning with a new >>>>>>>>>>>>>>> virtualbox >>>>>>>>>>>>>>> image, but I found the same problem. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks in advance! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>> "Android Building" mailing list. >>>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> For more options, visit this group at >>>>>>>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>> Google Groups "Android Building" group. >>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>> from it, send an email to [email protected] >>>>>>>>>>>>>>> . >>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> "Android Building" mailing list. >>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> For more options, visit this group at >>>>>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>>>>> >>>>>>>>>>>>> --- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "Android Building" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> "Android Building" mailing list. >>>>>>>>>>> To post to this group, send email to >>>>>>>>>>> [email protected] >>>>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>>>> [email protected] >>>>>>>>>>> For more options, visit this group at >>>>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>>>> >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "Android Building" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the >>>>>>>>> "Android Building" mailing list. >>>>>>>>> To post to this group, send email to [email protected] >>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>> [email protected] >>>>>>>>> For more options, visit this group at >>>>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>>>> >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Android Building" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>> -- >>>>>>> You received this message because you are subscribed to the "Android >>>>>>> Building" mailing list. >>>>>>> To post to this group, send email to [email protected] >>>>>>> To unsubscribe from this group, send email to >>>>>>> [email protected] >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/android-building?hl=en >>>>>>> >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Android Building" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>> -- >>>>> You received this message because you are subscribed to the "Android >>>>> Building" mailing list. >>>>> To post to this group, send email to [email protected] >>>>> To unsubscribe from this group, send email to >>>>> [email protected] >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/android-building?hl=en >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Android Building" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> -- >>> You received this message because you are subscribed to the "Android >>> Building" mailing list. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/android-building?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Android Building" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > -- > You received this message because you are subscribed to the "Android > Building" mailing list. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/android-building?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "Android Building" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- -- You received this message because you are subscribed to the "Android Building" mailing list. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/android-building?hl=en --- You received this message because you are subscribed to the Google Groups "Android Building" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
