Re: [android-building] "Build sandboxing disabled due to nsjail error. This may become fatal in the future."

2020-04-30 Thread 'Dan Willemsen' via Android Building 
Yes, we're familiar with the docker problems, and we'll likely have to live
with it for the time being: https://issuetracker.google.com/123210688 .
I've removed the above message from master and Android 11 (the other common
case was distributions that turned off user namespaces).

> I suppose this is not very helpful in my case for me as Docker is already
doing process isolation?

I'm actually less interested in it as a security isolation boundary, and
more interested in it providing more assurances about the behaviors we
expect from the build, and provide more safety by default:

* Guarantees that processes don't continue running past the end of the
build (or the part of the build). In practice, this should just be a safety
precaution, and if you're starting the build via the docker command line
every time, it should provide similar guarantees (if you're attaching to a
long-running docker instance, not so much).
* Allows us to turn off the network for part of the build. In most cases
right now this can be emulated by just turning off the network for that
docker container, but with the remote execution work that's ongoing, we
allow a daemon access to the network, but turn it off for the rest of the
build (which tunnels build requests to the daemon as necessary).
* Recently on master, turns off write access to the rest of the system
(except $srcdir, $outdir, $distdir, $home? something like that). Depending
on your docker setup, this safety may not be too important (just mounting
the necessary directories, and throwing away any changes made after the
build exists, for example).

So while we can't require the use of nsjail at this point, it may mean that
your build succeeds while the same build on another system using nsjail
fails. You may be able to configure docker similarly, but changing the
configuration during different parts of the build likely wouldn't be
possible.

I'd like to do things like turn the source directory read only, but I've
been hesitant to do so because it'll cause a larger behavior difference
between the nsjail users and the rest. Also on the list is hiding things
like /usr/include from the build, as we never want to use it. Potentially
changing what parts of the output tree are read/write vs read-only vs
invisible during different parts of the build is another idea I've had.

- Dan



On Thu, Apr 30, 2020 at 9:43 AM Ricky Ng-Adam  wrote:

> Hello!
>
> I'm building Android 10 in a Docker with overlay and getting the following
> error:
>
> Building Android
>
>
> 06:58:14 Build sandboxing disabled due to nsjail error. This may become
> fatal in the future.
>
> 06:58:14 Please let us know why nsjail doesn't work in your environment
> at:
>
> 06:58:14   https://groups.google.com/forum/#!forum/android-building
>
>
> 06:58:14   https://issuetracker.google.com/issues/new?component=381517
>
> https://github.com/google/nsjail
>
> A light-weight process isolation tool, making use of Linux namespaces and
> seccomp-bpf syscall filters (with help of the kafel bpf language)
>
> I suppose this is not very helpful in my case for me as Docker is already
> doing process isolation? I see it's possible to run nsjail but it requires
> --disable_proc and --privileged.
>
> I'm supposing nsjail exists to provide better performance?
>
> Does it work well with Overlay FS? I'm trying to build for every branch
> without copying over all source tree and output build objects.
>
> Thanks,
>
> Ricky
>
> --
> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-building@googlegroups.com
> To unsubscribe from this group, send email to
> android-building+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Android Building" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to android-building+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/android-building/8cd8ef17-d322-404e-986e-feb0f5c7c3c8%40googlegroups.com
> 
> .
>

-- 
-- 
You received this message because you are subscribed to the "Android Building" 
mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to android-building+unsubscr...@googlegroups.com.
To view this discussion on the

Re: [android-building] Memory leak?

2020-04-30 Thread 'Dan Willemsen' via Android Building 
The minimum requirements for the build is 16GB of RAM available (and even
with that it can be difficult later in the build). The virtualization
mention was meant to say that a machine with 16GB of RAM wasn't enough when
virtualizing -- the virtual machine would need 16GB allocated to it.

`m -j1` means that we'll only launch a single process at a time, but many
of those spin up a number of threads on their own. Parallelism can have
some effect on peak RAM use, but I wouldn't expect that to significantly
affect things, at least for soong_build.

Which target you're building can have a reasonably big impact on RAM usage
(as 64-bit targets need both 64 and 32 bit configurations). On my machine
with aosp-master:

 aosp_cf_x86_phone - soong_build peaks at ~6GB  (and allocated ~20GB over
its lifetime). next, kati takes ~4GB
 aosp_crosshatch - soong_build peaks at ~8.5GB  (and allocated ~26GB over
its lifetime). next, kati takes ~6GB

For some of our larger branches/targets, I've seen peaks at >14GB (~50GB
lifetime allocs), with kati using >10GB.

- Dan

On Thu, Apr 30, 2020 at 12:05 PM Eliane Fonseca 
wrote:

> same problem with 2 processes and I have only 8 GB of physical RAM and it
> is not virtualized system . any timing for fix ?
>
> Em 30 de abr de 2020 13:43, "Silver Rampart" 
> escreveu:
>
> When running a normal build, process soong_build quickly fills up all
> available memory (6GB in under 5 minutes) and starts to swap. Quad-core i5.
> I understand this might have something to do with parallelism, but I run
> make with -j1 and soong_build is still running multiple threads.
>
> --
> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-building@googlegroups.com
> To unsubscribe from this group, send email to
> android-building+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Android Building" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to android-building+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/android-building/ca25a692-cd1f-4096-be61-552017644208%40googlegroups.com
> 
> .
>
>
> --
> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-building@googlegroups.com
> To unsubscribe from this group, send email to
> android-building+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Android Building" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to android-building+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/android-building/CAJ%2BSQK6ow5BwiNHUpt9bxTcjnJsyVBptX6mkL%2B4Pn9KaA3mNRA%40mail.gmail.com
> 
> .
>

-- 
-- 
You received this message because you are subscribed to the "Android Building" 
mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to android-building+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/android-building/CALQgHdnkEefTzYxU%2BYnGH5RN27vROLmBitc6VkfgVCwjLMuN3g%40mail.gmail.com.

Re: [android-building] Memory leak?

2020-04-30 Thread Eliane Fonseca 
same problem with 2 processes and I have only 8 GB of physical RAM and it
is not virtualized system . any timing for fix ?

Em 30 de abr de 2020 13:43, "Silver Rampart" 
escreveu:

When running a normal build, process soong_build quickly fills up all
available memory (6GB in under 5 minutes) and starts to swap. Quad-core i5.
I understand this might have something to do with parallelism, but I run
make with -j1 and soong_build is still running multiple threads.

-- 
-- 
You received this message because you are subscribed to the "Android
Building" mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups
"Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to android-building+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/android-building/ca25a692-cd1f-4096-be61-552017644208%40googlegroups.
com

.

-- 
-- 
You received this message because you are subscribed to the "Android Building" 
mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to android-building+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/android-building/CAJ%2BSQK6ow5BwiNHUpt9bxTcjnJsyVBptX6mkL%2B4Pn9KaA3mNRA%40mail.gmail.com.

[android-building] Memory leak?

2020-04-30 Thread Silver Rampart 
When running a normal build, process soong_build quickly fills up all 
available memory (6GB in under 5 minutes) and starts to swap. Quad-core i5. 
I understand this might have something to do with parallelism, but I run 
make with -j1 and soong_build is still running multiple threads.

-- 
-- 
You received this message because you are subscribed to the "Android Building" 
mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to android-building+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/android-building/ca25a692-cd1f-4096-be61-552017644208%40googlegroups.com.

Re: [android-building] equivalent flag "optimize" from Android.bp in Android.mk

2020-04-30 Thread Kbc K 
Thank Dan, I will try it

On Tuesday, April 28, 2020 at 10:29:24 PM UTC+3, Dan Willemsen wrote:
>
> `LOCAL_PROGUARD_ENABLED := disabled` would be the equivalent of `optimize: 
> { enabled: false }`
>
> - Dan
>
> On Tue, Apr 28, 2020 at 11:51 AM Kbc K > 
> wrote:
>
>> Hi Everyone,
>>
>> Soong build system "Android.bp" has a useful flag called "optimize" which 
>> allow to disable compiler optimization.
>>
>> I'm looking for something equivalent in Android.mk system.
>>
>> BR, K
>>
>> -- 
>> -- 
>> You received this message because you are subscribed to the "Android 
>> Building" mailing list.
>> To post to this group, send email to android-...@googlegroups.com 
>> 
>> To unsubscribe from this group, send email to
>> android-...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/android-building?hl=en
>>
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Android Building" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to android-...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/android-building/8d5ed84a-8753-44c9-999f-c55e7874b665%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
-- 
You received this message because you are subscribed to the "Android Building" 
mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to android-building+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/android-building/378de9ef-f56c-4087-ad87-688337f00876%40googlegroups.com.

[android-building] "Build sandboxing disabled due to nsjail error. This may become fatal in the future."

2020-04-30 Thread Ricky Ng-Adam 
Hello!

I'm building Android 10 in a Docker with overlay and getting the following 
error:

Building Android

   
06:58:14 Build sandboxing disabled due to nsjail error. This may become 
fatal in the future.
   
06:58:14 Please let us know why nsjail doesn't work in your environment 
at:

06:58:14   https://groups.google.com/forum/#!forum/android-building


06:58:14   https://issuetracker.google.com/issues/new?component=381517   

https://github.com/google/nsjail

A light-weight process isolation tool, making use of Linux namespaces and 
seccomp-bpf syscall filters (with help of the kafel bpf language) 

I suppose this is not very helpful in my case for me as Docker is already 
doing process isolation? I see it's possible to run nsjail but it requires
--disable_proc and --privileged.

I'm supposing nsjail exists to provide better performance?

Does it work well with Overlay FS? I'm trying to build for every branch 
without copying over all source tree and output build objects.

Thanks, 

Ricky

-- 
-- 
You received this message because you are subscribed to the "Android Building" 
mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to android-building+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/android-building/8cd8ef17-d322-404e-986e-feb0f5c7c3c8%40googlegroups.com.