Re: [android-building] Generate external/llvm Intrinsics.gen and Attributes.inc on android10

[Paschalis Mpeis]

I've eventually made this work. My last issue was due to the target 
*libLLVM_android* being only available for host architectures (linux/darwin 
x86).
I made another target just for Android, and it did work, but since llvm 
version was too old (3.8 in particular), including those c++11 headers in 
art/compiler
have caused a lot of other issues. Making it another library and using that 
one from art/compiler would still wouldn't solve it.

Tinkering some art headers was messy, so I decided that the only solution 
was to use a more up-to-date llvm sources.
For this I've tried many many things, including:
1. cross compiling llvm, which worked but linking *libLLVM.*so against 
*libart-compiler.so* wouldn't, because Android NDK, has modified the std 
namespace. So I had unresolved symbols.
2. tried to compile NDK from sources to "work around" the namespace issue 
but I eventually gave up
3. tried to compile using the exact same flags that AOSP users for libart 
(except the c++11 one which LLVM requires), but stuck on issues as well.

I ended up doing what I've tried to avoid: *integrated llvm9 sources into 
the soong build system as another external project*, in a similar fashion 
with the outdated external/llvm.
To avoid collisions I named it external/llvmx (also for any blueprint 
targets as well), since the outdated llvm has dependencies with other parts 
of the AOSP.
I've created new blueprint targets (and referenced them in existing ones) 
for new code that were required for: libLLVM.so, llc, opt, and llvm-link.

The only difference from the external/llvm build process is that I didn't 
followed the same route for files generated by llvm's tblgen.
The tblgen dependencies have increased, and each of them required special 
handling (targets in .bp backed by some go lang scripts).
So I removed all the tblgen related targets from .bp files, and I'm 
including them as prebuilt headers from my cross-compiled version.

I can now link libLLVM.so against libart-compiler.so and actually use it. 
llvm9 headers do not cause any issues now.

Cheers,
Paschalis





On Tuesday, March 17, 2020 at 7:42:16 PM UTC+2, Paschalis Mpeis wrote:
>
> Hi again,
>
> I hope everybody is doing well given these days' circumstances!
>
> I have a follow-up question on my previous attempts to use LLVM.
> What I want is to include *libLLVM_android* on l*ibart-compiler* and use 
> it on an arm64 device (sailfish) in particular.
>
> Now, I can successfully compile llvm components I need using:
> m libLLVM_android opt llc llvm-linker
>
> Then I install the library and the 3 tools on a sailfish device, and when 
> executing llc for example I can see that it can successfully utilize the 
> *libLLVM_android* library.
> But when I try to include the libLLVM_android on art/compiler/Android.bp I 
> get issues.
>
> Here's my changes:
> *art/compiler/Android.bp:*
> libart_cc_defaults {
> name: "some_llvm_defaults",
> defaults: [
> // I am not using llvm-defaults as compilation will break on 
> warnings like: -Wc++14-extensions
> // "llvm-defaults",
> "force_build_llvm_components",
> ],
> target: {
> android: {
> srcs: [
> // my additional .cc files go here..
> ],
> },
> },
> // I think this is the proper way to include the headers
> header_libs: ["llvm-headers"],
> // not sure about the next 2 lines
> // export_shared_lib_headers: ["libLLVM_android"],
> // compile_multilib: "first",
>
> // that's how I should include the library I think, but it gives me 
> the error#1 listed below
> shared_libs: ["libLLVM_android"],
> cflags: [
> // some flags necessary for LLVM
> "-D__STDC_LIMIT_MACROS",
> "-D__STDC_CONSTANT_MACROS",
> "-Wno-unused-parameter",
> ],
> // headers shouldn't be included like this I think, but instead
> // with someting like header_libs that I did above
> include_dirs: [
> "external/llvm/include",
> ],
> }
>
> On the external/llvm/Android.bp I can see here 
> 
>  that 
> it does export the relevant header directories for android targets,
> and since I have successfully compiled and installed libLLVM_android on a 
> device, I can't see why it won't work.
>
> The error#1 is:
>
>> out/soong/.bootstrap/bin/soong_build -t -l out/.module_paths/Android.bp.list 
> -b out/soong
> -n out -d out/soong/build.ninja.d -globFile out/soong/.bootstrap/build-
> globs.ninja -o
> out/soong/build.ninja Android.bp
>
> internal error: panic in WalkDeps(module "apex_test_build_features" 
> variant "android_common_apex_test_build_features",
>
> github.com/google/blueprint.(*baseModuleContext).WalkDeps.func1, ) for 
> dependency module "prebuilt_libLLVM_android"
> variant "android_arm_armv8-a_kryo_core_shared_apex_test_build_features" in 
> GenerateBuildActions for 

Re: [android-building] Generate external/llvm Intrinsics.gen and Attributes.inc on android10

[Paschalis Mpeis]

Hi again,

I hope everybody is doing well given these days' circumstances!

I have a follow-up question on my previous attempts to use LLVM.
What I want is to include *libLLVM_android* on l*ibart-compiler* and use it 
on an arm64 device (sailfish) in particular.

Now, I can successfully compile llvm components I need using:
m libLLVM_android opt llc llvm-linker

Then I install the library and the 3 tools on a sailfish device, and when 
executing llc for example I can see that it can successfully utilize the 
*libLLVM_android* library.
But when I try to include the libLLVM_android on art/compiler/Android.bp I 
get issues.

Here's my changes:
*art/compiler/Android.bp:*
libart_cc_defaults {
name: "some_llvm_defaults",
defaults: [
// I am not using llvm-defaults as compilation will break on 
warnings like: -Wc++14-extensions
// "llvm-defaults",
"force_build_llvm_components",
],
target: {
android: {
srcs: [
// my additional .cc files go here..
],
},
},
// I think this is the proper way to include the headers
header_libs: ["llvm-headers"],
// not sure about the next 2 lines
// export_shared_lib_headers: ["libLLVM_android"],
// compile_multilib: "first",

// that's how I should include the library I think, but it gives me the 
error#1 listed below
shared_libs: ["libLLVM_android"],
cflags: [
// some flags necessary for LLVM
"-D__STDC_LIMIT_MACROS",
"-D__STDC_CONSTANT_MACROS",
"-Wno-unused-parameter",
],
// headers shouldn't be included like this I think, but instead
// with someting like header_libs that I did above
include_dirs: [
"external/llvm/include",
],
}

On the external/llvm/Android.bp I can see here 

 that 
it does export the relevant header directories for android targets,
and since I have successfully compiled and installed libLLVM_android on a 
device, I can't see why it won't work.

The error#1 is:

> out/soong/.bootstrap/bin/soong_build -t -l out/.module_paths/Android.bp.list 
-b out/soong
-n out -d out/soong/build.ninja.d -globFile out/soong/.bootstrap/build-globs
.ninja -o
out/soong/build.ninja Android.bp

internal error: panic in WalkDeps(module "apex_test_build_features" variant 
"android_common_apex_test_build_features",

github.com/google/blueprint.(*baseModuleContext).WalkDeps.func1, ) for 
dependency module "prebuilt_libLLVM_android"
variant "android_arm_armv8-a_kryo_core_shared_apex_test_build_features" in 
GenerateBuildActions for module
"apex_test_build_features" variant "android_common_apex_test_build_features"

Requesting an invalid path
(full error log here 
)

Is it because I have to include libLLVM_android in some apex configuration 
so some checks for its inclusion to libart-compiler can now pass?
Or is it  that the prebuilt_libLLVM_android is missing?

On android6, I used to include libLLVM like this and it worked fine:
*art/compiler/Android.mk:*
...
LOCAL_SHARED_LIBRARIES += libLLVM
LOCAL_CFLAGS += -D__STDC_LIMIT_MACROS \
 -D__STDC_CONSTANT_MACROS
LOCAL_C_INCLUDES += $(LOCAL_LLVM_INCLUDES)


ifeq ($$(art_target_or_host),target)
  # I think I am missing this part on Android.bp
  include $(LLVM_DEVICE_BUILD_MK)
endif
include $(LLVM_GEN_INTRINSICS_MK)
...


Thanks in advance,
Paschalis


On Tuesday, March 3, 2020 at 8:41:49 PM UTC+2, Paschalis Mpeis wrote:
>
> Thanks a lot Colin for taking the time to read through my message and 
> reply.
> I ended up using Android.bp files as you suggested.
>
> Since all these tools were dropped from building in android10, I though 
> they were never converted to blueprint files.
> It turns out commit 7623033 had them, and with minor changes I made them 
> compiling for target device.
>
>
>
> On Tuesday, March 3, 2020 at 12:25:31 AM UTC+2, Colin Cross wrote:
>>
>> Those files are generated by the llvm_tblgen modules in 
>> external/llvm/Android.bp.  You'll find their outputs at:
>>
>> out/soong/.intermediates/external/llvm/llvm-gen-attributes/gen/llvm/IR/Attributes.inc
>>
>> out/soong/.intermediates/external/llvm/llvm-gen-intrinsics/gen/llvm/IR/Intrinsics.gen
>>
>> As a hack you could copy those files to the places where the Android.mk 
>> files create them, just be aware they will not be updated if you make 
>> changes to the source files and rebuild.  You could also try reverting the 
>> change that removed generating those files in Make:
>> commit 7ebb1e3516ebd630aa3fe94f9b4cca2c2dbb09ae
>> Author: Dan Willemsen 
>> Date:   Thu Nov 1 00:16:34 2018 -0700
>>
>> Disable tblgen in Make
>> 
>> The last user of these have been removed. simpleperf is still using
>> LLVM_HOST_BUILD_MK and LLVM_DEVICE_BUILD_MK, so these can't all 

Re: [android-building] Generate external/llvm Intrinsics.gen and Attributes.inc on android10

[Paschalis Mpeis]

Thanks a lot Colin for taking the time to read through my message and reply.
I ended up using Android.bp files as you suggested.

Since all these tools were dropped from building in android10, I though 
they were never converted to blueprint files.
It turns out commit 7623033 had them, and with minor changes I made them 
compiling for target device.



On Tuesday, March 3, 2020 at 12:25:31 AM UTC+2, Colin Cross wrote:
>
> Those files are generated by the llvm_tblgen modules in 
> external/llvm/Android.bp.  You'll find their outputs at:
>
> out/soong/.intermediates/external/llvm/llvm-gen-attributes/gen/llvm/IR/Attributes.inc
>
> out/soong/.intermediates/external/llvm/llvm-gen-intrinsics/gen/llvm/IR/Intrinsics.gen
>
> As a hack you could copy those files to the places where the Android.mk 
> files create them, just be aware they will not be updated if you make 
> changes to the source files and rebuild.  You could also try reverting the 
> change that removed generating those files in Make:
> commit 7ebb1e3516ebd630aa3fe94f9b4cca2c2dbb09ae
> Author: Dan Willemsen >
> Date:   Thu Nov 1 00:16:34 2018 -0700
>
> Disable tblgen in Make
> 
> The last user of these have been removed. simpleperf is still using
> LLVM_HOST_BUILD_MK and LLVM_DEVICE_BUILD_MK, so these can't all be
> removed yet.
> 
> Test: treehugger
> Change-Id: I0b3966fa5630e9795a1e7d7b9658f9a2bcad5271
>
> You'll likely have better success building opt, llvm-link and llc using 
> Android.bp files than trying to get the old Android.mk files to work though.
>
> On Mon, Mar 2, 2020 at 12:49 PM Paschalis Mpeis  > wrote:
>
>> Hi there,
>>
>> I am building some tools from *external/llvm*, specifically *opt*, 
>> *llvm-link*, and* llc*.
>> For this, I revived the relevant Android.mk from previous commits
>> (for each of the tools, and a relevant portion at 
>> external/llvm/Android.mk).
>> They are normally picked up by the soong build system, and I can for 
>> example call: `m opt`.
>>
>> However, these tools depend on *two intermediates*:
>>
>>- 
>>
>> target/product//gen/EXECUTABLES/opt_intermediates/llvm/IR/Attributes.inc
>>- 
>>
>> target/product//gen/EXECUTABLES/opt_intermediates/llvm/IR/Intrinsics.gen
>>
>> I have enabled FORCE_BUILD_LLVM_COMPONENTS (see here 
>> 
>> )
>> and I can see there are relevant entries for what I want at lines: 148 
>> 
>>  and  154 
>> 
>> .
>>
>> So, is my question is: is there a way to generate those 2 intermediate 
>> files using the existing Android.bp setup on android10-release branch 
>> 
>> ?
>>
>> Cheers,
>> Paschalis
>>
>> -- 
>> -- 
>> You received this message because you are subscribed to the "Android 
>> Building" mailing list.
>> To post to this group, send email to android-...@googlegroups.com 
>> 
>> To unsubscribe from this group, send email to
>> android-...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/android-building?hl=en
>>
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Android Building" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to android-...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/android-building/6e937488-80cb-42c8-b15c-087cafb7c400%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
-- 
You received this message because you are subscribed to the "Android Building" 
mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to android-building+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/android-building/0feac059-4b45-4dce-a239-62c303bc9e26%40googlegroups.com.

Re: [android-building] Generate external/llvm Intrinsics.gen and Attributes.inc on android10

['Colin Cross' via Android Building]

Those files are generated by the llvm_tblgen modules in
external/llvm/Android.bp.  You'll find their outputs at:
out/soong/.intermediates/external/llvm/llvm-gen-attributes/gen/llvm/IR/Attributes.inc
out/soong/.intermediates/external/llvm/llvm-gen-intrinsics/gen/llvm/IR/Intrinsics.gen

As a hack you could copy those files to the places where the Android.mk
files create them, just be aware they will not be updated if you make
changes to the source files and rebuild.  You could also try reverting the
change that removed generating those files in Make:
commit 7ebb1e3516ebd630aa3fe94f9b4cca2c2dbb09ae
Author: Dan Willemsen 
Date:   Thu Nov 1 00:16:34 2018 -0700

Disable tblgen in Make

The last user of these have been removed. simpleperf is still using
LLVM_HOST_BUILD_MK and LLVM_DEVICE_BUILD_MK, so these can't all be
removed yet.

Test: treehugger
Change-Id: I0b3966fa5630e9795a1e7d7b9658f9a2bcad5271

You'll likely have better success building opt, llvm-link and llc using
Android.bp files than trying to get the old Android.mk files to work though.

On Mon, Mar 2, 2020 at 12:49 PM Paschalis Mpeis 
wrote:

> Hi there,
>
> I am building some tools from *external/llvm*, specifically *opt*,
> *llvm-link*, and* llc*.
> For this, I revived the relevant Android.mk from previous commits
> (for each of the tools, and a relevant portion at
> external/llvm/Android.mk).
> They are normally picked up by the soong build system, and I can for
> example call: `m opt`.
>
> However, these tools depend on *two intermediates*:
>
>-
>
> target/product//gen/EXECUTABLES/opt_intermediates/llvm/IR/Attributes.inc
>-
>
> target/product//gen/EXECUTABLES/opt_intermediates/llvm/IR/Intrinsics.gen
>
> I have enabled FORCE_BUILD_LLVM_COMPONENTS (see here
> 
> )
> and I can see there are relevant entries for what I want at lines: 148
> 
>  and  154
> 
> .
>
> So, is my question is: is there a way to generate those 2 intermediate
> files using the existing Android.bp setup on android10-release branch
> 
> ?
>
> Cheers,
> Paschalis
>
> --
> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-building@googlegroups.com
> To unsubscribe from this group, send email to
> android-building+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Android Building" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to android-building+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/android-building/6e937488-80cb-42c8-b15c-087cafb7c400%40googlegroups.com
> 
> .
>

-- 
-- 
You received this message because you are subscribed to the "Android Building" 
mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to android-building+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/android-building/CAMbhsRT279w0e%2BT%2B3o2zEQBc0JFEo6gEjfgrJjiLq%3DSXpZ25MQ%40mail.gmail.com.

[android-building] Generate external/llvm Intrinsics.gen and Attributes.inc on android10

[Paschalis Mpeis]

Hi there,

I am building some tools from *external/llvm*, specifically *opt*, 
*llvm-link*, and* llc*.
For this, I revived the relevant Android.mk from previous commits
(for each of the tools, and a relevant portion at external/llvm/Android.mk).
They are normally picked up by the soong build system, and I can for 
example call: `m opt`.

However, these tools depend on *two intermediates*:

   - 
   
target/product//gen/EXECUTABLES/opt_intermediates/llvm/IR/Attributes.inc
   - 
   
target/product//gen/EXECUTABLES/opt_intermediates/llvm/IR/Intrinsics.gen

I have enabled FORCE_BUILD_LLVM_COMPONENTS (see here 

)
and I can see there are relevant entries for what I want at lines: 148 

 and  154 

.

So, is my question is: is there a way to generate those 2 intermediate 
files using the existing Android.bp setup on android10-release branch 

?

Cheers,
Paschalis

-- 
-- 
You received this message because you are subscribed to the "Android Building" 
mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to android-building+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/android-building/6e937488-80cb-42c8-b15c-087cafb7c400%40googlegroups.com.