Hi Dmitry,
> But only by that moment I've noticed that there is a significant
difference between github.com/cloudera/Impala and github.com/apache/impala
- while build.
> And all previous steps I tried with github apache impala and cloudera
native-toolchain.
That is the right thing to do. Apache Impala master is the latest and
greatest. github.com/cloudera/impala mainly hosts snapshots of the source
from different Cloudera releases. It looks like the default branch was
cdh5-trunk, which is a fairly old version of Impala at this point.
On Mon, Aug 26, 2019 at 1:50 PM Koltsov Dmitriy <d.kolt...@yadro.com> wrote:
> Hello Tim.
>
> After some tricks I have success in building Kudu stub. Thanks for advice.
>
> But only by that moment I've noticed that there is a significant
> difference between github.com/cloudera/Impala and github.com/apache/impala
> - while build.
> And all previous steps I tried with github apache impala and cloudera
> native-toolchain.
>
> So, for now, after I successfully got Kudu stub, I've tried to build final
> Impala build with github apache impala and cloudera native-toolchain in
> Ubuntu 16.04 ppc64le. And I have a lot of error messages while compiling
> process, e.g.:
>
> ...
> make[2]: *** Deleting file
> 'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
> In file included from
> /home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
> from
> /home/build/impala/be/src/gutil/spinlock_internal.cc:44:
> /home/build/impala/be/src/gutil/atomicops.h:88:2: error: #error You need
> to implement atomic operations for this architecture
> #error You need to implement atomic operations for this architecture
> ^
> /home/build/impala/be/src/gutil/atomicops.h:321:8: error: âAtomic32â does
> not name a type
> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr,
>
> ...
>
> Impala build process failed.
>
>
>
> So, hence now I'm trying only to use cloudera github impala and cloudera
> native-toolchain. I use previously build cloudera native-toolchain (git
> cloned from github), set IMPALA_TOOLCHAIN var to the appropriate location;
> current commit of cloudera impala is 7dd2c81. But when ./buildall.sh is
> executed I have the following error:
>
> -- Added static library dependency pprof:
> /home/build/impala/toolchain/gperftools-2.5/lib/libprofiler.a
> -- ----------> Adding thirdparty library gtest. <----------
> -- Header files: /home/build/impala/toolchain/gtest-1.6.0/include
> -- Added static library dependency gtest:
> /home/build/impala/toolchain/gtest-1.6.0/lib/libgtest.a
> CMake Error at cmake_modules/FindLlvmBinaries.cmake:33 (message):
> Could not find llvm-config
> Call Stack (most recent call first):
> CMakeLists.txt:208 (find_package)
>
>
> -- Configuring incomplete, errors occurred!
> See also "/home/build/impala_cloudera/CMakeFiles/CMakeOutput.log".
> See also "/home/build/impala_cloudera/CMakeFiles/CMakeError.log".
> Error in /home/build/impala_cloudera/bin/make_impala.sh at line 161: cmake
> . ${CMAKE_ARGS[@]}
>
> Though IMPALA_TOOLCHAIN path points to directory which contains LLVM
> build: "llvm-5.0.1-p1" dir is in the path, with all appropriate files.
>
>
>
>
>
> *Question:* Could you, please, give a comment about possible reason of
> the error and about how it could be resolved ?
>
>
>
>
>
> btw: there is some inconsistency about llvm versions between one in
> ./bin/impala-config.sh (export IMPALA_LLVM_VERSION=3.9.1) and default one
> of llvm build in cloudera native-toolchain (5.0.1-p1).
>
>
>
>
>
>
> On 08/24/19 03:49, Tim Armstrong wrote:
>
> Ah I see. I think if you run bin/bootstrap_toolchain.py manually it will
> build the stub for you.
>
> I wouldn't be surprised if this requires some tweaks to the scripts to
> make it work since they weren't written for this exact case in mind (the
> original use case for the kudu stubs was for OSes that Kudu didn't support
> but Impala did, like CentOS 5).
>
> On Wed, Aug 21, 2019 at 11:12 AM Koltsov Dmitriy <d.kolt...@yadro.com>
> wrote:
>
>> Hello, Tim.
>>
>> Thanks for reply.
>>
>> When I'm trying to build Impala I am of course using already built
>> clouder native-toolchain (cloned from github repo). This native-toolchain
>> has no
>>
>> kudu in final build directory. So, when I'm trying to build Impala
>> pointing this build through IMPALA_TOOLCHAIN variable with
>> SKIP_TOOLCHAIN_BOOSTRAP=true then I have an error mentioned by me above
>> (see the fragment below). Could you, please, inform if there is a
>> possibility to instruct Impala building process not to use Kudu in Impala
>> build on Ubuntu 16.04 ppc64le ?
>>
>> > > -- ----------> Adding thirdparty library libunwind. <---------->
>>
>> > > -- Header files:
>> /home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
>>
>> > > -- Added shared library dependency libunwind:>
>>
>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
>>
>> > > CMake Error at CMakeLists.txt:409 (find_package):>
>>
>> > > Could not find a package configuration file provided by "kuduClient"
>> with>
>>
>> > > any of the following names:>
>>
>> > >>
>>
>> > > kuduClientConfig.cmake>
>>
>> > > kuduclient-config.cmake>
>>
>> > >>
>>
>> > > Add the installation prefix of "kuduClient" to CMAKE_PREFIX_PATH or
>> set>
>>
>> > > "kuduClient_DIR" to a directory containing one of the above files.
>> If>
>>
>> > > "kuduClient" provides a separate development package or SDK, be sure
>> it>
>>
>> > > has>
>>
>> > > been installed.>
>>
>> > >>
>>
>> > >>
>>
>> > > -- Configuring incomplete, errors occurred!>
>>
>> > > See also "/home/build/impala/CMakeFiles/CMakeOutput.log".>
>>
>> > > See also "/home/build/impala/CMakeFiles/CMakeError.log".>
>>
>> > > /home/build/impala#>
>>
>> > > /home/build/impala#>
>>
>> On 2019/08/21 16:47:48, Tim Armstrong <t...@cloudera.com>
>> <t...@cloudera.com> wrote:
>>
>> > Hi Dmitry,>
>>
>> >
>>
>> > The toolchain bootstrap process downloads pre-built binaries, and we
>> don't>
>>
>> > have any for PPC.>
>>
>> >
>>
>> > You would need to build the toolchain locally for PPC, then point the>
>>
>> > Impala build at it. These instructions might be helpful for you ->
>>
>> >
>> https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala>
>>
>>
>> > .>
>>
>> >
>>
>> >
>>
>> >
>>
>> > On Tue, Aug 20, 2019 at 3:53 PM Koltsov Dmitriy <d....@yadro.com>
>> <d....@yadro.com> wrote:>
>>
>> >
>>
>> > > I've looked at the contents of impala-config.sh. As I understand - I>
>>
>> > > should manually set KUDU_IS_SUPPORTED=false in order to make general
>> impala>
>>
>> > > build process to download kudu and build stub. But I use
>> native_toolchain>
>>
>> > > and so var SKIP_TOOLCHAIN_BOOTSTRAP=true. So it seems that there is
>> mutual>
>>
>> > > exclusion: if I want to build impala with github cloudera
>> native_toolchain>
>>
>> > > then code branch for kudu stub building is disabled>
>>
>> > > (SKIP_TOOLCHAIN_BOOTSTRAP=true), and if I set>
>>
>> > > SKIP_TOOLCHAIN_BOOTSTRAP=false to enable general build process to
>> download>
>>
>> > > and build kudu stub then there will be no other components available
>> from>
>>
>> > > cloudera native_toolchain.>
>>
>> > >>
>>
>> > > For example, when I use first case with SKIP_TOOLCHAIN_BOOTSTRAP=true
>> and>
>>
>> > > manually KUDU_IS_SUPPORTED=false set in impala-config.sh, then I have
>> the>
>>
>> > > following error when trying ./buildall.sh for cloudera impala:>
>>
>> > >>
>>
>> > > ...>
>>
>> > > -- ----------> Adding thirdparty library krb5. <---------->
>>
>> > > -- Header files: /usr/include>
>>
>> > > -- Added shared library dependency krb5:>
>>
>> > > /usr/lib/powerpc64le-linux-gnu/libkrb5.so>
>>
>> > > -- Found Kerberos: /usr/sbin/kadmin.local>
>>
>> > > -- Added shared library dependency rt:>
>>
>> > > /usr/lib/powerpc64le-linux-gnu/librt.so>
>>
>> > > -- Added shared library dependency dl:>
>>
>> > > /usr/lib/powerpc64le-linux-gnu/libdl.so>
>>
>> > > -- Found LIBUNWIND:>
>>
>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
>>
>> > > -- ----------> Adding thirdparty library libunwind. <---------->
>>
>> > > -- Header files:
>> /home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
>>
>> > > -- Added shared library dependency libunwind:>
>>
>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
>>
>> > > CMake Error at CMakeLists.txt:409 (find_package):>
>>
>> > > Could not find a package configuration file provided by "kuduClient"
>> with>
>>
>> > > any of the following names:>
>>
>> > >>
>>
>> > > kuduClientConfig.cmake>
>>
>> > > kuduclient-config.cmake>
>>
>> > >>
>>
>> > > Add the installation prefix of "kuduClient" to CMAKE_PREFIX_PATH or
>> set>
>>
>> > > "kuduClient_DIR" to a directory containing one of the above files.
>> If>
>>
>> > > "kuduClient" provides a separate development package or SDK, be sure
>> it>
>>
>> > > has>
>>
>> > > been installed.>
>>
>> > >>
>>
>> > >>
>>
>> > > -- Configuring incomplete, errors occurred!>
>>
>> > > See also "/home/build/impala/CMakeFiles/CMakeOutput.log".>
>>
>> > > See also "/home/build/impala/CMakeFiles/CMakeError.log".>
>>
>> > > /home/build/impala#>
>>
>> > > /home/build/impala#>
>>
>> > >>
>>
>> > >>
>>
>> > > Could you please describe how can one exactly build cloudera impala
>> with>
>>
>> > > its native_toolchain and kudu stub simultaneously on Ubuntu 16.04
>> ppc64le ?>
>>
>> > >>
>>
>> > >>
>>
>> > >>
>>
>> > >>
>>
>> > >>
>>
>> > > Regards,>
>>
>> > > Koltsov Dmitriy>
>>
>> > > System Engineer, YADRO>
>>
>> > >>
>>
>> > >>
>>
>> > >>
>>
>> > > On 08/16/19 18:15, Lars Volker wrote:>
>>
>> > >>
>>
>> > > For the Impala toolchain you can start with this list of commits>
>>
>> > > <https://gerrit.cloudera.org/#/q/ppc64le>
>> <https://gerrit.cloudera.org/#/q/ppc64le>, in particular this review>
>>
>> > > <https://gerrit.cloudera.org/#/c/9077/>
>> <https://gerrit.cloudera.org/#/c/9077/>.>
>>
>> > >>
>>
>> > > I don't know about kudu, you might want to ask on their mailing list
>> or>
>>
>> > > their slack channel. If you look in impala-config.sh for>
>>
>> > > the KUDU_IS_SUPPORTED flag, you should find a way to build Impala
>> without>
>>
>> > > Kudu.>
>>
>> > >>
>>
>> > > Cheers, Lars>
>>
>> > >>
>>
>> > >>
>>
>> > >>
>>
>> > > On Fri, Aug 16, 2019 at 5:32 AM Koltsov Dmitriy <d....@yadro.com>
>> <d....@yadro.com>>
>>
>> > > wrote:>
>>
>> > >>
>>
>> > >> Hello, Lars!>
>>
>> > >>>
>>
>> > >> Thanks for operative answer.>
>>
>> > >>>
>>
>> > >> And could please inform about actual commit number of the>
>>
>> > >> native-toolchain on github which can be build successfully at that
>> moment ?>
>>
>> > >>>
>>
>> > >> Btw: I ask about it because build process for current commit of>
>>
>> > >> native-toolchain finishes with error. And error is because of kudu
>> build>
>>
>> > >> fail (other components of toolchain have been built fine). For now
>> build.sh>
>>
>> > >> file in this commit of kudu src have explicit switch to disable
>> ppc64le.>
>>
>> > >> And if I manually enable ppc64le case then code will try to build
>> kudu from>
>>
>> > >> github; but github version kudu tries to build own toolchain while>
>>
>> > >> thirdparty building and it seems that for know current commit of
>> github>
>>
>> > >> kudu has an error on the way to build kudu on Ubuntu 16.04 ppc64le.>
>>
>> > >>>
>>
>> > >> So another question is - Can Kudu be build on Ubuntu 16.04 ppc64le ?
>> If>
>>
>> > >> yes - could you please inform me about appropriate commit of github
>> kudu or>
>>
>> > >> maybe about another way to build it ?>
>>
>> > >>>
>>
>> > >>>
>>
>> > >>>
>>
>> > >> Regards,>
>>
>> > >> Koltsov Dmitriy>
>>
>> > >> System Engineer, YADRO>
>>
>> > >>>
>>
>> > >>>
>>
>> > >> On 16.08.2019 00:39, Lars Volker wrote:>
>>
>> > >>>
>>
>> > >> In short the answer is "likely not". We don't have any tests for
>> ppc64le.>
>>
>> > >> About two years ago we had some commits to enable the toolchain on
>> power,>
>>
>> > >> but I can't recall or find any changes to Impala itself related to
>> that>
>>
>> > >> effort.>
>>
>> > >>>
>>
>> > >> Cheers, Lars>
>>
>> > >>>
>>
>> > >> On Thu, Aug 15, 2019 at 11:53 AM Koltsov Dmitriy <d....@yadro.com>
>> <d....@yadro.com>>
>>
>> > >> wrote:>
>>
>> > >>>
>>
>> > >>> Hello!>
>>
>> > >>>>
>>
>> > >>> Could you please inform if Impala can be build and run on Ubuntu
>> 16.04>
>>
>> > >>> ppc64le ?>
>>
>> > >>>>
>>
>> > >>>>
>>
>> > >>>>
>>
>> > >>>>
>>
>> > >>> -->
>>
>> > >>> Regards,>
>>
>> > >>> Koltsov Dmitriy>
>>
>> > >>> System Engineer, YADRO.>
>>
>> > >>>>
>>
>> > >>>>
>>
>> > >>>
>>
>> > >>
>>
>> >
>>
>
>
