Lars said on August 15 "In short the answer is 'likely not'." It seems we
now know he was exactly right.
On Tue, Aug 27, 2019 at 9:57 AM Koltsov Dmitriy <d.kolt...@yadro.com> wrote:
> Hello, Jim.
>
> Ok. Here is the explanation.
>
> I wrote to this mailing list in order to find out the possibility of
> successful build Impala in Ubuntu 16.04 ppc64le (if there is a support of
> ppc64le in Impala project/code). As one can see in previous messages of
> this thread, colleagues describe me a possible way in which one can try to
> successfully build github apache Impala with cloudera native-toolchain.
> There were no noticed objections from colleagues in this thread which
> obviously lead to stop try to build Impala on ppc64le.
>
> But for now, after passing some "barriers" in build process and after some
> tricks, I finally pass configure stage of ./buildall.sh Impala script
> (including toolchain check) and met the following error (see below) on
> console while Impala build process (./buildall.sh executed).
>
> So, looking at the error, naturally the question arises: is there de facto
> a possibility for now to build github apache Impala with cloudera
> native-toolchain in Ubuntu 16.04 ppc64le ? (Impala project supports Ubuntu
> 16.04 ppc64le installation?)
>
> If 'yes' - then could you, please, give some comments about the reason of
> the following error (below) while Impala+"native-toolchain" build in Ubuntu
> 16.04 ppc64le ? and about the way to resolve the error to successfully
> build Impala ?
> (May be there is some another commit of impala which is more "friendly"
> with ppc64le and native-toolchain for now)
>
> >>And I have a lot of error messages while Impala compiling process in
> Ubuntu 16.04 ppc64le, e.g.:
>
>
> >>
> >>
> >>...
> >>make[2]: *** Deleting file
> 'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
> >>In file included from
> /home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
> >> from
> /home/build/impala/be/src/gutil/spinlock_internal.cc:44:
> >>/home/build/impala/be/src/gutil/atomicops.h:88:2: error: #error You need
> to implement atomic operations for this architecture
> >> #error You need to implement atomic operations for this architecture
> >> ^
> >>/home/build/impala/be/src/gutil/atomicops.h:321:8: error: âAtomic32â
> does not name a type
> >> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr,
> >>
> >>...
> >>
> >> Impala build process failed.
>
>
>
> On 08/27/19 17:32, Jim Apple wrote:
>
> I'm a bit confused. The text of the error was "You need to implement
> atomic operations for this architecture", right? Maybe could you be a bit
> more descriptive about what your understanding of that is and why it is a
> surprising error?
>
> On Tue, Aug 27, 2019 at 2:51 AM Koltsov Dmitriy <d.kolt...@yadro.com>
> wrote:
>
>> Ok, Tim.
>>
>> Now it's clear - github.com/apache/impala and
>> github.com/cloudera/native-toolchain should be used for build in Ubuntu
>> 16.04 ppc64le.
>>
>> But then still there is the question - could you, please, give some
>> comments about possible reason of the error shown below when I tried this
>> combination of impala and native-toolchain ? ... and about possible ways to
>> solve it ? May be there is some another commit of impala which is more
>> "friendly" with ppc64le and native-toolchain for now ?
>>
>> >>And I have a lot of error messages while compiling process, e.g.:
>> >>
>> >>
>> >>...
>> >>make[2]: *** Deleting file
>> 'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
>> >>In file included from
>> /home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
>> >> from
>> /home/build/impala/be/src/gutil/spinlock_internal.cc:44:
>> >>/home/build/impala/be/src/gutil/atomicops.h:88:2: error: #error You
>> need to implement atomic operations for this architecture
>> >> #error You need to implement atomic operations for this architecture
>> >> ^
>> >>/home/build/impala/be/src/gutil/atomicops.h:321:8: error: âAtomic32â
>> does not name a type
>> >> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr,
>> >>
>> >>...
>> >>
>> >> Impala build process failed.
>>
>>
>>
>>
>> On 08/27/19 01:51, Tim Armstrong wrote:
>>
>> Hi Dmitry,
>>
>> > But only by that moment I've noticed that there is a significant
>> difference between github.com/cloudera/Impala and
>> github.com/apache/impala - while build.
>> > And all previous steps I tried with github apache impala and cloudera
>> native-toolchain.
>> That is the right thing to do. Apache Impala master is the latest and
>> greatest. github.com/cloudera/impala mainly hosts snapshots of the
>> source from different Cloudera releases. It looks like the default branch
>> was cdh5-trunk, which is a fairly old version of Impala at this point.
>>
>>
>>
>> On Mon, Aug 26, 2019 at 1:50 PM Koltsov Dmitriy <d.kolt...@yadro.com>
>> wrote:
>>
>>> Hello Tim.
>>>
>>> After some tricks I have success in building Kudu stub. Thanks for
>>> advice.
>>>
>>> But only by that moment I've noticed that there is a significant
>>> difference between github.com/cloudera/Impala and
>>> github.com/apache/impala - while build.
>>> And all previous steps I tried with github apache impala and cloudera
>>> native-toolchain.
>>>
>>> So, for now, after I successfully got Kudu stub, I've tried to build
>>> final Impala build with github apache impala and cloudera native-toolchain
>>> in Ubuntu 16.04 ppc64le. And I have a lot of error messages while compiling
>>> process, e.g.:
>>>
>>> ...
>>> make[2]: *** Deleting file
>>> 'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
>>> In file included from
>>> /home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
>>> from
>>> /home/build/impala/be/src/gutil/spinlock_internal.cc:44:
>>> /home/build/impala/be/src/gutil/atomicops.h:88:2: error: #error You need
>>> to implement atomic operations for this architecture
>>> #error You need to implement atomic operations for this architecture
>>> ^
>>> /home/build/impala/be/src/gutil/atomicops.h:321:8: error: âAtomic32â
>>> does not name a type
>>> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr,
>>>
>>> ...
>>>
>>> Impala build process failed.
>>>
>>>
>>>
>>> So, hence now I'm trying only to use cloudera github impala and cloudera
>>> native-toolchain. I use previously build cloudera native-toolchain (git
>>> cloned from github), set IMPALA_TOOLCHAIN var to the appropriate location;
>>> current commit of cloudera impala is 7dd2c81. But when ./buildall.sh is
>>> executed I have the following error:
>>>
>>> -- Added static library dependency pprof:
>>> /home/build/impala/toolchain/gperftools-2.5/lib/libprofiler.a
>>> -- ----------> Adding thirdparty library gtest. <----------
>>> -- Header files: /home/build/impala/toolchain/gtest-1.6.0/include
>>> -- Added static library dependency gtest:
>>> /home/build/impala/toolchain/gtest-1.6.0/lib/libgtest.a
>>> CMake Error at cmake_modules/FindLlvmBinaries.cmake:33 (message):
>>> Could not find llvm-config
>>> Call Stack (most recent call first):
>>> CMakeLists.txt:208 (find_package)
>>>
>>>
>>> -- Configuring incomplete, errors occurred!
>>> See also "/home/build/impala_cloudera/CMakeFiles/CMakeOutput.log".
>>> See also "/home/build/impala_cloudera/CMakeFiles/CMakeError.log".
>>> Error in /home/build/impala_cloudera/bin/make_impala.sh at line 161:
>>> cmake . ${CMAKE_ARGS[@]}
>>>
>>> Though IMPALA_TOOLCHAIN path points to directory which contains LLVM
>>> build: "llvm-5.0.1-p1" dir is in the path, with all appropriate files.
>>>
>>>
>>>
>>>
>>>
>>> *Question:* Could you, please, give a comment about possible reason of
>>> the error and about how it could be resolved ?
>>>
>>>
>>>
>>>
>>>
>>> btw: there is some inconsistency about llvm versions between one in
>>> ./bin/impala-config.sh (export IMPALA_LLVM_VERSION=3.9.1) and default one
>>> of llvm build in cloudera native-toolchain (5.0.1-p1).
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 08/24/19 03:49, Tim Armstrong wrote:
>>>
>>> Ah I see. I think if you run bin/bootstrap_toolchain.py manually it will
>>> build the stub for you.
>>>
>>> I wouldn't be surprised if this requires some tweaks to the scripts to
>>> make it work since they weren't written for this exact case in mind (the
>>> original use case for the kudu stubs was for OSes that Kudu didn't support
>>> but Impala did, like CentOS 5).
>>>
>>> On Wed, Aug 21, 2019 at 11:12 AM Koltsov Dmitriy <d.kolt...@yadro.com>
>>> wrote:
>>>
>>>> Hello, Tim.
>>>>
>>>> Thanks for reply.
>>>>
>>>> When I'm trying to build Impala I am of course using already built
>>>> clouder native-toolchain (cloned from github repo). This native-toolchain
>>>> has no
>>>>
>>>> kudu in final build directory. So, when I'm trying to build Impala
>>>> pointing this build through IMPALA_TOOLCHAIN variable with
>>>> SKIP_TOOLCHAIN_BOOSTRAP=true then I have an error mentioned by me above
>>>> (see the fragment below). Could you, please, inform if there is a
>>>> possibility to instruct Impala building process not to use Kudu in Impala
>>>> build on Ubuntu 16.04 ppc64le ?
>>>>
>>>> > > -- ----------> Adding thirdparty library libunwind. <---------->
>>>>
>>>> > > -- Header files:
>>>> /home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
>>>>
>>>> > > -- Added shared library dependency libunwind:>
>>>>
>>>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
>>>>
>>>> > > CMake Error at CMakeLists.txt:409 (find_package):>
>>>>
>>>> > > Could not find a package configuration file provided by
>>>> "kuduClient" with>
>>>>
>>>> > > any of the following names:>
>>>>
>>>> > >>
>>>>
>>>> > > kuduClientConfig.cmake>
>>>>
>>>> > > kuduclient-config.cmake>
>>>>
>>>> > >>
>>>>
>>>> > > Add the installation prefix of "kuduClient" to CMAKE_PREFIX_PATH or
>>>> set>
>>>>
>>>> > > "kuduClient_DIR" to a directory containing one of the above files.
>>>> If>
>>>>
>>>> > > "kuduClient" provides a separate development package or SDK, be
>>>> sure it>
>>>>
>>>> > > has>
>>>>
>>>> > > been installed.>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > > -- Configuring incomplete, errors occurred!>
>>>>
>>>> > > See also "/home/build/impala/CMakeFiles/CMakeOutput.log".>
>>>>
>>>> > > See also "/home/build/impala/CMakeFiles/CMakeError.log".>
>>>>
>>>> > > /home/build/impala#>
>>>>
>>>> > > /home/build/impala#>
>>>>
>>>> On 2019/08/21 16:47:48, Tim Armstrong <t...@cloudera.com>
>>>> <t...@cloudera.com> wrote:
>>>>
>>>> > Hi Dmitry,>
>>>>
>>>> >
>>>>
>>>> > The toolchain bootstrap process downloads pre-built binaries, and we
>>>> don't>
>>>>
>>>> > have any for PPC.>
>>>>
>>>> >
>>>>
>>>> > You would need to build the toolchain locally for PPC, then point
>>>> the>
>>>>
>>>> > Impala build at it. These instructions might be helpful for you ->
>>>>
>>>> >
>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala>
>>>>
>>>>
>>>> > .>
>>>>
>>>> >
>>>>
>>>> >
>>>>
>>>> >
>>>>
>>>> > On Tue, Aug 20, 2019 at 3:53 PM Koltsov Dmitriy <d....@yadro.com>
>>>> <d....@yadro.com> wrote:>
>>>>
>>>> >
>>>>
>>>> > > I've looked at the contents of impala-config.sh. As I understand -
>>>> I>
>>>>
>>>> > > should manually set KUDU_IS_SUPPORTED=false in order to make
>>>> general impala>
>>>>
>>>> > > build process to download kudu and build stub. But I use
>>>> native_toolchain>
>>>>
>>>> > > and so var SKIP_TOOLCHAIN_BOOTSTRAP=true. So it seems that there is
>>>> mutual>
>>>>
>>>> > > exclusion: if I want to build impala with github cloudera
>>>> native_toolchain>
>>>>
>>>> > > then code branch for kudu stub building is disabled>
>>>>
>>>> > > (SKIP_TOOLCHAIN_BOOTSTRAP=true), and if I set>
>>>>
>>>> > > SKIP_TOOLCHAIN_BOOTSTRAP=false to enable general build process to
>>>> download>
>>>>
>>>> > > and build kudu stub then there will be no other components
>>>> available from>
>>>>
>>>> > > cloudera native_toolchain.>
>>>>
>>>> > >>
>>>>
>>>> > > For example, when I use first case with
>>>> SKIP_TOOLCHAIN_BOOTSTRAP=true and>
>>>>
>>>> > > manually KUDU_IS_SUPPORTED=false set in impala-config.sh, then I
>>>> have the>
>>>>
>>>> > > following error when trying ./buildall.sh for cloudera impala:>
>>>>
>>>> > >>
>>>>
>>>> > > ...>
>>>>
>>>> > > -- ----------> Adding thirdparty library krb5. <---------->
>>>>
>>>> > > -- Header files: /usr/include>
>>>>
>>>> > > -- Added shared library dependency krb5:>
>>>>
>>>> > > /usr/lib/powerpc64le-linux-gnu/libkrb5.so>
>>>>
>>>> > > -- Found Kerberos: /usr/sbin/kadmin.local>
>>>>
>>>> > > -- Added shared library dependency rt:>
>>>>
>>>> > > /usr/lib/powerpc64le-linux-gnu/librt.so>
>>>>
>>>> > > -- Added shared library dependency dl:>
>>>>
>>>> > > /usr/lib/powerpc64le-linux-gnu/libdl.so>
>>>>
>>>> > > -- Found LIBUNWIND:>
>>>>
>>>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
>>>>
>>>> > > -- ----------> Adding thirdparty library libunwind. <---------->
>>>>
>>>> > > -- Header files:
>>>> /home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
>>>>
>>>> > > -- Added shared library dependency libunwind:>
>>>>
>>>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
>>>>
>>>> > > CMake Error at CMakeLists.txt:409 (find_package):>
>>>>
>>>> > > Could not find a package configuration file provided by
>>>> "kuduClient" with>
>>>>
>>>> > > any of the following names:>
>>>>
>>>> > >>
>>>>
>>>> > > kuduClientConfig.cmake>
>>>>
>>>> > > kuduclient-config.cmake>
>>>>
>>>> > >>
>>>>
>>>> > > Add the installation prefix of "kuduClient" to CMAKE_PREFIX_PATH or
>>>> set>
>>>>
>>>> > > "kuduClient_DIR" to a directory containing one of the above files.
>>>> If>
>>>>
>>>> > > "kuduClient" provides a separate development package or SDK, be
>>>> sure it>
>>>>
>>>> > > has>
>>>>
>>>> > > been installed.>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > > -- Configuring incomplete, errors occurred!>
>>>>
>>>> > > See also "/home/build/impala/CMakeFiles/CMakeOutput.log".>
>>>>
>>>> > > See also "/home/build/impala/CMakeFiles/CMakeError.log".>
>>>>
>>>> > > /home/build/impala#>
>>>>
>>>> > > /home/build/impala#>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > > Could you please describe how can one exactly build cloudera impala
>>>> with>
>>>>
>>>> > > its native_toolchain and kudu stub simultaneously on Ubuntu 16.04
>>>> ppc64le ?>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > > Regards,>
>>>>
>>>> > > Koltsov Dmitriy>
>>>>
>>>> > > System Engineer, YADRO>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > > On 08/16/19 18:15, Lars Volker wrote:>
>>>>
>>>> > >>
>>>>
>>>> > > For the Impala toolchain you can start with this list of commits>
>>>>
>>>> > > <https://gerrit.cloudera.org/#/q/ppc64le>
>>>> <https://gerrit.cloudera.org/#/q/ppc64le>, in particular this review>
>>>>
>>>> > > <https://gerrit.cloudera.org/#/c/9077/>
>>>> <https://gerrit.cloudera.org/#/c/9077/>.>
>>>>
>>>> > >>
>>>>
>>>> > > I don't know about kudu, you might want to ask on their mailing
>>>> list or>
>>>>
>>>> > > their slack channel. If you look in impala-config.sh for>
>>>>
>>>> > > the KUDU_IS_SUPPORTED flag, you should find a way to build Impala
>>>> without>
>>>>
>>>> > > Kudu.>
>>>>
>>>> > >>
>>>>
>>>> > > Cheers, Lars>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > >>
>>>>
>>>> > > On Fri, Aug 16, 2019 at 5:32 AM Koltsov Dmitriy <d....@yadro.com>
>>>> <d....@yadro.com>>
>>>>
>>>> > > wrote:>
>>>>
>>>> > >>
>>>>
>>>> > >> Hello, Lars!>
>>>>
>>>> > >>>
>>>>
>>>> > >> Thanks for operative answer.>
>>>>
>>>> > >>>
>>>>
>>>> > >> And could please inform about actual commit number of the>
>>>>
>>>> > >> native-toolchain on github which can be build successfully at that
>>>> moment ?>
>>>>
>>>> > >>>
>>>>
>>>> > >> Btw: I ask about it because build process for current commit of>
>>>>
>>>> > >> native-toolchain finishes with error. And error is because of kudu
>>>> build>
>>>>
>>>> > >> fail (other components of toolchain have been built fine). For now
>>>> build.sh>
>>>>
>>>> > >> file in this commit of kudu src have explicit switch to disable
>>>> ppc64le.>
>>>>
>>>> > >> And if I manually enable ppc64le case then code will try to build
>>>> kudu from>
>>>>
>>>> > >> github; but github version kudu tries to build own toolchain
>>>> while>
>>>>
>>>> > >> thirdparty building and it seems that for know current commit of
>>>> github>
>>>>
>>>> > >> kudu has an error on the way to build kudu on Ubuntu 16.04
>>>> ppc64le.>
>>>>
>>>> > >>>
>>>>
>>>> > >> So another question is - Can Kudu be build on Ubuntu 16.04 ppc64le
>>>> ? If>
>>>>
>>>> > >> yes - could you please inform me about appropriate commit of
>>>> github kudu or>
>>>>
>>>> > >> maybe about another way to build it ?>
>>>>
>>>> > >>>
>>>>
>>>> > >>>
>>>>
>>>> > >>>
>>>>
>>>> > >> Regards,>
>>>>
>>>> > >> Koltsov Dmitriy>
>>>>
>>>> > >> System Engineer, YADRO>
>>>>
>>>> > >>>
>>>>
>>>> > >>>
>>>>
>>>> > >> On 16.08.2019 00:39, Lars Volker wrote:>
>>>>
>>>> > >>>
>>>>
>>>> > >> In short the answer is "likely not". We don't have any tests for
>>>> ppc64le.>
>>>>
>>>> > >> About two years ago we had some commits to enable the toolchain on
>>>> power,>
>>>>
>>>> > >> but I can't recall or find any changes to Impala itself related to
>>>> that>
>>>>
>>>> > >> effort.>
>>>>
>>>> > >>>
>>>>
>>>> > >> Cheers, Lars>
>>>>
>>>> > >>>
>>>>
>>>> > >> On Thu, Aug 15, 2019 at 11:53 AM Koltsov Dmitriy <d....@yadro.com>
>>>> <d....@yadro.com>>
>>>>
>>>> > >> wrote:>
>>>>
>>>> > >>>
>>>>
>>>> > >>> Hello!>
>>>>
>>>> > >>>>
>>>>
>>>> > >>> Could you please inform if Impala can be build and run on Ubuntu
>>>> 16.04>
>>>>
>>>> > >>> ppc64le ?>
>>>>
>>>> > >>>>
>>>>
>>>> > >>>>
>>>>
>>>> > >>>>
>>>>
>>>> > >>>>
>>>>
>>>> > >>> -->
>>>>
>>>> > >>> Regards,>
>>>>
>>>> > >>> Koltsov Dmitriy>
>>>>
>>>> > >>> System Engineer, YADRO.>
>>>>
>>>> > >>>>
>>>>
>>>> > >>>>
>>>>
>>>> > >>>
>>>>
>>>> > >>
>>>>
>>>> >
>>>>
>>>
>>>
>>
>
