Right, it's not gonna work without some code changes and so on. I think the
previous PPCLE effort disabled the Kudu client entirely to avoid that work.
On Tue, Aug 27, 2019 at 4:41 PM Jim Apple <jbap...@apache.org> wrote:
> Lars said on August 15 "In short the answer is 'likely not'." It seems we
> now know he was exactly right.
>
> On Tue, Aug 27, 2019 at 9:57 AM Koltsov Dmitriy <d.kolt...@yadro.com>
> wrote:
>
>> Hello, Jim.
>>
>> Ok. Here is the explanation.
>>
>> I wrote to this mailing list in order to find out the possibility of
>> successful build Impala in Ubuntu 16.04 ppc64le (if there is a support of
>> ppc64le in Impala project/code). As one can see in previous messages of
>> this thread, colleagues describe me a possible way in which one can try to
>> successfully build github apache Impala with cloudera native-toolchain.
>> There were no noticed objections from colleagues in this thread which
>> obviously lead to stop try to build Impala on ppc64le.
>>
>> But for now, after passing some "barriers" in build process and after
>> some tricks, I finally pass configure stage of ./buildall.sh Impala script
>> (including toolchain check) and met the following error (see below) on
>> console while Impala build process (./buildall.sh executed).
>>
>> So, looking at the error, naturally the question arises: is there de
>> facto a possibility for now to build github apache Impala with cloudera
>> native-toolchain in Ubuntu 16.04 ppc64le ? (Impala project supports Ubuntu
>> 16.04 ppc64le installation?)
>>
>> If 'yes' - then could you, please, give some comments about the reason of
>> the following error (below) while Impala+"native-toolchain" build in Ubuntu
>> 16.04 ppc64le ? and about the way to resolve the error to successfully
>> build Impala ?
>> (May be there is some another commit of impala which is more "friendly"
>> with ppc64le and native-toolchain for now)
>>
>> >>And I have a lot of error messages while Impala compiling process in
>> Ubuntu 16.04 ppc64le, e.g.:
>>
>>
>> >>
>> >>
>> >>...
>> >>make[2]: *** Deleting file
>> 'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
>> >>In file included from
>> /home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
>> >> from
>> /home/build/impala/be/src/gutil/spinlock_internal.cc:44:
>> >>/home/build/impala/be/src/gutil/atomicops.h:88:2: error: #error You
>> need to implement atomic operations for this architecture
>> >> #error You need to implement atomic operations for this architecture
>> >> ^
>> >>/home/build/impala/be/src/gutil/atomicops.h:321:8: error: âAtomic32â
>> does not name a type
>> >> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr,
>> >>
>> >>...
>> >>
>> >> Impala build process failed.
>>
>>
>>
>> On 08/27/19 17:32, Jim Apple wrote:
>>
>> I'm a bit confused. The text of the error was "You need to implement
>> atomic operations for this architecture", right? Maybe could you be a bit
>> more descriptive about what your understanding of that is and why it is a
>> surprising error?
>>
>> On Tue, Aug 27, 2019 at 2:51 AM Koltsov Dmitriy <d.kolt...@yadro.com>
>> wrote:
>>
>>> Ok, Tim.
>>>
>>> Now it's clear - github.com/apache/impala and
>>> github.com/cloudera/native-toolchain should be used for build in Ubuntu
>>> 16.04 ppc64le.
>>>
>>> But then still there is the question - could you, please, give some
>>> comments about possible reason of the error shown below when I tried this
>>> combination of impala and native-toolchain ? ... and about possible ways to
>>> solve it ? May be there is some another commit of impala which is more
>>> "friendly" with ppc64le and native-toolchain for now ?
>>>
>>> >>And I have a lot of error messages while compiling process, e.g.:
>>> >>
>>> >>
>>> >>...
>>> >>make[2]: *** Deleting file
>>> 'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
>>> >>In file included from
>>> /home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
>>> >> from
>>> /home/build/impala/be/src/gutil/spinlock_internal.cc:44:
>>> >>/home/build/impala/be/src/gutil/atomicops.h:88:2: error: #error You
>>> need to implement atomic operations for this architecture
>>> >> #error You need to implement atomic operations for this architecture
>>> >> ^
>>> >>/home/build/impala/be/src/gutil/atomicops.h:321:8: error: âAtomic32â
>>> does not name a type
>>> >> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr,
>>> >>
>>> >>...
>>> >>
>>> >> Impala build process failed.
>>>
>>>
>>>
>>>
>>> On 08/27/19 01:51, Tim Armstrong wrote:
>>>
>>> Hi Dmitry,
>>>
>>> > But only by that moment I've noticed that there is a significant
>>> difference between github.com/cloudera/Impala and
>>> github.com/apache/impala - while build.
>>> > And all previous steps I tried with github apache impala and cloudera
>>> native-toolchain.
>>> That is the right thing to do. Apache Impala master is the latest and
>>> greatest. github.com/cloudera/impala mainly hosts snapshots of the
>>> source from different Cloudera releases. It looks like the default branch
>>> was cdh5-trunk, which is a fairly old version of Impala at this point.
>>>
>>>
>>>
>>> On Mon, Aug 26, 2019 at 1:50 PM Koltsov Dmitriy <d.kolt...@yadro.com>
>>> wrote:
>>>
>>>> Hello Tim.
>>>>
>>>> After some tricks I have success in building Kudu stub. Thanks for
>>>> advice.
>>>>
>>>> But only by that moment I've noticed that there is a significant
>>>> difference between github.com/cloudera/Impala and
>>>> github.com/apache/impala - while build.
>>>> And all previous steps I tried with github apache impala and cloudera
>>>> native-toolchain.
>>>>
>>>> So, for now, after I successfully got Kudu stub, I've tried to build
>>>> final Impala build with github apache impala and cloudera native-toolchain
>>>> in Ubuntu 16.04 ppc64le. And I have a lot of error messages while compiling
>>>> process, e.g.:
>>>>
>>>> ...
>>>> make[2]: *** Deleting file
>>>> 'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
>>>> In file included from
>>>> /home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
>>>> from
>>>> /home/build/impala/be/src/gutil/spinlock_internal.cc:44:
>>>> /home/build/impala/be/src/gutil/atomicops.h:88:2: error: #error You
>>>> need to implement atomic operations for this architecture
>>>> #error You need to implement atomic operations for this architecture
>>>> ^
>>>> /home/build/impala/be/src/gutil/atomicops.h:321:8: error: âAtomic32â
>>>> does not name a type
>>>> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr,
>>>>
>>>> ...
>>>>
>>>> Impala build process failed.
>>>>
>>>>
>>>>
>>>> So, hence now I'm trying only to use cloudera github impala and
>>>> cloudera native-toolchain. I use previously build cloudera native-toolchain
>>>> (git cloned from github), set IMPALA_TOOLCHAIN var to the appropriate
>>>> location; current commit of cloudera impala is 7dd2c81. But when
>>>> ./buildall.sh is executed I have the following error:
>>>>
>>>> -- Added static library dependency pprof:
>>>> /home/build/impala/toolchain/gperftools-2.5/lib/libprofiler.a
>>>> -- ----------> Adding thirdparty library gtest. <----------
>>>> -- Header files: /home/build/impala/toolchain/gtest-1.6.0/include
>>>> -- Added static library dependency gtest:
>>>> /home/build/impala/toolchain/gtest-1.6.0/lib/libgtest.a
>>>> CMake Error at cmake_modules/FindLlvmBinaries.cmake:33 (message):
>>>> Could not find llvm-config
>>>> Call Stack (most recent call first):
>>>> CMakeLists.txt:208 (find_package)
>>>>
>>>>
>>>> -- Configuring incomplete, errors occurred!
>>>> See also "/home/build/impala_cloudera/CMakeFiles/CMakeOutput.log".
>>>> See also "/home/build/impala_cloudera/CMakeFiles/CMakeError.log".
>>>> Error in /home/build/impala_cloudera/bin/make_impala.sh at line 161:
>>>> cmake . ${CMAKE_ARGS[@]}
>>>>
>>>> Though IMPALA_TOOLCHAIN path points to directory which contains LLVM
>>>> build: "llvm-5.0.1-p1" dir is in the path, with all appropriate files.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Question:* Could you, please, give a comment about possible reason of
>>>> the error and about how it could be resolved ?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> btw: there is some inconsistency about llvm versions between one in
>>>> ./bin/impala-config.sh (export IMPALA_LLVM_VERSION=3.9.1) and default one
>>>> of llvm build in cloudera native-toolchain (5.0.1-p1).
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 08/24/19 03:49, Tim Armstrong wrote:
>>>>
>>>> Ah I see. I think if you run bin/bootstrap_toolchain.py manually it
>>>> will build the stub for you.
>>>>
>>>> I wouldn't be surprised if this requires some tweaks to the scripts to
>>>> make it work since they weren't written for this exact case in mind (the
>>>> original use case for the kudu stubs was for OSes that Kudu didn't support
>>>> but Impala did, like CentOS 5).
>>>>
>>>> On Wed, Aug 21, 2019 at 11:12 AM Koltsov Dmitriy <d.kolt...@yadro.com>
>>>> wrote:
>>>>
>>>>> Hello, Tim.
>>>>>
>>>>> Thanks for reply.
>>>>>
>>>>> When I'm trying to build Impala I am of course using already built
>>>>> clouder native-toolchain (cloned from github repo). This native-toolchain
>>>>> has no
>>>>>
>>>>> kudu in final build directory. So, when I'm trying to build Impala
>>>>> pointing this build through IMPALA_TOOLCHAIN variable with
>>>>> SKIP_TOOLCHAIN_BOOSTRAP=true then I have an error mentioned by me above
>>>>> (see the fragment below). Could you, please, inform if there is a
>>>>> possibility to instruct Impala building process not to use Kudu in Impala
>>>>> build on Ubuntu 16.04 ppc64le ?
>>>>>
>>>>> > > -- ----------> Adding thirdparty library libunwind. <---------->
>>>>>
>>>>> > > -- Header files:
>>>>> /home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
>>>>>
>>>>> > > -- Added shared library dependency libunwind:>
>>>>>
>>>>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
>>>>>
>>>>> > > CMake Error at CMakeLists.txt:409 (find_package):>
>>>>>
>>>>> > > Could not find a package configuration file provided by
>>>>> "kuduClient" with>
>>>>>
>>>>> > > any of the following names:>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > kuduClientConfig.cmake>
>>>>>
>>>>> > > kuduclient-config.cmake>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > Add the installation prefix of "kuduClient" to CMAKE_PREFIX_PATH
>>>>> or set>
>>>>>
>>>>> > > "kuduClient_DIR" to a directory containing one of the above files.
>>>>> If>
>>>>>
>>>>> > > "kuduClient" provides a separate development package or SDK, be
>>>>> sure it>
>>>>>
>>>>> > > has>
>>>>>
>>>>> > > been installed.>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > -- Configuring incomplete, errors occurred!>
>>>>>
>>>>> > > See also "/home/build/impala/CMakeFiles/CMakeOutput.log".>
>>>>>
>>>>> > > See also "/home/build/impala/CMakeFiles/CMakeError.log".>
>>>>>
>>>>> > > /home/build/impala#>
>>>>>
>>>>> > > /home/build/impala#>
>>>>>
>>>>> On 2019/08/21 16:47:48, Tim Armstrong <t...@cloudera.com>
>>>>> <t...@cloudera.com> wrote:
>>>>>
>>>>> > Hi Dmitry,>
>>>>>
>>>>> >
>>>>>
>>>>> > The toolchain bootstrap process downloads pre-built binaries, and we
>>>>> don't>
>>>>>
>>>>> > have any for PPC.>
>>>>>
>>>>> >
>>>>>
>>>>> > You would need to build the toolchain locally for PPC, then point
>>>>> the>
>>>>>
>>>>> > Impala build at it. These instructions might be helpful for you ->
>>>>>
>>>>> >
>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala>
>>>>>
>>>>>
>>>>> > .>
>>>>>
>>>>> >
>>>>>
>>>>> >
>>>>>
>>>>> >
>>>>>
>>>>> > On Tue, Aug 20, 2019 at 3:53 PM Koltsov Dmitriy <d....@yadro.com>
>>>>> <d....@yadro.com> wrote:>
>>>>>
>>>>> >
>>>>>
>>>>> > > I've looked at the contents of impala-config.sh. As I understand -
>>>>> I>
>>>>>
>>>>> > > should manually set KUDU_IS_SUPPORTED=false in order to make
>>>>> general impala>
>>>>>
>>>>> > > build process to download kudu and build stub. But I use
>>>>> native_toolchain>
>>>>>
>>>>> > > and so var SKIP_TOOLCHAIN_BOOTSTRAP=true. So it seems that there
>>>>> is mutual>
>>>>>
>>>>> > > exclusion: if I want to build impala with github cloudera
>>>>> native_toolchain>
>>>>>
>>>>> > > then code branch for kudu stub building is disabled>
>>>>>
>>>>> > > (SKIP_TOOLCHAIN_BOOTSTRAP=true), and if I set>
>>>>>
>>>>> > > SKIP_TOOLCHAIN_BOOTSTRAP=false to enable general build process to
>>>>> download>
>>>>>
>>>>> > > and build kudu stub then there will be no other components
>>>>> available from>
>>>>>
>>>>> > > cloudera native_toolchain.>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > For example, when I use first case with
>>>>> SKIP_TOOLCHAIN_BOOTSTRAP=true and>
>>>>>
>>>>> > > manually KUDU_IS_SUPPORTED=false set in impala-config.sh, then I
>>>>> have the>
>>>>>
>>>>> > > following error when trying ./buildall.sh for cloudera impala:>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > ...>
>>>>>
>>>>> > > -- ----------> Adding thirdparty library krb5. <---------->
>>>>>
>>>>> > > -- Header files: /usr/include>
>>>>>
>>>>> > > -- Added shared library dependency krb5:>
>>>>>
>>>>> > > /usr/lib/powerpc64le-linux-gnu/libkrb5.so>
>>>>>
>>>>> > > -- Found Kerberos: /usr/sbin/kadmin.local>
>>>>>
>>>>> > > -- Added shared library dependency rt:>
>>>>>
>>>>> > > /usr/lib/powerpc64le-linux-gnu/librt.so>
>>>>>
>>>>> > > -- Added shared library dependency dl:>
>>>>>
>>>>> > > /usr/lib/powerpc64le-linux-gnu/libdl.so>
>>>>>
>>>>> > > -- Found LIBUNWIND:>
>>>>>
>>>>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
>>>>>
>>>>> > > -- ----------> Adding thirdparty library libunwind. <---------->
>>>>>
>>>>> > > -- Header files:
>>>>> /home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
>>>>>
>>>>> > > -- Added shared library dependency libunwind:>
>>>>>
>>>>> > > /home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
>>>>>
>>>>> > > CMake Error at CMakeLists.txt:409 (find_package):>
>>>>>
>>>>> > > Could not find a package configuration file provided by
>>>>> "kuduClient" with>
>>>>>
>>>>> > > any of the following names:>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > kuduClientConfig.cmake>
>>>>>
>>>>> > > kuduclient-config.cmake>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > Add the installation prefix of "kuduClient" to CMAKE_PREFIX_PATH
>>>>> or set>
>>>>>
>>>>> > > "kuduClient_DIR" to a directory containing one of the above files.
>>>>> If>
>>>>>
>>>>> > > "kuduClient" provides a separate development package or SDK, be
>>>>> sure it>
>>>>>
>>>>> > > has>
>>>>>
>>>>> > > been installed.>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > -- Configuring incomplete, errors occurred!>
>>>>>
>>>>> > > See also "/home/build/impala/CMakeFiles/CMakeOutput.log".>
>>>>>
>>>>> > > See also "/home/build/impala/CMakeFiles/CMakeError.log".>
>>>>>
>>>>> > > /home/build/impala#>
>>>>>
>>>>> > > /home/build/impala#>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > Could you please describe how can one exactly build cloudera
>>>>> impala with>
>>>>>
>>>>> > > its native_toolchain and kudu stub simultaneously on Ubuntu 16.04
>>>>> ppc64le ?>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > Regards,>
>>>>>
>>>>> > > Koltsov Dmitriy>
>>>>>
>>>>> > > System Engineer, YADRO>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > On 08/16/19 18:15, Lars Volker wrote:>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > For the Impala toolchain you can start with this list of commits>
>>>>>
>>>>> > > <https://gerrit.cloudera.org/#/q/ppc64le>
>>>>> <https://gerrit.cloudera.org/#/q/ppc64le>, in particular this review>
>>>>>
>>>>> > > <https://gerrit.cloudera.org/#/c/9077/>
>>>>> <https://gerrit.cloudera.org/#/c/9077/>.>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > I don't know about kudu, you might want to ask on their mailing
>>>>> list or>
>>>>>
>>>>> > > their slack channel. If you look in impala-config.sh for>
>>>>>
>>>>> > > the KUDU_IS_SUPPORTED flag, you should find a way to build Impala
>>>>> without>
>>>>>
>>>>> > > Kudu.>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > Cheers, Lars>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >>
>>>>>
>>>>> > > On Fri, Aug 16, 2019 at 5:32 AM Koltsov Dmitriy <d....@yadro.com>
>>>>> <d....@yadro.com>>
>>>>>
>>>>> > > wrote:>
>>>>>
>>>>> > >>
>>>>>
>>>>> > >> Hello, Lars!>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >> Thanks for operative answer.>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >> And could please inform about actual commit number of the>
>>>>>
>>>>> > >> native-toolchain on github which can be build successfully at
>>>>> that moment ?>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >> Btw: I ask about it because build process for current commit of>
>>>>>
>>>>> > >> native-toolchain finishes with error. And error is because of
>>>>> kudu build>
>>>>>
>>>>> > >> fail (other components of toolchain have been built fine). For
>>>>> now build.sh>
>>>>>
>>>>> > >> file in this commit of kudu src have explicit switch to disable
>>>>> ppc64le.>
>>>>>
>>>>> > >> And if I manually enable ppc64le case then code will try to build
>>>>> kudu from>
>>>>>
>>>>> > >> github; but github version kudu tries to build own toolchain
>>>>> while>
>>>>>
>>>>> > >> thirdparty building and it seems that for know current commit of
>>>>> github>
>>>>>
>>>>> > >> kudu has an error on the way to build kudu on Ubuntu 16.04
>>>>> ppc64le.>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >> So another question is - Can Kudu be build on Ubuntu 16.04
>>>>> ppc64le ? If>
>>>>>
>>>>> > >> yes - could you please inform me about appropriate commit of
>>>>> github kudu or>
>>>>>
>>>>> > >> maybe about another way to build it ?>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >> Regards,>
>>>>>
>>>>> > >> Koltsov Dmitriy>
>>>>>
>>>>> > >> System Engineer, YADRO>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >> On 16.08.2019 00:39, Lars Volker wrote:>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >> In short the answer is "likely not". We don't have any tests for
>>>>> ppc64le.>
>>>>>
>>>>> > >> About two years ago we had some commits to enable the toolchain
>>>>> on power,>
>>>>>
>>>>> > >> but I can't recall or find any changes to Impala itself related
>>>>> to that>
>>>>>
>>>>> > >> effort.>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >> Cheers, Lars>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >> On Thu, Aug 15, 2019 at 11:53 AM Koltsov Dmitriy
>>>>> <d....@yadro.com> <d....@yadro.com>>
>>>>>
>>>>> > >> wrote:>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >>> Hello!>
>>>>>
>>>>> > >>>>
>>>>>
>>>>> > >>> Could you please inform if Impala can be build and run on Ubuntu
>>>>> 16.04>
>>>>>
>>>>> > >>> ppc64le ?>
>>>>>
>>>>> > >>>>
>>>>>
>>>>> > >>>>
>>>>>
>>>>> > >>>>
>>>>>
>>>>> > >>>>
>>>>>
>>>>> > >>> -->
>>>>>
>>>>> > >>> Regards,>
>>>>>
>>>>> > >>> Koltsov Dmitriy>
>>>>>
>>>>> > >>> System Engineer, YADRO.>
>>>>>
>>>>> > >>>>
>>>>>
>>>>> > >>>>
>>>>>
>>>>> > >>>
>>>>>
>>>>> > >>
>>>>>
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
