Hello, Jim.
Ok. Here is the explanation.
I wrote to this mailing list in order to find out the possibility of
successful build Impala in Ubuntu 16.04 ppc64le (if there is a support
of ppc64le in Impala project/code). As one can see in previous messages
of this thread, colleagues describe me a possible way in which one can
try to successfully build github apache Impala with cloudera
native-toolchain. There were no noticed objections from colleagues in
this thread which obviously lead to stop try to build Impala on ppc64le.
But for now, after passing some "barriers" in build process and after
some tricks, I finally pass configure stage of ./buildall.sh Impala
script (including toolchain check) and met the following error (see
below) on console while Impala build process (./buildall.sh executed).
So, looking at the error, naturally the question arises: is there de
facto a possibility for now to build github apache Impala with cloudera
native-toolchain in Ubuntu 16.04 ppc64le ? (Impala project supports
Ubuntu 16.04 ppc64le installation?)
If 'yes' - then could you, please, give some comments about the reason
of the following error (below) while Impala+"native-toolchain" build in
Ubuntu 16.04 ppc64le ? and about the way to resolve the error to
successfully build Impala ?
(May be there is some another commit of impala which is more "friendly"
with ppc64le and native-toolchain for now)
>>And I have a lot of error messages while Impala compiling process
in Ubuntu 16.04 ppc64le, e.g.:
>>
>>
>>...
>>make[2]: *** Deleting file
'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
>>In file included from
/home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
>> from
/home/build/impala/be/src/gutil/spinlock_internal.cc:44:
>>/home/build/impala/be/src/gutil/atomicops.h:88:2: error: #error
You need to implement atomic operations for this architecture
>> #error You need to implement atomic operations for this architecture
>> ^
>>/home/build/impala/be/src/gutil/atomicops.h:321:8: error:
âAtomic32â does not name a type
>> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr,
>>
>>...
>>
>> Impala build process failed.
On 08/27/19 17:32, Jim Apple wrote:
I'm a bit confused. The text of the error was "You need to implement
atomic operations for this architecture", right? Maybe could you be a
bit more descriptive about what your understanding of that is and why
it is a surprising error?
On Tue, Aug 27, 2019 at 2:51 AM Koltsov Dmitriy <d.kolt...@yadro.com
<mailto:d.kolt...@yadro.com>> wrote:
Ok, Tim.
Now it's clear - github.com/apache/impala
<http://github.com/apache/impala> and
github.com/cloudera/native-toolchain
<http://github.com/cloudera/native-toolchain> should be used for
build in Ubuntu 16.04 ppc64le.
But then still there is the question - could you, please, give
some comments about possible reason of the error shown below when
I tried this combination of impala and native-toolchain ? ... and
about possible ways to solve it ? May be there is some another
commit of impala which is more "friendly" with ppc64le and
native-toolchain for now ?
>>And I have a lot of error messages while compiling process,
e.g.:
>>
>>
>>...
>>make[2]: *** Deleting file
'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
>>In file included from
/home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
>> from
/home/build/impala/be/src/gutil/spinlock_internal.cc:44:
>>/home/build/impala/be/src/gutil/atomicops.h:88:2: error:
#error You need to implement atomic operations for this
architecture
>> #error You need to implement atomic operations for this
architecture
>> ^
>>/home/build/impala/be/src/gutil/atomicops.h:321:8: error:
âAtomic32â does not name a type
>> inline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr,
>>
>>...
>>
>> Impala build process failed.
On 08/27/19 01:51, Tim Armstrong wrote:
Hi Dmitry,
> But only by that moment I've noticed that there is a
significant difference between github.com/cloudera/Impala
<http://github.com/cloudera/Impala> and github.com/apache/impala
<http://github.com/apache/impala> - while build.
> And all previous steps I tried with github apache impala and
cloudera native-toolchain.
That is the right thing to do. Apache Impala master is the latest
and greatest. github.com/cloudera/impala
<http://github.com/cloudera/impala> mainly hosts snapshots of the
source from different Cloudera releases. It looks like the
default branch was cdh5-trunk, which is a fairly old version of
Impala at this point.
On Mon, Aug 26, 2019 at 1:50 PM Koltsov Dmitriy
<d.kolt...@yadro.com <mailto:d.kolt...@yadro.com>> wrote:
Hello Tim.
After some tricks I have success in building Kudu stub.
Thanks for advice.
But only by that moment I've noticed that there is a
significant difference between github.com/cloudera/Impala
<http://github.com/cloudera/Impala> and
github.com/apache/impala <http://github.com/apache/impala> -
while build.
And all previous steps I tried with github apache impala and
cloudera native-toolchain.
So, for now, after I successfully got Kudu stub, I've tried
to build final Impala build with github apache impala and
cloudera native-toolchain in Ubuntu 16.04 ppc64le. And I have
a lot of error messages while compiling process, e.g.:
...
make[2]: *** Deleting file
'be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp'
In file included from
/home/build/impala/be/src/gutil/spinlock_internal.h:40:0,
from
/home/build/impala/be/src/gutil/spinlock_internal.cc:44:
/home/build/impala/be/src/gutil/atomicops.h:88:2: error:
#error You need to implement atomic operations for this
architecture
#error You need to implement atomic operations for this
architecture
^
/home/build/impala/be/src/gutil/atomicops.h:321:8: error:
âAtomic32â does not name a type
inline Atomic32 Acquire_CompareAndSwap(volatile
Atomic32* ptr,
...
Impala build process failed.
So, hence now I'm trying only to use cloudera github impala
and cloudera native-toolchain. I use previously build
cloudera native-toolchain (git cloned from github), set
IMPALA_TOOLCHAIN var to the appropriate location; current
commit of cloudera impala is 7dd2c81. But when ./buildall.sh
is executed I have the following error:
-- Added static library dependency pprof:
/home/build/impala/toolchain/gperftools-2.5/lib/libprofiler.a
-- ----------> Adding thirdparty library gtest. <----------
-- Header files: /home/build/impala/toolchain/gtest-1.6.0/include
-- Added static library dependency gtest:
/home/build/impala/toolchain/gtest-1.6.0/lib/libgtest.a
CMake Error at cmake_modules/FindLlvmBinaries.cmake:33 (message):
Could not find llvm-config
Call Stack (most recent call first):
CMakeLists.txt:208 (find_package)
-- Configuring incomplete, errors occurred!
See also
"/home/build/impala_cloudera/CMakeFiles/CMakeOutput.log".
See also "/home/build/impala_cloudera/CMakeFiles/CMakeError.log".
Error in /home/build/impala_cloudera/bin/make_impala.sh at
line 161: cmake . ${CMAKE_ARGS[@]}
Though IMPALA_TOOLCHAIN path points to directory which
contains LLVM build: "llvm-5.0.1-p1" dir is in the path, with
all appropriate files.
_
__*Question:*_ Could you, please, give a comment about
possible reason of the error and about how it could be resolved ?
btw: there is some inconsistency about llvm versions between
one in ./bin/impala-config.sh (export
IMPALA_LLVM_VERSION=3.9.1) and default one of llvm build in
cloudera native-toolchain (5.0.1-p1).
On 08/24/19 03:49, Tim Armstrong wrote:
Ah I see. I think if you run bin/bootstrap_toolchain.py
manually it will build the stub for you.
I wouldn't be surprised if this requires some tweaks to the
scripts to make it work since they weren't written for this
exact case in mind (the original use case for the kudu stubs
was for OSes that Kudu didn't support but Impala did, like
CentOS 5).
On Wed, Aug 21, 2019 at 11:12 AM Koltsov Dmitriy
<d.kolt...@yadro.com <mailto:d.kolt...@yadro.com>> wrote:
Hello, Tim.
Thanks for reply.
When I'm trying to build Impala I am of course using
already built clouder native-toolchain (cloned from
github repo). This native-toolchain has no
kudu in final build directory. So, when I'm trying to
build Impala pointing this build through
IMPALA_TOOLCHAIN variable with
SKIP_TOOLCHAIN_BOOSTRAP=true then I have an error
mentioned by me above (see the fragment below). Could
you, please, inform if there is a possibility to
instruct Impala building process not to use Kudu in
Impala build on Ubuntu 16.04 ppc64le ?
> > -- ----------> Adding thirdparty library libunwind.
<---------->
> > -- Header files:
/home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
> > -- Added shared library dependency libunwind:>
> >
/home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
> > CMake Error at CMakeLists.txt:409 (find_package):>
> > Could not find a package configuration file provided
by "kuduClient" with>
> > any of the following names:>
> >>
> > kuduClientConfig.cmake>
> > kuduclient-config.cmake>
> >>
> > Add the installation prefix of "kuduClient" to
CMAKE_PREFIX_PATH or set>
> > "kuduClient_DIR" to a directory containing one of
the above files. If>
> > "kuduClient" provides a separate development package
or SDK, be sure it>
> > has>
> > been installed.>
> >>
> >>
> > -- Configuring incomplete, errors occurred!>
> > See also
"/home/build/impala/CMakeFiles/CMakeOutput.log".>
> > See also
"/home/build/impala/CMakeFiles/CMakeError.log".>
> > /home/build/impala#>
> > /home/build/impala#>
On 2019/08/21 16:47:48, Tim Armstrong
<t...@cloudera.com> <mailto:t...@cloudera.com> wrote:
> Hi Dmitry,>
>
> The toolchain bootstrap process downloads pre-built
binaries, and we don't>
> have any for PPC.>
>
> You would need to build the toolchain locally for PPC,
then point the>
> Impala build at it. These instructions might be
helpful for you ->
>
https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala>
> .>
>
>
>
> On Tue, Aug 20, 2019 at 3:53 PM Koltsov Dmitriy
<d....@yadro.com> <mailto:d....@yadro.com> wrote:>
>
> > I've looked at the contents of impala-config.sh. As
I understand - I>
> > should manually set KUDU_IS_SUPPORTED=false in order
to make general impala>
> > build process to download kudu and build stub. But I
use native_toolchain>
> > and so var SKIP_TOOLCHAIN_BOOTSTRAP=true. So it
seems that there is mutual>
> > exclusion: if I want to build impala with github
cloudera native_toolchain>
> > then code branch for kudu stub building is disabled>
> > (SKIP_TOOLCHAIN_BOOTSTRAP=true), and if I set>
> > SKIP_TOOLCHAIN_BOOTSTRAP=false to enable general
build process to download>
> > and build kudu stub then there will be no other
components available from>
> > cloudera native_toolchain.>
> >>
> > For example, when I use first case with
SKIP_TOOLCHAIN_BOOTSTRAP=true and>
> > manually KUDU_IS_SUPPORTED=false set in
impala-config.sh, then I have the>
> > following error when trying ./buildall.sh for
cloudera impala:>
> >>
> > ...>
> > -- ----------> Adding thirdparty library krb5.
<---------->
> > -- Header files: /usr/include>
> > -- Added shared library dependency krb5:>
> > /usr/lib/powerpc64le-linux-gnu/libkrb5.so>
> > -- Found Kerberos: /usr/sbin/kadmin.local>
> > -- Added shared library dependency rt:>
> > /usr/lib/powerpc64le-linux-gnu/librt.so>
> > -- Added shared library dependency dl:>
> > /usr/lib/powerpc64le-linux-gnu/libdl.so>
> > -- Found LIBUNWIND:>
> >
/home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
> > -- ----------> Adding thirdparty library libunwind.
<---------->
> > -- Header files:
/home/build/impala/toolchain/libunwind-1.3-rc1-p3/include>
> > -- Added shared library dependency libunwind:>
> >
/home/build/native-toolchain/build/libunwind-1.3-rc1-p3/lib64/>
> > CMake Error at CMakeLists.txt:409 (find_package):>
> > Could not find a package configuration file provided
by "kuduClient" with>
> > any of the following names:>
> >>
> > kuduClientConfig.cmake>
> > kuduclient-config.cmake>
> >>
> > Add the installation prefix of "kuduClient" to
CMAKE_PREFIX_PATH or set>
> > "kuduClient_DIR" to a directory containing one of
the above files. If>
> > "kuduClient" provides a separate development package
or SDK, be sure it>
> > has>
> > been installed.>
> >>
> >>
> > -- Configuring incomplete, errors occurred!>
> > See also
"/home/build/impala/CMakeFiles/CMakeOutput.log".>
> > See also
"/home/build/impala/CMakeFiles/CMakeError.log".>
> > /home/build/impala#>
> > /home/build/impala#>
> >>
> >>
> > Could you please describe how can one exactly build
cloudera impala with>
> > its native_toolchain and kudu stub simultaneously on
Ubuntu 16.04 ppc64le ?>
> >>
> >>
> >>
> >>
> >>
> > Regards,>
> > Koltsov Dmitriy>
> > System Engineer, YADRO>
> >>
> >>
> >>
> > On 08/16/19 18:15, Lars Volker wrote:>
> >>
> > For the Impala toolchain you can start with this
list of commits>
> > <https://gerrit.cloudera.org/#/q/ppc64le>
<https://gerrit.cloudera.org/#/q/ppc64le>, in particular
this review>
> > <https://gerrit.cloudera.org/#/c/9077/>
<https://gerrit.cloudera.org/#/c/9077/>.>
> >>
> > I don't know about kudu, you might want to ask on
their mailing list or>
> > their slack channel. If you look in impala-config.sh
for>
> > the KUDU_IS_SUPPORTED flag, you should find a way to
build Impala without>
> > Kudu.>
> >>
> > Cheers, Lars>
> >>
> >>
> >>
> > On Fri, Aug 16, 2019 at 5:32 AM Koltsov Dmitriy
<d....@yadro.com> <mailto:d....@yadro.com>>
> > wrote:>
> >>
> >> Hello, Lars!>
> >>>
> >> Thanks for operative answer.>
> >>>
> >> And could please inform about actual commit number
of the>
> >> native-toolchain on github which can be build
successfully at that moment ?>
> >>>
> >> Btw: I ask about it because build process for
current commit of>
> >> native-toolchain finishes with error. And error is
because of kudu build>
> >> fail (other components of toolchain have been built
fine). For now build.sh>
> >> file in this commit of kudu src have explicit
switch to disable ppc64le.>
> >> And if I manually enable ppc64le case then code
will try to build kudu from>
> >> github; but github version kudu tries to build own
toolchain while>
> >> thirdparty building and it seems that for know
current commit of github>
> >> kudu has an error on the way to build kudu on
Ubuntu 16.04 ppc64le.>
> >>>
> >> So another question is - Can Kudu be build on
Ubuntu 16.04 ppc64le ? If>
> >> yes - could you please inform me about appropriate
commit of github kudu or>
> >> maybe about another way to build it ?>
> >>>
> >>>
> >>>
> >> Regards,>
> >> Koltsov Dmitriy>
> >> System Engineer, YADRO>
> >>>
> >>>
> >> On 16.08.2019 00:39, Lars Volker wrote:>
> >>>
> >> In short the answer is "likely not". We don't have
any tests for ppc64le.>
> >> About two years ago we had some commits to enable
the toolchain on power,>
> >> but I can't recall or find any changes to Impala
itself related to that>
> >> effort.>
> >>>
> >> Cheers, Lars>
> >>>
> >> On Thu, Aug 15, 2019 at 11:53 AM Koltsov Dmitriy
<d....@yadro.com> <mailto:d....@yadro.com>>
> >> wrote:>
> >>>
> >>> Hello!>
> >>>>
> >>> Could you please inform if Impala can be build and
run on Ubuntu 16.04>
> >>> ppc64le ?>
> >>>>
> >>>>
> >>>>
> >>>>
> >>> -->
> >>> Regards,>
> >>> Koltsov Dmitriy>
> >>> System Engineer, YADRO.>
> >>>>
> >>>>
> >>>
> >>
>
