Re: Improving Kudu Build Support

[Thomas Tauber-Marshall]

This has gone in now. You should be able to rebase, re-source
impala-config.sh, restart the minicluster (eg. with ./buildall.sh -noclean
-notests -start_minicluster), and continue working as normal.

If you rebase and fail to re-source impala-config.sh, you may see an error
like:
Traceback (most recent call last):
  File "/home/thomas/Impala/infra/python/bootstrap_virtualenv.py", line
383, in 
kudu_client_dir = find_kudu_client_install_dir()
  File "/home/thomas/Impala/infra/python/bootstrap_virtualenv.py", line
316, in find_kudu_client_install_dir
kudu_base_dir = os.environ["IMPALA_KUDU_HOME"]
  File "/usr/lib/python2.7/UserDict.py", line 40, in __getitem__
raise KeyError(key)
KeyError: 'IMPALA_KUDU_HOME'

Let me know if you encounter any other issues.

On Thu, Aug 30, 2018 at 1:31 PM Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

> The patch for this is finally out: https://gerrit.cloudera.org/#/c/11363/
>
> I put some work into trying to make sure this will be as seamless as
> possible for everyone.
> - For people on Ubuntu 16 or the other CDH supported distros, you'll
> automatically start using the CDH Kudu.
> - For people on Ubuntu 14, Impala will automatically set
> USE_CDH_KUDU=false and fall back to the toolchain Kudu. You should see no
> change.
> - For people who want to develop on other OSes, KUDU_IS_SUPPORTED will get
> set to false, we will build a stub Kudu client, and you should be able to
> build and run as normal, except that Kudu related operations will fail. Or
> you can build Kudu locally and link Impala against it using the
> KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables
>
> I also filed IMPALA-7515
>  to track the eventual
> removal of toolchain Kudu support
>
> On Wed, Aug 22, 2018 at 11:31 AM Thomas Tauber-Marshall <
> tmarsh...@cloudera.com> wrote:
>
>> So the toolchain binaries are provided for: centos5,6,7, debian7,8,
>> sles11,12, ubuntu14,16
>> The new CDH component binaries will be available for: redhat6,7, debian8,
>> sles12, ubuntu16
>> so we would be dropping easy support for building on basically centos5,
>> debian7, sles11, and ubuntu14
>>
>> For people who want to develop on other oses like Ubuntu 18.04, it would
>> still be possible by compiling Kudu locally and using the
>> KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables.
>>
>> I'm also strongly leaning towards hiding this functionality behind a flag
>> for now (not many people chimed in on this thread, but I suspect there are
>> still a decent number of people developing on Ubuntu 14 who I'll get angry
>> emails from if they rebase and discover they have to upgrade their os), in
>> which case building the toolchain would remain a viable option for using
>> other oses as well, as least for now.
>>
>> On Tue, Aug 21, 2018 at 4:47 PM Tim Armstrong 
>> wrote:
>>
>>> Is there a path to building a version of Kudu locally for an arbitrary
>>> linux distro?
>>>
>>> Personally I am less concerned about 14.04 support and more concerned
>>> about
>>> what the path to upgrading to 18.04. It would also be nice for it to be
>>> at
>>> least possible to develop on RedHat-derived distros even if it requires
>>> some extra effort.
>>>
>>> On Tue, Aug 21, 2018 at 6:48 AM, Laszlo Gaal 
>>> wrote:
>>>
>>> > +1 for simplifying Kudu updates.
>>> >
>>> > I am also still on Ubuntu 14.04, but I am all for simplifying Kudu
>>> > integration:
>>> > I agree with Thomas that Kudu snapshots should be grouped with the
>>> other
>>> > CDH components.
>>> > Given that Ubuntu 14.04 will be EOL'd next spring, upgrading the
>>> > development OS
>>> > is a reasonably small price to pay -- especially that it will soon
>>> become
>>> > necessary anyway.
>>> >
>>> > Thanks for doing this Thomas!
>>> >
>>> >   - Laszlo
>>> >
>>> > On Tue, Aug 21, 2018 at 12:34 AM Lars Volker  wrote:
>>> >
>>> > > I'm in favor of not spending developer time and effort to maintain
>>> > > compatibility with 14.04. Personally I'm still developing on Ubuntu
>>> 14.04
>>> > > so I'd be happy if we can support it without much pain. On the other
>>> hand
>>> > > it EOLs in April 2019, so I might as well go to 18.04 now, should we
>>> > decide
>>> > > to drop support. Maybe not many other folks are on 14.04 after all?
>>> > >
>>> > >
>>> > >
>>> > > On Mon, Aug 20, 2018 at 10:06 AM Thomas Tauber-Marshall <
>>> > > tmarsh...@cloudera.com> wrote:
>>> > >
>>> > > > Impala community,
>>> > > >
>>> > > > For years now, Impala has utilized tarballs built by Cloudera and
>>> > > uploaded
>>> > > > to S3 for running most of the Hadoop components in the testing
>>> > > minicluster.
>>> > > > The one exception to this is Kudu, which is instead provided by the
>>> > > > toolchain.
>>> > > >
>>> > > > This was never ideal - native-toolchain makes more sense for
>>> libraries
>>> > > > where we want to build against a fairly static version, but Kudu is
>>> > under
>>> > > > active development and we'd like to always build against a

Re: Improving Kudu Build Support

[Thomas Tauber-Marshall]

The patch for this is finally out: https://gerrit.cloudera.org/#/c/11363/

I put some work into trying to make sure this will be as seamless as
possible for everyone.
- For people on Ubuntu 16 or the other CDH supported distros, you'll
automatically start using the CDH Kudu.
- For people on Ubuntu 14, Impala will automatically set USE_CDH_KUDU=false
and fall back to the toolchain Kudu. You should see no change.
- For people who want to develop on other OSes, KUDU_IS_SUPPORTED will get
set to false, we will build a stub Kudu client, and you should be able to
build and run as normal, except that Kudu related operations will fail. Or
you can build Kudu locally and link Impala against it using the
KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables

I also filed IMPALA-7515 
to track the eventual removal of toolchain Kudu support

On Wed, Aug 22, 2018 at 11:31 AM Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

> So the toolchain binaries are provided for: centos5,6,7, debian7,8,
> sles11,12, ubuntu14,16
> The new CDH component binaries will be available for: redhat6,7, debian8,
> sles12, ubuntu16
> so we would be dropping easy support for building on basically centos5,
> debian7, sles11, and ubuntu14
>
> For people who want to develop on other oses like Ubuntu 18.04, it would
> still be possible by compiling Kudu locally and using the
> KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables.
>
> I'm also strongly leaning towards hiding this functionality behind a flag
> for now (not many people chimed in on this thread, but I suspect there are
> still a decent number of people developing on Ubuntu 14 who I'll get angry
> emails from if they rebase and discover they have to upgrade their os), in
> which case building the toolchain would remain a viable option for using
> other oses as well, as least for now.
>
> On Tue, Aug 21, 2018 at 4:47 PM Tim Armstrong 
> wrote:
>
>> Is there a path to building a version of Kudu locally for an arbitrary
>> linux distro?
>>
>> Personally I am less concerned about 14.04 support and more concerned
>> about
>> what the path to upgrading to 18.04. It would also be nice for it to be at
>> least possible to develop on RedHat-derived distros even if it requires
>> some extra effort.
>>
>> On Tue, Aug 21, 2018 at 6:48 AM, Laszlo Gaal 
>> wrote:
>>
>> > +1 for simplifying Kudu updates.
>> >
>> > I am also still on Ubuntu 14.04, but I am all for simplifying Kudu
>> > integration:
>> > I agree with Thomas that Kudu snapshots should be grouped with the other
>> > CDH components.
>> > Given that Ubuntu 14.04 will be EOL'd next spring, upgrading the
>> > development OS
>> > is a reasonably small price to pay -- especially that it will soon
>> become
>> > necessary anyway.
>> >
>> > Thanks for doing this Thomas!
>> >
>> >   - Laszlo
>> >
>> > On Tue, Aug 21, 2018 at 12:34 AM Lars Volker  wrote:
>> >
>> > > I'm in favor of not spending developer time and effort to maintain
>> > > compatibility with 14.04. Personally I'm still developing on Ubuntu
>> 14.04
>> > > so I'd be happy if we can support it without much pain. On the other
>> hand
>> > > it EOLs in April 2019, so I might as well go to 18.04 now, should we
>> > decide
>> > > to drop support. Maybe not many other folks are on 14.04 after all?
>> > >
>> > >
>> > >
>> > > On Mon, Aug 20, 2018 at 10:06 AM Thomas Tauber-Marshall <
>> > > tmarsh...@cloudera.com> wrote:
>> > >
>> > > > Impala community,
>> > > >
>> > > > For years now, Impala has utilized tarballs built by Cloudera and
>> > > uploaded
>> > > > to S3 for running most of the Hadoop components in the testing
>> > > minicluster.
>> > > > The one exception to this is Kudu, which is instead provided by the
>> > > > toolchain.
>> > > >
>> > > > This was never ideal - native-toolchain makes more sense for
>> libraries
>> > > > where we want to build against a fairly static version, but Kudu is
>> > under
>> > > > active development and we'd like to always build against a
>> relatively
>> > > > up-to-date version. As a result, patches just bumping the version of
>> > Kudu
>> > > > make up a significant portion of the commit history of
>> > native-toolchain.
>> > > >
>> > > > Thanks to work I'm currently doing at Cloudera, there will soon be
>> > > snapshot
>> > > > tarballs of Kudu getting uploaded to S3 along with the other Hadoop
>> > > > components. I would like to propose that Impala switch to using
>> those
>> > > > instead of the toolchain Kudu.
>> > > >
>> > > > One problem here is that the new Kudu tarballs will not be getting
>> > build
>> > > > for Ubuntu 14.04, only 16.04, but we still officially say we support
>> > > > development on 14.04.
>> > > >
>> > > > One option here would be to maintain the toolchain Kudu for now and
>> > hide
>> > > > downloading of the new tarballs behind a flag. We could also
>> postpone
>> > > some
>> > > > of this work until 14.04 is less common. Or, given that the
>> > > > bootstrap_development script 

Re: Improving Kudu Build Support

[Thomas Tauber-Marshall]

So the toolchain binaries are provided for: centos5,6,7, debian7,8,
sles11,12, ubuntu14,16
The new CDH component binaries will be available for: redhat6,7, debian8,
sles12, ubuntu16
so we would be dropping easy support for building on basically centos5,
debian7, sles11, and ubuntu14

For people who want to develop on other oses like Ubuntu 18.04, it would
still be possible by compiling Kudu locally and using the
KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables.

I'm also strongly leaning towards hiding this functionality behind a flag
for now (not many people chimed in on this thread, but I suspect there are
still a decent number of people developing on Ubuntu 14 who I'll get angry
emails from if they rebase and discover they have to upgrade their os), in
which case building the toolchain would remain a viable option for using
other oses as well, as least for now.

On Tue, Aug 21, 2018 at 4:47 PM Tim Armstrong 
wrote:

> Is there a path to building a version of Kudu locally for an arbitrary
> linux distro?
>
> Personally I am less concerned about 14.04 support and more concerned about
> what the path to upgrading to 18.04. It would also be nice for it to be at
> least possible to develop on RedHat-derived distros even if it requires
> some extra effort.
>
> On Tue, Aug 21, 2018 at 6:48 AM, Laszlo Gaal 
> wrote:
>
> > +1 for simplifying Kudu updates.
> >
> > I am also still on Ubuntu 14.04, but I am all for simplifying Kudu
> > integration:
> > I agree with Thomas that Kudu snapshots should be grouped with the other
> > CDH components.
> > Given that Ubuntu 14.04 will be EOL'd next spring, upgrading the
> > development OS
> > is a reasonably small price to pay -- especially that it will soon become
> > necessary anyway.
> >
> > Thanks for doing this Thomas!
> >
> >   - Laszlo
> >
> > On Tue, Aug 21, 2018 at 12:34 AM Lars Volker  wrote:
> >
> > > I'm in favor of not spending developer time and effort to maintain
> > > compatibility with 14.04. Personally I'm still developing on Ubuntu
> 14.04
> > > so I'd be happy if we can support it without much pain. On the other
> hand
> > > it EOLs in April 2019, so I might as well go to 18.04 now, should we
> > decide
> > > to drop support. Maybe not many other folks are on 14.04 after all?
> > >
> > >
> > >
> > > On Mon, Aug 20, 2018 at 10:06 AM Thomas Tauber-Marshall <
> > > tmarsh...@cloudera.com> wrote:
> > >
> > > > Impala community,
> > > >
> > > > For years now, Impala has utilized tarballs built by Cloudera and
> > > uploaded
> > > > to S3 for running most of the Hadoop components in the testing
> > > minicluster.
> > > > The one exception to this is Kudu, which is instead provided by the
> > > > toolchain.
> > > >
> > > > This was never ideal - native-toolchain makes more sense for
> libraries
> > > > where we want to build against a fairly static version, but Kudu is
> > under
> > > > active development and we'd like to always build against a relatively
> > > > up-to-date version. As a result, patches just bumping the version of
> > Kudu
> > > > make up a significant portion of the commit history of
> > native-toolchain.
> > > >
> > > > Thanks to work I'm currently doing at Cloudera, there will soon be
> > > snapshot
> > > > tarballs of Kudu getting uploaded to S3 along with the other Hadoop
> > > > components. I would like to propose that Impala switch to using those
> > > > instead of the toolchain Kudu.
> > > >
> > > > One problem here is that the new Kudu tarballs will not be getting
> > build
> > > > for Ubuntu 14.04, only 16.04, but we still officially say we support
> > > > development on 14.04.
> > > >
> > > > One option here would be to maintain the toolchain Kudu for now and
> > hide
> > > > downloading of the new tarballs behind a flag. We could also postpone
> > > some
> > > > of this work until 14.04 is less common. Or, given that the
> > > > bootstrap_development script already only supports 16.04, we might
> want
> > > to
> > > > just drop support for building on 14.04.
> > > >
> > > > Thoughts?
> > > >
> > >
> >
>

Re: Improving Kudu Build Support

[Tim Armstrong]

Is there a path to building a version of Kudu locally for an arbitrary
linux distro?

Personally I am less concerned about 14.04 support and more concerned about
what the path to upgrading to 18.04. It would also be nice for it to be at
least possible to develop on RedHat-derived distros even if it requires
some extra effort.

On Tue, Aug 21, 2018 at 6:48 AM, Laszlo Gaal 
wrote:

> +1 for simplifying Kudu updates.
>
> I am also still on Ubuntu 14.04, but I am all for simplifying Kudu
> integration:
> I agree with Thomas that Kudu snapshots should be grouped with the other
> CDH components.
> Given that Ubuntu 14.04 will be EOL'd next spring, upgrading the
> development OS
> is a reasonably small price to pay -- especially that it will soon become
> necessary anyway.
>
> Thanks for doing this Thomas!
>
>   - Laszlo
>
> On Tue, Aug 21, 2018 at 12:34 AM Lars Volker  wrote:
>
> > I'm in favor of not spending developer time and effort to maintain
> > compatibility with 14.04. Personally I'm still developing on Ubuntu 14.04
> > so I'd be happy if we can support it without much pain. On the other hand
> > it EOLs in April 2019, so I might as well go to 18.04 now, should we
> decide
> > to drop support. Maybe not many other folks are on 14.04 after all?
> >
> >
> >
> > On Mon, Aug 20, 2018 at 10:06 AM Thomas Tauber-Marshall <
> > tmarsh...@cloudera.com> wrote:
> >
> > > Impala community,
> > >
> > > For years now, Impala has utilized tarballs built by Cloudera and
> > uploaded
> > > to S3 for running most of the Hadoop components in the testing
> > minicluster.
> > > The one exception to this is Kudu, which is instead provided by the
> > > toolchain.
> > >
> > > This was never ideal - native-toolchain makes more sense for libraries
> > > where we want to build against a fairly static version, but Kudu is
> under
> > > active development and we'd like to always build against a relatively
> > > up-to-date version. As a result, patches just bumping the version of
> Kudu
> > > make up a significant portion of the commit history of
> native-toolchain.
> > >
> > > Thanks to work I'm currently doing at Cloudera, there will soon be
> > snapshot
> > > tarballs of Kudu getting uploaded to S3 along with the other Hadoop
> > > components. I would like to propose that Impala switch to using those
> > > instead of the toolchain Kudu.
> > >
> > > One problem here is that the new Kudu tarballs will not be getting
> build
> > > for Ubuntu 14.04, only 16.04, but we still officially say we support
> > > development on 14.04.
> > >
> > > One option here would be to maintain the toolchain Kudu for now and
> hide
> > > downloading of the new tarballs behind a flag. We could also postpone
> > some
> > > of this work until 14.04 is less common. Or, given that the
> > > bootstrap_development script already only supports 16.04, we might want
> > to
> > > just drop support for building on 14.04.
> > >
> > > Thoughts?
> > >
> >
>

Re: Improving Kudu Build Support

[Laszlo Gaal]

+1 for simplifying Kudu updates.

I am also still on Ubuntu 14.04, but I am all for simplifying Kudu
integration:
I agree with Thomas that Kudu snapshots should be grouped with the other
CDH components.
Given that Ubuntu 14.04 will be EOL'd next spring, upgrading the
development OS
is a reasonably small price to pay -- especially that it will soon become
necessary anyway.

Thanks for doing this Thomas!

  - Laszlo

On Tue, Aug 21, 2018 at 12:34 AM Lars Volker  wrote:

> I'm in favor of not spending developer time and effort to maintain
> compatibility with 14.04. Personally I'm still developing on Ubuntu 14.04
> so I'd be happy if we can support it without much pain. On the other hand
> it EOLs in April 2019, so I might as well go to 18.04 now, should we decide
> to drop support. Maybe not many other folks are on 14.04 after all?
>
>
>
> On Mon, Aug 20, 2018 at 10:06 AM Thomas Tauber-Marshall <
> tmarsh...@cloudera.com> wrote:
>
> > Impala community,
> >
> > For years now, Impala has utilized tarballs built by Cloudera and
> uploaded
> > to S3 for running most of the Hadoop components in the testing
> minicluster.
> > The one exception to this is Kudu, which is instead provided by the
> > toolchain.
> >
> > This was never ideal - native-toolchain makes more sense for libraries
> > where we want to build against a fairly static version, but Kudu is under
> > active development and we'd like to always build against a relatively
> > up-to-date version. As a result, patches just bumping the version of Kudu
> > make up a significant portion of the commit history of native-toolchain.
> >
> > Thanks to work I'm currently doing at Cloudera, there will soon be
> snapshot
> > tarballs of Kudu getting uploaded to S3 along with the other Hadoop
> > components. I would like to propose that Impala switch to using those
> > instead of the toolchain Kudu.
> >
> > One problem here is that the new Kudu tarballs will not be getting build
> > for Ubuntu 14.04, only 16.04, but we still officially say we support
> > development on 14.04.
> >
> > One option here would be to maintain the toolchain Kudu for now and hide
> > downloading of the new tarballs behind a flag. We could also postpone
> some
> > of this work until 14.04 is less common. Or, given that the
> > bootstrap_development script already only supports 16.04, we might want
> to
> > just drop support for building on 14.04.
> >
> > Thoughts?
> >
>

Re: Improving Kudu Build Support

[Lars Volker]

I'm in favor of not spending developer time and effort to maintain
compatibility with 14.04. Personally I'm still developing on Ubuntu 14.04
so I'd be happy if we can support it without much pain. On the other hand
it EOLs in April 2019, so I might as well go to 18.04 now, should we decide
to drop support. Maybe not many other folks are on 14.04 after all?



On Mon, Aug 20, 2018 at 10:06 AM Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

> Impala community,
>
> For years now, Impala has utilized tarballs built by Cloudera and uploaded
> to S3 for running most of the Hadoop components in the testing minicluster.
> The one exception to this is Kudu, which is instead provided by the
> toolchain.
>
> This was never ideal - native-toolchain makes more sense for libraries
> where we want to build against a fairly static version, but Kudu is under
> active development and we'd like to always build against a relatively
> up-to-date version. As a result, patches just bumping the version of Kudu
> make up a significant portion of the commit history of native-toolchain.
>
> Thanks to work I'm currently doing at Cloudera, there will soon be snapshot
> tarballs of Kudu getting uploaded to S3 along with the other Hadoop
> components. I would like to propose that Impala switch to using those
> instead of the toolchain Kudu.
>
> One problem here is that the new Kudu tarballs will not be getting build
> for Ubuntu 14.04, only 16.04, but we still officially say we support
> development on 14.04.
>
> One option here would be to maintain the toolchain Kudu for now and hide
> downloading of the new tarballs behind a flag. We could also postpone some
> of this work until 14.04 is less common. Or, given that the
> bootstrap_development script already only supports 16.04, we might want to
> just drop support for building on 14.04.
>
> Thoughts?
>