Re: Help with fixing clang-diagnostic-over-aligned

[Sailesh Mukil]

Thanks for the explanation Todd and Dan! I had introduced a new
CACHELINE_ALIGNED object into the class which caused this problem. Making
the parent classes CACHELINE_ALIGNED too fixed the problem.

On Wed, Feb 14, 2018 at 12:58 PM, Todd Lipcon  wrote:

> This probably means that QueryExecMgr (or one of its descendent members) is
> annotated with __attribute__((aligned(64)), perhaps through something like
> a CACHELINE_ALIGNED macro.
>
> The issue is that, as the warning says, normal libc malloc doesn't make any
> hard guarantees about the alignment of allocations, and the way that
> 'operator new' is implemented by default only gets the size and not the
> desired allocation. I believe C++17 is adding support for an 'operator new'
> overload that also passes the desired alignment, so that it can call down
> into posix_memalign and ensure the appropriate allocation.
>
> The "correct" fix prior to C++17 is probably to do something like overload
> 'operator new' for the type, or to use posix_memalign manually to allocate
> aligned memory, and then placement-new it into that memory.
>
> Another reasonable fix would probably be to add a CHECK in the constructor
> that 'this' is aligned as desired so that, if you ever run on an allocator
> that isn't guaranteeing what you think it is, you'll know immediately
> rather than getting a surprising perf or atomicity regression due to
> cross-cacheline loads/stores.
>
> -Todd
>
>
> On Wed, Feb 14, 2018 at 12:33 PM, Sailesh Mukil 
> wrote:
>
> > Does anyone have experience with fixing the following type of
> > clang-tidy warning?
> >
> >
> > /home/ubuntu/Impala/be/src/runtime/exec-env.cc:159:21: warning: type
> > 'impala::QueryExecMgr' requires 64 bytes of alignment and the default
> > allocator only guarantees 16 bytes [clang-diagnostic-over-aligned]
> > query_exec_mgr_(new QueryExecMgr()),
> >
> >
> > /home/ubuntu/Impala/be/src/service/impalad-main.cc:80:49: warning:
> > type 'impala::ImpalaServer' requires 64 bytes of alignment and the
> > default allocator only guarantees 16 bytes
> > [clang-diagnostic-over-aligned]
> >   boost::shared_ptr impala_server(new
> > ImpalaServer(_env));
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: IMPALA-3343: impala-shell compatibility with Python 3

[Diggs, Asoka]

Fair enough Philip.  I will do some more research and add the results of that 
research to IMPALA-3343 at a minimum.  I don't currently have an impala 
development environment configured, along with test case execution, and I have 
enough other project work going on that I don't know if I really have the time 
to do that incremental infrastructure building.  I've done enough research to 
find some tools to help with Python 2 to 3 conversions, and see that there will 
be at least a little bit more than print statements.

What I know I can do is start modifying my local copy of the file and see if I 
can get impala-shell working again for myself, and share the results of that 
research / exploration.

Thanks

On 2/14/18, 1:58 PM, "Philip Zeyliger"  wrote:

I'm not aware of anyone having tried it, so I don't think we know. I'd also
check in on Thrift's support for Python3. That seems like the most likely
library. You can certainly give removing print a shot!

-- Philip

On Wed, Feb 14, 2018 at 11:50 AM, Diggs, Asoka 
wrote:

> I have just encountered this error myself after my project made the
> decision to switch from Python 2 to 3.
>
> I’ve added a comment to the JIRA with some additional details from my own
> encounter with the error.
>
>
> Looking at the impala-shell.py file, it’s clear that the error I
> encountered – print statements formulated using Python 2 syntax that is 
not
> compatible with Python 3 – exist throughout the code.  If this is the
> extent of the incompatibility, then that’s a problem I am ready and 
willing
> to tackle.
>
> Is there reason to believe that this problem is as simple as changing
> print statement syntax, or reason to believe it is significantly more than
> that?
>
> Thanks for the insight.
> Asoka Diggs
>
>
>
>

Re: Re: Re: Build fail in native-toolchain project

[Laszlo Gaal]

I found the difference between your build and mine, which may be
significant. My  hint was that your build tried to compile the db_berkeley
module in sasldb when it errored out, but my build compiled something
called db_none at the corresponding location. Digging into the "configure"
phase (at the start of the log you posted) your build logged:

checking db.h usability... yes
checking db.h presence... yes
checking for db.h... yes
checking DB library to use... berkeley
checking if Berkeley DB handle is kept open in SASLDB... no


my corresponding log is:

checking db.h usability... no

checking db.h presence... no

checking for db.h... no

checking ndbm.h usability... no

checking ndbm.h presence... no

checking for ndbm.h... no

checking gdbm.h usability... no

checking gdbm.h presence... no

checking for gdbm.h... no

checking DB library to use... no

configure: WARNING: Disabling SASL authentication database support

checking if Berkeley DB handle is kept open in SASLDB... no

so, apparently the configure phase on your system managed to find
BerkeleyDB whereas during my build it was not found, which explains the
difference between the "configure" phases.

Checking source/cyrus_sasl/build.sh supplies another interesting detail:

  CONFIGURE_FLAGS=

  # If libdb5 is installed, it would be used by default and would lead to a
failure.

  # If libdb4 appears to be installed, use that instead. The location below
was found

  # on a CentOS 7 machine.

  if [[ -e /usr/include/libdb4 && -e /usr/lib64/libdb4 ]]; then

CONFIGURE_FLAGS+=" --with-bdb-incdir=/usr/include/libdb4"

CONFIGURE_FLAGS+=" --with-bdb-libdir=/usr/lib64/libdb4"

  fi

Recalling the neighborhood of the offending line from my previous message:

(this is in *sasldb/db_berkeley.c*, starting on *line 103*):

#if DB_VERSION_MAJOR == 4 && DB_VERSION_MINOR >= 1
ret = (*mbdb)->open(*mbdb, NULL, path, NULL, DB_HASH, flags, 0660);
#else
ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
#endif

to me it looks like you have BerkeleyDB v5 installed on your system, and it
breaks the cyrus_sasl build.
I'm at a bit of a loss here, as this has never happened to me, but my first
suggestion would be to remove the BerkeleyDB package (or at least the dev
package, to get rid of the header files) if you are not using it. If you
need it, then you may have some success tweaking
native-toolchain/source/cyrus-sasl/build.sh at the location listed above.

Please let us know if you managed to overcome this obstacle; looks like not
a lot of people have encountered this particular problem in the near past.

Good luck,

  - Laszlo

PS: Attaching my build log of cyrus-sasl for added context (this may be
removed by the apache.org list processor though):




On Wed, Feb 14, 2018 at 10:44 PM, Tim Armstrong 
wrote:

> There are. I tried to document the list at some point here:
> https://cwiki.apache.org/confluence/display/IMPALA/
> Building+native-toolchain+from+scratch+and+using+with+Impala
>
> I believe that covers most of them, but it may be slightly stale. Please
> feel free to update the list or let us know if something is missing.
>
> On Wed, Feb 14, 2018 at 1:36 PM, Quanlong Huang 
> wrote:
>
>> Are there any pre-requirements for native-toolchain? Maybe I need to
>> install something first.
>> I just check out the master branch and run buildall.sh. Here's more
>> info in my env:
>>
>> $ lsb_release -a
>>
>> No LSB modules are available.
>>
>> Distributor ID: Ubuntu
>>
>> Description:Ubuntu 14.04.2 LTS
>>
>> Release:14.04
>>
>> Codename:   trusty
>>
>> $ cat /etc/os-release
>>
>> NAME="Ubuntu"
>>
>> VERSION="14.04.2 LTS, Trusty Tahr"
>>
>> ID=ubuntu
>>
>> ID_LIKE=debian
>>
>> PRETTY_NAME="Ubuntu 14.04.2 LTS"
>>
>> VERSION_ID="14.04"
>>
>> HOME_URL="http://www.ubuntu.com/;
>>
>> SUPPORT_URL="http://help.ubuntu.com/;
>>
>> BUG_REPORT_URL="
>> http://bugs.launchpad.net/ubuntu/;
>>
>> $ uname -a
>>
>> Linux iadhadoop-c39 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28
>> 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>>
>> I also attach the build log of cyrus-sasl in the attachment.
>>
>> Thanks,
>> Quanlong
>>
>> At 2018-02-15 00:55:51, "Laszlo Gaal"  wrote:
>> >My build has progressed past the cyrus-sasl phase with no errors, so I
>> >don't have a repro case handy.
>> >My build env is:
>> >
>> >laszlog@laszlog1:~/native-toolchain$ lsb_release -a
>> >No LSB modules are available.
>> >Distributor ID: Ubuntu
>> >Description: Ubuntu 14.04.5 LTS
>> >Release: 14.04
>> >Codename: trusty
>> >laszlog@laszlog1:~/native-toolchain$ uname -a
>> >Linux laszlog1 4.4.0-112-generic #135~14.04.1-Ubuntu SMP Tue Jan 23
>> >20:41:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>> >
>> >
>> >OTOH grepping in cyrus-sasl finds the following:
>> >
>> >laszlog@laszlog1:~/native-toolchain/source/cyrus-sasl/cyrus-sasl-2.1.23$
>> >find . -name "*.c" -exec fgrep -n 'NULL, DB_HASH, 

Re: IMPALA-3343: impala-shell compatibility with Python 3

[Philip Zeyliger]

I'm not aware of anyone having tried it, so I don't think we know. I'd also
check in on Thrift's support for Python3. That seems like the most likely
library. You can certainly give removing print a shot!

-- Philip

On Wed, Feb 14, 2018 at 11:50 AM, Diggs, Asoka 
wrote:

> I have just encountered this error myself after my project made the
> decision to switch from Python 2 to 3.
>
> I’ve added a comment to the JIRA with some additional details from my own
> encounter with the error.
>
>
> Looking at the impala-shell.py file, it’s clear that the error I
> encountered – print statements formulated using Python 2 syntax that is not
> compatible with Python 3 – exist throughout the code.  If this is the
> extent of the incompatibility, then that’s a problem I am ready and willing
> to tackle.
>
> Is there reason to believe that this problem is as simple as changing
> print statement syntax, or reason to believe it is significantly more than
> that?
>
> Thanks for the insight.
> Asoka Diggs
>
>
>
>

Re: Re: Re: Build fail in native-toolchain project

[Tim Armstrong]

There are. I tried to document the list at some point here:
https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala

I believe that covers most of them, but it may be slightly stale. Please
feel free to update the list or let us know if something is missing.

On Wed, Feb 14, 2018 at 1:36 PM, Quanlong Huang 
wrote:

> Are there any pre-requirements for native-toolchain? Maybe I need to
> install something first.
> I just check out the master branch and run buildall.sh. Here's more
> info in my env:
>
> $ lsb_release -a
>
> No LSB modules are available.
>
> Distributor ID: Ubuntu
>
> Description:Ubuntu 14.04.2 LTS
>
> Release:14.04
>
> Codename:   trusty
>
> $ cat /etc/os-release
>
> NAME="Ubuntu"
>
> VERSION="14.04.2 LTS, Trusty Tahr"
>
> ID=ubuntu
>
> ID_LIKE=debian
>
> PRETTY_NAME="Ubuntu 14.04.2 LTS"
>
> VERSION_ID="14.04"
>
> HOME_URL="http://www.ubuntu.com/;
>
> SUPPORT_URL="http://help.ubuntu.com/;
>
> BUG_REPORT_URL="
> http://bugs.launchpad.net/ubuntu/;
>
> $ uname -a
>
> Linux iadhadoop-c39 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28
> 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>
> I also attach the build log of cyrus-sasl in the attachment.
>
> Thanks,
> Quanlong
>
> At 2018-02-15 00:55:51, "Laszlo Gaal"  wrote:
> >My build has progressed past the cyrus-sasl phase with no errors, so I
> >don't have a repro case handy.
> >My build env is:
> >
> >laszlog@laszlog1:~/native-toolchain$ lsb_release -a
> >No LSB modules are available.
> >Distributor ID: Ubuntu
> >Description: Ubuntu 14.04.5 LTS
> >Release: 14.04
> >Codename: trusty
> >laszlog@laszlog1:~/native-toolchain$ uname -a
> >Linux laszlog1 4.4.0-112-generic #135~14.04.1-Ubuntu SMP Tue Jan 23
> >20:41:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> >OTOH grepping in cyrus-sasl finds the following:
> >
> >laszlog@laszlog1:~/native-toolchain/source/cyrus-sasl/cyrus-sasl-2.1.23$
> >find . -name "*.c" -exec fgrep -n 'NULL, DB_HASH, flags, 0660' {} \;
> >104: ret = (*mbdb)->open(*mbdb, NULL, path, NULL, DB_HASH, flags, 0660);
> >106: ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
> >
> >
> >
> >(this is in *sasldb/db_berkeley.c*, starting on *line 103*):
> >
> >#if DB_VERSION_MAJOR == 4 && DB_VERSION_MINOR >= 1
> >ret = (*mbdb)->open(*mbdb, NULL, path, NULL, DB_HASH, flags, 0660);
> >#else
> >ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
> >#endif
> >
> >
> >so DB_VERSION_MAJOR may have a wrong value in your build -- but admittedly
> >I don't know what could influence that value.
> >
> >Could you post the command line you used to start the build, to see if
> >there could have been extra settings impacting the build?
> >
> >On Wed, Feb 14, 2018 at 3:06 PM, Quanlong Huang 
> >wrote:
> >
> >> Thank you, Laszlo! I build the master branch.
> >>
> >>
> >> At 2018-02-14 21:47:48, "Laszlo Gaal"  wrote:
> >> >Hey Quanlong,
> >> >
> >> >I have made a couple of minor changes, focusing on how our (Cloudera's)
> >> >internal automation drives the toolchain build.
> >> >I did not expect these changes to break you; I'll look into it shortly;
> >> >I'll keep you updated.
> >> >
> >> >Which branch dod you try to build? Was is "master"?
> >> >
> >> >Thanks,
> >> >
> >> >- Laszlo
> >> >
> >> >On Wed, Feb 14, 2018 at 1:03 PM, Quanlong Huang 
> >> >wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >>
> >> >> The build of cyrus-sasl-2.1.23 failed in my env. The errors are all in
> >> the
> >> >> db_berkeley.c:
> >> >>
> >> >>
> >> >>
> >> >> db_berkeley.c:106:8: error: too few arguments to function
> >> '(*mbdb)->open'
> >> >>
> >> >>   ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
> >> >>
> >> >> ^
> >> >>
> >> >> make[2]: *** [db_berkeley.lo] Error 1
> >> >>
> >> >>
> >> >>
> >> >> I'm using Ubuntu 14.04.2. It used to build success months ago. Are there
> >> >> any changes relate to this?
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Quanlong
> >>
>
>

Re:Re: Re: Build fail in native-toolchain project

[Quanlong Huang]

Are there any pre-requirements for native-toolchain? Maybe I need to install 
something first.
I just check out the master branch and run buildall.sh. Here's more info in my 
env:



$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description:Ubuntu 14.04.2 LTS

Release:14.04

Codename:   trusty

$ cat /etc/os-release 

NAME="Ubuntu"

VERSION="14.04.2 LTS, Trusty Tahr"

ID=ubuntu

ID_LIKE=debian

PRETTY_NAME="Ubuntu 14.04.2 LTS"

VERSION_ID="14.04"

HOME_URL="http://www.ubuntu.com/;

SUPPORT_URL="http://help.ubuntu.com/;

BUG_REPORT_URL="

http://bugs.launchpad.net/ubuntu/;

$ uname -a

Linux iadhadoop-c39 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 
19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux



I also attach the build log of cyrus-sasl in the attachment.
Thanks,
Quanlong


At 2018-02-15 00:55:51, "Laszlo Gaal"  wrote:
>My build has progressed past the cyrus-sasl phase with no errors, so I
>don't have a repro case handy.
>My build env is:
>
>laszlog@laszlog1:~/native-toolchain$ lsb_release -a
>No LSB modules are available.
>Distributor ID: Ubuntu
>Description: Ubuntu 14.04.5 LTS
>Release: 14.04
>Codename: trusty
>laszlog@laszlog1:~/native-toolchain$ uname -a
>Linux laszlog1 4.4.0-112-generic #135~14.04.1-Ubuntu SMP Tue Jan 23
>20:41:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>
>
>OTOH grepping in cyrus-sasl finds the following:
>
>laszlog@laszlog1:~/native-toolchain/source/cyrus-sasl/cyrus-sasl-2.1.23$
>find . -name "*.c" -exec fgrep -n 'NULL, DB_HASH, flags, 0660' {} \;
>104: ret = (*mbdb)->open(*mbdb, NULL, path, NULL, DB_HASH, flags, 0660);
>106: ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
>
>
>
>(this is in *sasldb/db_berkeley.c*, starting on *line 103*):
>
>#if DB_VERSION_MAJOR == 4 && DB_VERSION_MINOR >= 1
>ret = (*mbdb)->open(*mbdb, NULL, path, NULL, DB_HASH, flags, 0660);
>#else
>ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
>#endif
>
>
>so DB_VERSION_MAJOR may have a wrong value in your build -- but admittedly
>I don't know what could influence that value.
>
>Could you post the command line you used to start the build, to see if
>there could have been extra settings impacting the build?
>
>On Wed, Feb 14, 2018 at 3:06 PM, Quanlong Huang 
>wrote:
>
>> Thank you, Laszlo! I build the master branch.
>>
>>
>> At 2018-02-14 21:47:48, "Laszlo Gaal"  wrote:
>> >Hey Quanlong,
>> >
>> >I have made a couple of minor changes, focusing on how our (Cloudera's)
>> >internal automation drives the toolchain build.
>> >I did not expect these changes to break you; I'll look into it shortly;
>> >I'll keep you updated.
>> >
>> >Which branch dod you try to build? Was is "master"?
>> >
>> >Thanks,
>> >
>> >- Laszlo
>> >
>> >On Wed, Feb 14, 2018 at 1:03 PM, Quanlong Huang 
>> >wrote:
>> >
>> >> Hi all,
>> >>
>> >>
>> >> The build of cyrus-sasl-2.1.23 failed in my env. The errors are all in
>> the
>> >> db_berkeley.c:
>> >>
>> >>
>> >>
>> >> db_berkeley.c:106:8: error: too few arguments to function
>> '(*mbdb)->open'
>> >>
>> >>   ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
>> >>
>> >> ^
>> >>
>> >> make[2]: *** [db_berkeley.lo] Error 1
>> >>
>> >>
>> >>
>> >> I'm using Ubuntu 14.04.2. It used to build success months ago. Are there
>> >> any changes relate to this?
>> >>
>> >>
>> >> Thanks,
>> >> Quanlong
>>
configure: creating cache ./config.cache
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking for gcc... 
/mnt/volume1/impala-orc/tmp/native-toolchain/build/gcc-4.9.2/bin/gcc
checking for C compiler default output... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether 
/mnt/volume1/impala-orc/tmp/native-toolchain/build/gcc-4.9.2/bin/gcc accepts 
-g... yes
checking for 
/mnt/volume1/impala-orc/tmp/native-toolchain/build/gcc-4.9.2/bin/gcc option to 
accept ANSI C... none needed
checking for style of include used by make... GNU
checking dependency style of 
/mnt/volume1/impala-orc/tmp/native-toolchain/build/gcc-4.9.2/bin/gcc... gcc3
checking how to run the C preprocessor... 
/mnt/volume1/impala-orc/tmp/native-toolchain/build/gcc-4.9.2/bin/gcc -E
checking for gawk... (cached) mawk
checking whether ln -s works... yes
checking whether make sets $(MAKE)... (cached) yes
checking for a BSD-compatible install... /usr/bin/install -c
checking for __attribute__... no
checking for runpath 

Re: Help with fixing clang-diagnostic-over-aligned

[Todd Lipcon]

This probably means that QueryExecMgr (or one of its descendent members) is
annotated with __attribute__((aligned(64)), perhaps through something like
a CACHELINE_ALIGNED macro.

The issue is that, as the warning says, normal libc malloc doesn't make any
hard guarantees about the alignment of allocations, and the way that
'operator new' is implemented by default only gets the size and not the
desired allocation. I believe C++17 is adding support for an 'operator new'
overload that also passes the desired alignment, so that it can call down
into posix_memalign and ensure the appropriate allocation.

The "correct" fix prior to C++17 is probably to do something like overload
'operator new' for the type, or to use posix_memalign manually to allocate
aligned memory, and then placement-new it into that memory.

Another reasonable fix would probably be to add a CHECK in the constructor
that 'this' is aligned as desired so that, if you ever run on an allocator
that isn't guaranteeing what you think it is, you'll know immediately
rather than getting a surprising perf or atomicity regression due to
cross-cacheline loads/stores.

-Todd


On Wed, Feb 14, 2018 at 12:33 PM, Sailesh Mukil 
wrote:

> Does anyone have experience with fixing the following type of
> clang-tidy warning?
>
>
> /home/ubuntu/Impala/be/src/runtime/exec-env.cc:159:21: warning: type
> 'impala::QueryExecMgr' requires 64 bytes of alignment and the default
> allocator only guarantees 16 bytes [clang-diagnostic-over-aligned]
> query_exec_mgr_(new QueryExecMgr()),
>
>
> /home/ubuntu/Impala/be/src/service/impalad-main.cc:80:49: warning:
> type 'impala::ImpalaServer' requires 64 bytes of alignment and the
> default allocator only guarantees 16 bytes
> [clang-diagnostic-over-aligned]
>   boost::shared_ptr impala_server(new
> ImpalaServer(_env));
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Help with fixing clang-diagnostic-over-aligned

[Daniel Hecht]

Anything that has an aligned field will also need to be aligned (otherwise
the field alignment won't be respected).  You can do it by inheriting from
CacheLinedAlign which includes an operator new (see comment in
aligned-new.h).

On Wed, Feb 14, 2018 at 12:33 PM, Sailesh Mukil 
wrote:

> Does anyone have experience with fixing the following type of
> clang-tidy warning?
>
>
> /home/ubuntu/Impala/be/src/runtime/exec-env.cc:159:21: warning: type
> 'impala::QueryExecMgr' requires 64 bytes of alignment and the default
> allocator only guarantees 16 bytes [clang-diagnostic-over-aligned]
> query_exec_mgr_(new QueryExecMgr()),
>
>
> /home/ubuntu/Impala/be/src/service/impalad-main.cc:80:49: warning:
> type 'impala::ImpalaServer' requires 64 bytes of alignment and the
> default allocator only guarantees 16 bytes
> [clang-diagnostic-over-aligned]
>   boost::shared_ptr impala_server(new
> ImpalaServer(_env));
>

Help with fixing clang-diagnostic-over-aligned

[Sailesh Mukil]

Does anyone have experience with fixing the following type of
clang-tidy warning?


/home/ubuntu/Impala/be/src/runtime/exec-env.cc:159:21: warning: type
'impala::QueryExecMgr' requires 64 bytes of alignment and the default
allocator only guarantees 16 bytes [clang-diagnostic-over-aligned]
query_exec_mgr_(new QueryExecMgr()),


/home/ubuntu/Impala/be/src/service/impalad-main.cc:80:49: warning:
type 'impala::ImpalaServer' requires 64 bytes of alignment and the
default allocator only guarantees 16 bytes
[clang-diagnostic-over-aligned]
  boost::shared_ptr impala_server(new ImpalaServer(_env));

IMPALA-3343: impala-shell compatibility with Python 3

[Diggs, Asoka]

I have just encountered this error myself after my project made the decision to 
switch from Python 2 to 3.

I’ve added a comment to the JIRA with some additional details from my own 
encounter with the error.


Looking at the impala-shell.py file, it’s clear that the error I encountered – 
print statements formulated using Python 2 syntax that is not compatible with 
Python 3 – exist throughout the code.  If this is the extent of the 
incompatibility, then that’s a problem I am ready and willing to tackle.

Is there reason to believe that this problem is as simple as changing print 
statement syntax, or reason to believe it is significantly more than that?

Thanks for the insight.
Asoka Diggs

Re: Jenkins.impala.io back up

[Lars Volker]

Thank you for applying the updates, Zach!

On Wed, Feb 14, 2018 at 11:17 AM, Zachary Amsden 
wrote:

> Everything went as planned and Jenkins is now upgraded.
>
>  - Zach
>

Jenkins.impala.io back up

[Zachary Amsden]

Everything went as planned and Jenkins is now upgraded.

 - Zach

Heads Up - Jenkins shut down.

[Zachary Amsden]

I am going to upgrade jenkins.impala.io to address a security advisory; no
GVOs will run for a bit.

Hopefully I should be done in 30 minutes or less.

 - Zach

Re: Re: Build fail in native-toolchain project

[Laszlo Gaal]

My build has progressed past the cyrus-sasl phase with no errors, so I
don't have a repro case handy.
My build env is:

laszlog@laszlog1:~/native-toolchain$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
laszlog@laszlog1:~/native-toolchain$ uname -a
Linux laszlog1 4.4.0-112-generic #135~14.04.1-Ubuntu SMP Tue Jan 23
20:41:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux


OTOH grepping in cyrus-sasl finds the following:

laszlog@laszlog1:~/native-toolchain/source/cyrus-sasl/cyrus-sasl-2.1.23$
find . -name "*.c" -exec fgrep -n 'NULL, DB_HASH, flags, 0660' {} \;
104: ret = (*mbdb)->open(*mbdb, NULL, path, NULL, DB_HASH, flags, 0660);
106: ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);



(this is in *sasldb/db_berkeley.c*, starting on *line 103*):

#if DB_VERSION_MAJOR == 4 && DB_VERSION_MINOR >= 1
ret = (*mbdb)->open(*mbdb, NULL, path, NULL, DB_HASH, flags, 0660);
#else
ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
#endif


so DB_VERSION_MAJOR may have a wrong value in your build -- but admittedly
I don't know what could influence that value.

Could you post the command line you used to start the build, to see if
there could have been extra settings impacting the build?

On Wed, Feb 14, 2018 at 3:06 PM, Quanlong Huang 
wrote:

> Thank you, Laszlo! I build the master branch.
>
>
> At 2018-02-14 21:47:48, "Laszlo Gaal"  wrote:
> >Hey Quanlong,
> >
> >I have made a couple of minor changes, focusing on how our (Cloudera's)
> >internal automation drives the toolchain build.
> >I did not expect these changes to break you; I'll look into it shortly;
> >I'll keep you updated.
> >
> >Which branch dod you try to build? Was is "master"?
> >
> >Thanks,
> >
> >- Laszlo
> >
> >On Wed, Feb 14, 2018 at 1:03 PM, Quanlong Huang 
> >wrote:
> >
> >> Hi all,
> >>
> >>
> >> The build of cyrus-sasl-2.1.23 failed in my env. The errors are all in
> the
> >> db_berkeley.c:
> >>
> >>
> >>
> >> db_berkeley.c:106:8: error: too few arguments to function
> '(*mbdb)->open'
> >>
> >>   ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
> >>
> >> ^
> >>
> >> make[2]: *** [db_berkeley.lo] Error 1
> >>
> >>
> >>
> >> I'm using Ubuntu 14.04.2. It used to build success months ago. Are there
> >> any changes relate to this?
> >>
> >>
> >> Thanks,
> >> Quanlong
>

Re:Re: Build fail in native-toolchain project

[Quanlong Huang]

Thank you, Laszlo! I build the master branch.


At 2018-02-14 21:47:48, "Laszlo Gaal"  wrote:
>Hey Quanlong,
>
>I have made a couple of minor changes, focusing on how our (Cloudera's)
>internal automation drives the toolchain build.
>I did not expect these changes to break you; I'll look into it shortly;
>I'll keep you updated.
>
>Which branch dod you try to build? Was is "master"?
>
>Thanks,
>
>- Laszlo
>
>On Wed, Feb 14, 2018 at 1:03 PM, Quanlong Huang 
>wrote:
>
>> Hi all,
>>
>>
>> The build of cyrus-sasl-2.1.23 failed in my env. The errors are all in the
>> db_berkeley.c:
>>
>>
>>
>> db_berkeley.c:106:8: error: too few arguments to function '(*mbdb)->open'
>>
>>   ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
>>
>> ^
>>
>> make[2]: *** [db_berkeley.lo] Error 1
>>
>>
>>
>> I'm using Ubuntu 14.04.2. It used to build success months ago. Are there
>> any changes relate to this?
>>
>>
>> Thanks,
>> Quanlong

Re: Build fail in native-toolchain project

[Laszlo Gaal]

Hey Quanlong,

I have made a couple of minor changes, focusing on how our (Cloudera's)
internal automation drives the toolchain build.
I did not expect these changes to break you; I'll look into it shortly;
I'll keep you updated.

Which branch dod you try to build? Was is "master"?

Thanks,

- Laszlo

On Wed, Feb 14, 2018 at 1:03 PM, Quanlong Huang 
wrote:

> Hi all,
>
>
> The build of cyrus-sasl-2.1.23 failed in my env. The errors are all in the
> db_berkeley.c:
>
>
>
> db_berkeley.c:106:8: error: too few arguments to function '(*mbdb)->open'
>
>   ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
>
> ^
>
> make[2]: *** [db_berkeley.lo] Error 1
>
>
>
> I'm using Ubuntu 14.04.2. It used to build success months ago. Are there
> any changes relate to this?
>
>
> Thanks,
> Quanlong