Re: Re: Re: Build fail in native-toolchain project

[Laszlo Gaal]

I found the difference between your build and mine, which may be
significant. My  hint was that your build tried to compile the db_berkeley
module in sasldb when it errored out, but my build compiled something
called db_none at the corresponding location. Digging into the "configure"
phase (at the start of the log you posted) your build logged:

checking db.h usability... yes
checking db.h presence... yes
checking for db.h... yes
checking DB library to use... berkeley
checking if Berkeley DB handle is kept open in SASLDB... no


my corresponding log is:

checking db.h usability... no

checking db.h presence... no

checking for db.h... no

checking ndbm.h usability... no

checking ndbm.h presence... no

checking for ndbm.h... no

checking gdbm.h usability... no

checking gdbm.h presence... no

checking for gdbm.h... no

checking DB library to use... no

configure: WARNING: Disabling SASL authentication database support

checking if Berkeley DB handle is kept open in SASLDB... no

so, apparently the configure phase on your system managed to find
BerkeleyDB whereas during my build it was not found, which explains the
difference between the "configure" phases.

Checking source/cyrus_sasl/build.sh supplies another interesting detail:

  CONFIGURE_FLAGS=

  # If libdb5 is installed, it would be used by default and would lead to a
failure.

  # If libdb4 appears to be installed, use that instead. The location below
was found

  # on a CentOS 7 machine.

  if [[ -e /usr/include/libdb4 && -e /usr/lib64/libdb4 ]]; then

CONFIGURE_FLAGS+=" --with-bdb-incdir=/usr/include/libdb4"

CONFIGURE_FLAGS+=" --with-bdb-libdir=/usr/lib64/libdb4"

  fi

Recalling the neighborhood of the offending line from my previous message:

(this is in *sasldb/db_berkeley.c*, starting on *line 103*):

#if DB_VERSION_MAJOR == 4 && DB_VERSION_MINOR >= 1
ret = (*mbdb)->open(*mbdb, NULL, path, NULL, DB_HASH, flags, 0660);
#else
ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
#endif

to me it looks like you have BerkeleyDB v5 installed on your system, and it
breaks the cyrus_sasl build.
I'm at a bit of a loss here, as this has never happened to me, but my first
suggestion would be to remove the BerkeleyDB package (or at least the dev
package, to get rid of the header files) if you are not using it. If you
need it, then you may have some success tweaking
native-toolchain/source/cyrus-sasl/build.sh at the location listed above.

Please let us know if you managed to overcome this obstacle; looks like not
a lot of people have encountered this particular problem in the near past.

Good luck,

  - Laszlo

PS: Attaching my build log of cyrus-sasl for added context (this may be
removed by the apache.org list processor though):




On Wed, Feb 14, 2018 at 10:44 PM, Tim Armstrong 
wrote:

> There are. I tried to document the list at some point here:
> https://cwiki.apache.org/confluence/display/IMPALA/
> Building+native-toolchain+from+scratch+and+using+with+Impala
>
> I believe that covers most of them, but it may be slightly stale. Please
> feel free to update the list or let us know if something is missing.
>
> On Wed, Feb 14, 2018 at 1:36 PM, Quanlong Huang 
> wrote:
>
>> Are there any pre-requirements for native-toolchain? Maybe I need to
>> install something first.
>> I just check out the master branch and run buildall.sh. Here's more
>> info in my env:
>>
>> $ lsb_release -a
>>
>> No LSB modules are available.
>>
>> Distributor ID: Ubuntu
>>
>> Description:Ubuntu 14.04.2 LTS
>>
>> Release:14.04
>>
>> Codename:   trusty
>>
>> $ cat /etc/os-release
>>
>> NAME="Ubuntu"
>>
>> VERSION="14.04.2 LTS, Trusty Tahr"
>>
>> ID=ubuntu
>>
>> ID_LIKE=debian
>>
>> PRETTY_NAME="Ubuntu 14.04.2 LTS"
>>
>> VERSION_ID="14.04"
>>
>> HOME_URL="http://www.ubuntu.com/;
>>
>> SUPPORT_URL="http://help.ubuntu.com/;
>>
>> BUG_REPORT_URL="
>> http://bugs.launchpad.net/ubuntu/;
>>
>> $ uname -a
>>
>> Linux iadhadoop-c39 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28
>> 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>>
>> I also attach the build log of cyrus-sasl in the attachment.
>>
>> Thanks,
>> Quanlong
>>
>> At 2018-02-15 00:55:51, "Laszlo Gaal"  wrote:
>> >My build has progressed past the cyrus-sasl phase with no errors, so I
>> >don't have a repro case handy.
>> >My build env is:
>> >
>> >laszlog@laszlog1:~/native-toolchain$ lsb_release -a
>> >No LSB modules are available.
>> >Distributor ID: Ubuntu
>> >Description: Ubuntu 14.04.5 LTS
>> >Release: 14.04
>> >Codename: trusty
>> >laszlog@laszlog1:~/native-toolchain$ uname -a
>> >Linux laszlog1 4.4.0-112-generic #135~14.04.1-Ubuntu SMP Tue Jan 23
>> >20:41:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>> >
>> >
>> >OTOH grepping in cyrus-sasl finds the following:
>> >
>> >laszlog@laszlog1:~/native-toolchain/source/cyrus-sasl/cyrus-sasl-2.1.23$
>> >find . -name "*.c" -exec fgrep -n 'NULL, DB_HASH, 

Re: Re: Re: Build fail in native-toolchain project

[Tim Armstrong]

There are. I tried to document the list at some point here:
https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala

I believe that covers most of them, but it may be slightly stale. Please
feel free to update the list or let us know if something is missing.

On Wed, Feb 14, 2018 at 1:36 PM, Quanlong Huang 
wrote:

> Are there any pre-requirements for native-toolchain? Maybe I need to
> install something first.
> I just check out the master branch and run buildall.sh. Here's more
> info in my env:
>
> $ lsb_release -a
>
> No LSB modules are available.
>
> Distributor ID: Ubuntu
>
> Description:Ubuntu 14.04.2 LTS
>
> Release:14.04
>
> Codename:   trusty
>
> $ cat /etc/os-release
>
> NAME="Ubuntu"
>
> VERSION="14.04.2 LTS, Trusty Tahr"
>
> ID=ubuntu
>
> ID_LIKE=debian
>
> PRETTY_NAME="Ubuntu 14.04.2 LTS"
>
> VERSION_ID="14.04"
>
> HOME_URL="http://www.ubuntu.com/;
>
> SUPPORT_URL="http://help.ubuntu.com/;
>
> BUG_REPORT_URL="
> http://bugs.launchpad.net/ubuntu/;
>
> $ uname -a
>
> Linux iadhadoop-c39 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28
> 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>
> I also attach the build log of cyrus-sasl in the attachment.
>
> Thanks,
> Quanlong
>
> At 2018-02-15 00:55:51, "Laszlo Gaal"  wrote:
> >My build has progressed past the cyrus-sasl phase with no errors, so I
> >don't have a repro case handy.
> >My build env is:
> >
> >laszlog@laszlog1:~/native-toolchain$ lsb_release -a
> >No LSB modules are available.
> >Distributor ID: Ubuntu
> >Description: Ubuntu 14.04.5 LTS
> >Release: 14.04
> >Codename: trusty
> >laszlog@laszlog1:~/native-toolchain$ uname -a
> >Linux laszlog1 4.4.0-112-generic #135~14.04.1-Ubuntu SMP Tue Jan 23
> >20:41:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> >OTOH grepping in cyrus-sasl finds the following:
> >
> >laszlog@laszlog1:~/native-toolchain/source/cyrus-sasl/cyrus-sasl-2.1.23$
> >find . -name "*.c" -exec fgrep -n 'NULL, DB_HASH, flags, 0660' {} \;
> >104: ret = (*mbdb)->open(*mbdb, NULL, path, NULL, DB_HASH, flags, 0660);
> >106: ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
> >
> >
> >
> >(this is in *sasldb/db_berkeley.c*, starting on *line 103*):
> >
> >#if DB_VERSION_MAJOR == 4 && DB_VERSION_MINOR >= 1
> >ret = (*mbdb)->open(*mbdb, NULL, path, NULL, DB_HASH, flags, 0660);
> >#else
> >ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
> >#endif
> >
> >
> >so DB_VERSION_MAJOR may have a wrong value in your build -- but admittedly
> >I don't know what could influence that value.
> >
> >Could you post the command line you used to start the build, to see if
> >there could have been extra settings impacting the build?
> >
> >On Wed, Feb 14, 2018 at 3:06 PM, Quanlong Huang 
> >wrote:
> >
> >> Thank you, Laszlo! I build the master branch.
> >>
> >>
> >> At 2018-02-14 21:47:48, "Laszlo Gaal"  wrote:
> >> >Hey Quanlong,
> >> >
> >> >I have made a couple of minor changes, focusing on how our (Cloudera's)
> >> >internal automation drives the toolchain build.
> >> >I did not expect these changes to break you; I'll look into it shortly;
> >> >I'll keep you updated.
> >> >
> >> >Which branch dod you try to build? Was is "master"?
> >> >
> >> >Thanks,
> >> >
> >> >- Laszlo
> >> >
> >> >On Wed, Feb 14, 2018 at 1:03 PM, Quanlong Huang 
> >> >wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >>
> >> >> The build of cyrus-sasl-2.1.23 failed in my env. The errors are all in
> >> the
> >> >> db_berkeley.c:
> >> >>
> >> >>
> >> >>
> >> >> db_berkeley.c:106:8: error: too few arguments to function
> >> '(*mbdb)->open'
> >> >>
> >> >>   ret = (*mbdb)->open(*mbdb, path, NULL, DB_HASH, flags, 0660);
> >> >>
> >> >> ^
> >> >>
> >> >> make[2]: *** [db_berkeley.lo] Error 1
> >> >>
> >> >>
> >> >>
> >> >> I'm using Ubuntu 14.04.2. It used to build success months ago. Are there
> >> >> any changes relate to this?
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Quanlong
> >>
>
>