Re: jenkins.impala.io pre-existing workspace

[Laszlo Gaal]

The layout of #6 has served me well for a year now.
It allows me to keep several such workspaces around
to work with different Impala versions, I just have
to be control carefully which workspace the minicluster
runs from -- but that's not an issue here.

On Thu, Aug 31, 2017 at 2:44 AM, Philip Zeyliger 
wrote:

> On Wed, Aug 30, 2017 at 4:45 PM, Jim Apple  wrote:
>
> > The workspace cleanup isn't working - see the last bit of any recent
> > ub1604 job: https://jenkins.impala.io/view/Utility/job/ubuntu-16.04-
> > from-scratch/206/console
> >
> > 03:56:40.920 [WS-CLEANUP] Deleting project workspace...Cannot delete
> > workspace :remote file operation failed: /home/ubuntu at
> > hudson.remoting.Channel@4384d5b9:ubuntu-16.04 (i-032d527b9c801df4c):
> > java.io.IOException: Unable to delete '/home/ubuntu'. Tried 3 times
> > (of a maximum of 3) waiting 0.1 sec between attempts.
> > 03:56:48.161 ERROR: Step ‘Delete workspace when build is done’ failed:
> > Cannot delete workspace: remote file operation failed: /home/ubuntu at
> > hudson.remoting.Channel@4384d5b9:ubuntu-16.04 (i-032d527b9c801df4c):
> > java.io.IOException: Unable to delete '/home/ubuntu'. Tried 3 times
> > (of a maximum of 3) waiting 0.1 sec between attempts.
> >
> > The workspace is $HOME, so you can't just delete it without being root.
> >
> > This could be changed to
> >
> > 1. A post-build script to "rm -rf ~/*". This doesn't reset everything,
> > though - the job makes changes to other parts of the filesystem.
> >
> > 2. A post-build script to "sudo shutdown -h now" to make sure ec2
> > instances are not re-used. I'm not sure how Jenkins would feel about
> > this. :-)
> >
> > 3. A post-build script to move $HOME to some archived location on the
> > disk, to preserve debuggability.
> >
> > 4. A bash trap in the script to do one of the above.
> >
> > 5. Run the whole thing in a docker in the build machine, then delete
> > the container when the script is done. Or don't, if there's enough
> > disk space to not worry about that.
> >
> > 6. Do all of the work in a workspace inside $HOME. This would require
> > some changes to bootstrap_development.sh.
> >
> > #5 is the most hermetic, I'd guess.
> >
>
> I like #6 in the short term: I don't think anything is too bound to $HOME
> except a few "~/" in that script which are easily approached. If you do #5,
> you have to teach Jenkins about how to save the build output logs from
> inside of Docker, which is, I think, more work.
>
> -- Philip
>
>
> >
> > On Thu, Aug 24, 2017 at 8:29 AM, Michael Brown 
> wrote:
> > > Looks like someone has done this.
> > >
> > > On Wed, Aug 23, 2017 at 8:16 PM, Alexander Behm <
> alex.b...@cloudera.com>
> > > wrote:
> > >
> > >> Yes, let's please add the post-build action for sanity and consistency
> > with
> > >> our other jobs.
> > >>
> > >> On Wed, Aug 23, 2017 at 7:42 PM, Tim Armstrong <
> tarmstr...@cloudera.com
> > >
> > >> wrote:
> > >>
> > >> > Maybe the workspace just got left in a weird state - I think in most
> > >> cases
> > >> > "git init" followed by checking out a branch and doing a clean would
> > >> work.
> > >> >
> > >> > Should we add the delete workspace post-build action?
> > >> >
> > >> > On Wed, Aug 23, 2017 at 5:32 PM, Michael Brown 
> > >> wrote:
> > >> >
> > >> > > Not a known issue. I noticed ubuntu-16.04-from-scratch is not set
> to
> > >> > clean
> > >> > > up its workspace, and its config has not been touched since Aug
> 11.
> > It
> > >> > > seems strange we only saw this now
> > >> > >
> > >> > > On Wed, Aug 23, 2017 at 5:25 PM, Tim Armstrong <
> > >> tarmstr...@cloudera.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Is this a known problem? My job failed because the Impala repo
> > >> already
> > >> > > > existed on the machine:
> > >> > > >
> > >> > > > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/164/
> > >> > > >
> > >> > > > *23:00:24* + /usr/bin/git init /home/ubuntu/Impala*23:00:24*
> > >> > > > Reinitialized existing Git repository in
> /home/ubuntu/Impala/.git/
> > >> > > > 
> > >> > > > *23:02:18* + for ITER in '$(seq 1 10)'*23:02:18* + echo
> 'ATTEMPT:
> > >> > > > 1'*23:02:18* ATTEMPT: 1*23:02:18* + /usr/bin/git checkout
> > >> > > > FETCH_HEAD*23:02:18* + cat
> > >> > > > /home/ubuntu/Impala/tmp.3tYBn0GUga*23:02:18* 23:02:18.712300
> > >> git.c:344
> > >> > > >   trace: built-in: git 'checkout'
> > 'FETCH_HEAD'*23:02:18*
> > >> > > > error: The following untracked working tree files would be
> > >> overwritten
> > >> > > > by checkout:*23:02:18*  .clang-format*23:02:18*
> > >> > > >  .clang-tidy*23:02:18*
> > >> > > > .gitignore*23:02:18*CMakeLists.txt*23:02:18*
> > >> > > > DISCLAIMER*23:02:18*
> > >> > > > EXPORT_CONTROL.md*23:02:18* LICENSE.txt*23:02:18*
> > >> > > >  LOGS.md*23:02:18*
> > >> > > > NOTICE.txt*23:02:18*README.md*23:02:18*
> > >> > > >  be/.gitignore*23:02:18*
> 

Re: GVO machines dependencies

[Laszlo Gaal]

Thanks a lot for fixing this, Sailesh, this has been bugging me for a while
when spinning up cloud-based workstations.

On Thu, May 25, 2017 at 6:18 PM, Jim Apple  wrote:

> Nice find, Sailesh!
>
> On Wed, May 24, 2017 at 6:39 PM, Sailesh Mukil 
> wrote:
> > The issue was that there was a bug in the install.sh script from the
> > impala-setup/ repo that always assumed that the repo would reside in
> > /home/$user.
> >
> > The Jenkins AMIs seem to have had a very old checkout (8 months old) at
> > /home/ubuntu/ which was being used for the GVOs. The Jenkins job would
> make
> > a fresh checkout in /tmp/ and run install.sh from that checkout. But due
> to
> > the above bug, the script would change its sources to the old repo in
> > /home/ubuntu. So, the new dependencies added to the project didn't get
> > picked up. I've submitted a fix, and will redo the GVO once it's merged
> > into impala-setup.
> >
> > Thanks for your help Michael and Jim!
> >
> > On Wed, May 24, 2017 at 4:58 PM, Michael Brown 
> wrote:
> >
> >> "from-scratch" is a misnomer: only the first build to run on a given
> node
> >> is truly from-scratch, because running impala-setup takes effect
> >> system-wide, and is run once, when the node is created.
> >>
> >> For http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/1371/ ,
> >> the
> >> worker's build history suggests it has been up roughly 17 hours.
> >> http://jenkins.impala.io:8080/computer/ub1404-c4.4xl-gp2%20(
> >> i-0d76efbc8de26926c)/builds
> >> This means your recent change hasn't taken effect on "older" workers.
> >>
> >> Any new workers that get created should get the change.
> >>
> >>
> >>
> >>
> >>
> >> On Wed, May 24, 2017 at 4:34 PM, Sailesh Mukil 
> >> wrote:
> >>
> >> > Thanks Michael and Jim,
> >> >
> >> > It looks like adding to bin/bootstrap_build.sh makes the
> ub14-build-only
> >> > job work fine, however, updating the impala-setup still doesn't seem
> to
> >> > work. I submitted a pull request that was merged by Dimitris here:
> >> > https://github.com/awleblang/impala-setup/commits/master
> >> >
> >> > Also, the impala-setup chef logs don't seem to get printed in the
> jenkins
> >> > console output. So I'm not sure if it's actually being run or not.
> >> >
> >> > On Wed, May 24, 2017 at 3:37 PM, Michael Brown 
> >> wrote:
> >> >
> >> > > Thanks Jim; I had forgotten bin/bootstrap_build.sh.
> >> > >
> >> > > On Wed, May 24, 2017 at 3:14 PM, Jim Apple 
> >> wrote:
> >> > >
> >> > > > I'd recommend against #2 using the node setup to install Impala
> >> > > > dependencies. The only reason it installs openjdk is that Jenkins
> >> > > > needs Java to talk to it. All of the other Impala dependencies are
> >> > > > installed in the jobs themselves.
> >> > > >
> >> > > > On Wed, May 24, 2017 at 3:11 PM, Michael Brown <
> mi...@cloudera.com>
> >> > > wrote:
> >> > > > >> Is there a way to add a new dependency to these machines?
> >> > > > >
> >> > > > > Our builds use two worker labels:
> >> > > > > 1. ubuntu14.04-c4.4xlarge-gp2
> >> > > > > 2. ub14-build-only
> >> > > > >
> >> > > > > 1 uses https://github.com/awleblang/impala-setup, and I think
> you
> >> > > should
> >> > > > > add libffi to there. It's something you should do anyway so new
> >> users
> >> > > get
> >> > > > > set up with the right dependencies.
> >> > > > >
> >> > > > > 2 is a different story, but I see that at
> >> > > > > http://jenkins.impala.io:8080/configure that worker is already
> >> > > > installing a
> >> > > > > JDK via apt-get. You could use a similar pattern to install
> libffi.
> >> > > >
> >> > >
> >> >
> >>
>

Could someone please add me to the native-toolchain project?

[Laszlo Gaal]

I'm trying to proceed with IMPALA-4846 (upgrade Snappy to 1.1.4) but I've
just discovered that I can't set Verified +1.

user details:

Username
gaallas
Full Name Laszlo Gaal
Email Address laszlo.g...@cloudera.com

Thanks a lot in advance!

Re: Permission to resolve IMPALA jiras?

[Laszlo Gaal]

I'm sorry Bharath, I missed your message earlier; yes, it works now. Thanks
all who helped.

On Fri, Nov 4, 2016 at 6:47 PM, Laszlo Gaal <laszlo.g...@cloudera.com>
wrote:

> Thank you, Jim.
>
> On Fri, Nov 4, 2016 at 6:02 PM, Jim Apple <jbap...@cloudera.com> wrote:
>
>> I have forwarded this message to the JIRA administrators.
>>
>> On Fri, Nov 4, 2016 at 9:51 AM, Laszlo Gaal <laszlo.g...@cloudera.com>
>> wrote:
>> > I seem to lack permission to resolve Impala issues on
>> issues.cloudera.org.
>> > Could you please point me to the right person or mailbox to request
>> this?
>> >
>> > Thanks a lot,
>> >
>> >- LaszloG
>>
>
>

Re: Permission to resolve IMPALA jiras?

[Laszlo Gaal]

Thank you, Jim.

On Fri, Nov 4, 2016 at 6:02 PM, Jim Apple <jbap...@cloudera.com> wrote:

> I have forwarded this message to the JIRA administrators.
>
> On Fri, Nov 4, 2016 at 9:51 AM, Laszlo Gaal <laszlo.g...@cloudera.com>
> wrote:
> > I seem to lack permission to resolve Impala issues on
> issues.cloudera.org.
> > Could you please point me to the right person or mailbox to request this?
> >
> > Thanks a lot,
> >
> >- LaszloG
>

Permission to resolve IMPALA jiras?

[Laszlo Gaal]

I seem to lack permission to resolve Impala issues on issues.cloudera.org.
Could you please point me to the right person or mailbox to request this?

Thanks a lot,

   - LaszloG

Re: Bootstrapping a impala dev failed on fresh installed Ubuntu 14.04

[Laszlo Gaal]

Hi Amos,

I've encountered this malformed directoy name as well. The quick workaround
is to delete it and create the subdir with the correct name:
1. rm -rf  ${IMPALA_HOME}/tests/*RESUL*
2. mkdir ${IMPALA_HOME}/tests/results

This will create the correct subdirectory under tests.

Hope this helps,

- LaszloG

On Wed, Nov 2, 2016 at 2:02 PM, Amos Bird  wrote:

>
> Ah, re-login does the trick. Thanks for you help ;).
>
> However, the e2e test yells so many errors.
>
> 1) the name of the directory containing the error log is strange. It
>  literaly looks like this:
> tests/"${RESULTS_DIR}/TEST-impala-custom-cluster.log"
>
> 2) the commit I tested is 7fc31b534d4c5cb118c559e16556a6c1ae6ca7fc
>
> 3) when executing tests/run-tests.py, it gave:
> -
> Traceback (most recent call last):
>   File "./tests/run-tests.py", line 94, in 
> test_executor.run_tests(args)
>   File "./tests/run-tests.py", line 63, in run_tests
> exit_code = pytest.main(args)
>   File "/home/amos/impala/infra/python/env/local/lib/python2.
> 7/site-packages/_pytest/config.py", line 32, in main
> config = _prepareconfig(args, plugins)
>   File "/home/amos/impala/infra/python/env/local/lib/python2.
> 7/site-packages/_pytest/config.py", line 78, in _prepareconfig
> args = shlex.split(args)
>   File "/usr/lib/python2.7/shlex.py", line 279, in split
> return list(lex)
>   File "/usr/lib/python2.7/shlex.py", line 269, in next
> token = self.get_token()
>   File "/usr/lib/python2.7/shlex.py", line 96, in get_token
> raw = self.read_token()
>   File "/usr/lib/python2.7/shlex.py", line 172, in read_token
> raise ValueError, "No closing quotation"
> ValueError: No closing quotation
> -
>
> 4) when executing "MAX_PYTEST_FAILURES=12345678 ./bin/run-all-tests.sh",
> be, fe tests are passed. e2e tests fail a lot. Log files are attached.
>
> I'm refering to this https://cwiki.apache.org/
> confluence/display/IMPALA/How+to+load+and+run+Impala+tests
>
> regards,
> Amos
>
>
>
> Lars Volker writes:
>
> > Yes, this is already committed to the impala-setup repo and I used it
> > yesterday on a fresh Ubuntu 14.04 machine with success.
> >
> > Amos, after running impala-setup you will need to re-login to make sure
> the
> > changes made to the system limits are effective. You can check them by
> > running "ulimit -n" in your shell.
> >
> > On Wed, Nov 2, 2016 at 5:48 AM, Jim Apple  wrote:
> >
> >> Isn't that already part of the script?
> >>
> >> https://github.com/awleblang/impala-setup/commit/
> >> 56fa829c99e997585eb63fcd49cb65eb8357e679
> >>
> >> https://git-wip-us.apache.org/repos/asf?p=incubator-impala.
> >> git;a=blob;f=bin/bootstrap_development.sh;h=
> 8c4f742ae058f8017858d2a749e882
> >> 4be58bd410;hb=HEAD#l68
> >>
> >> On Tue, Nov 1, 2016 at 9:44 PM, Dimitris Tsirogiannis
> >>  wrote:
> >> > Hi Amos,
> >> >
> >> > You need to increase your limits (/etc/security/limits.conf) for max
> >> > number of open files (nofile). Use a pretty big number (e.g. 500K) for
> >> > both soft and hard.
> >> >
> >> > Hope that helps.
> >> >
> >> > Dimitris
> >> >
> >> > On Tue, Nov 1, 2016 at 8:57 PM, Amos Bird  wrote:
> >> >>
> >> >> Hi there,
> >> >>
> >> >> After days of efforts to make impala's local tests work on my Centos
> >> >> machine, I finally gave up and turns to Ubuntu. I followed this
> simple
> >> >> guide
> >> >> https://cwiki.apache.org/confluence/display/IMPALA/
> >> Bootstrapping+an+Impala+Development+Environment+From+Scratch
> >> >> on a fresh installed Ubuntu 14.04. Unfortunately there are still
> errors
> >> >> in loading data phase. Here is the error log,
> >> >>
> >> >> 
> >> -
> >> >> Loading Kudu TPCH (logging to /home/amos/impala/logs/data_
> loading/load-kudu-tpch.log)...
> >> FAILED
> >> >> 'load-data tpch core kudu/none/none force' failed. Tail of log:
> >> >> distribute by hash (c_custkey) into 9 buckets stored as kudu
> >> >>
> >> >> (load-tpch-core-impala-generated-kudu-none-none.sql):
> >> >>
> >> >>
> >> >> Executing HBase Command: hbase shell load-tpch-core-hbase-
> >> generated.create
> >> >> 16/11/02 01:07:58 INFO Configuration.deprecation: hadoop.native.lib
> is
> >> deprecated. Instead, use io.native.lib.available
> >> >> SLF4J: Class path contains multiple SLF4J bindings.
> >> >> SLF4J: Found binding in [jar:file:/home/amos/impala/
> >> toolchain/cdh_components/hbase-1.2.0-cdh5.10.0-
> >> SNAPSHOT/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/
> >> StaticLoggerBinder.class]
> >> >> SLF4J: Found binding in [jar:file:/home/amos/impala/
> >> toolchain/cdh_components/hadoop-2.6.0-cdh5.10.0-
> >> SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/
> org/slf4j/impl/
> >> StaticLoggerBinder.class]
> >> >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> >> explanation.
> >> >> SLF4J: Actual 

Re: distcc now available

[Laszlo Gaal]

Thanks a lot Jim and Casey for making this happen.

Where does the distcc cluster live? Would it be usable from Budapest as
well? Looking forward to having fast full builds...

On Wed, Oct 26, 2016 at 6:51 PM, Jim Apple  wrote:

> Thanks to Casey Ching, who did almost all of the work a few months ago!
>
> On Wed, Oct 26, 2016 at 9:29 AM, Tim Armstrong 
> wrote:
> > Thanks Jim!
> >
> > I've been using distcc with Impala for a while now and it makes a huge
> > difference. I'm no longer scared to do a full build from scratch.
> >
> > On Tue, Oct 25, 2016 at 9:24 AM, Jim Apple  wrote:
> >
> >> Impala now supports distcc to speed up your builds. The instructions
> >> are in bin/distcc/README.md.
> >>
>

Re: Bootstrapping an Impala Development Environment From Scratch

[Laszlo Gaal]

Running in a VM is good idea. Do  you have a recommendation on how much
memory the VM should be configured with?

On Mon, Oct 17, 2016 at 7:58 PM, Jim Apple  wrote:

> If you are running Ubuntu 14.04, you can bootstrap a development
> environment using the script bin/bootstrap_development.sh[0]. It will
> alter your environment, including ~/.ssh/config and /etc/hosts, so
> consider running it in a VM or container.
>
> It takes 6-7 hours in total to load all of the testdata and run all of
> the tests. See the comments in the file for more information.
>
> [0]: https://git-wip-us.apache.org/repos/asf?p=incubator-impala.
> git;a=blob;f=bin/bootstrap_development.sh;h=8c4f742ae058f8017858d2a749e882
> 4be58bd410;hb=HEAD
>

Re: Can you please send the output of these commands?

[Laszlo Gaal]

This is my dev box:

laszlog@laszlog1:~$ *cat /etc/*release*

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=14.04

DISTRIB_CODENAME=trusty

DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"

NAME="Ubuntu"

VERSION="14.04.5 LTS, Trusty Tahr"

ID=ubuntu

ID_LIKE=debian

PRETTY_NAME="Ubuntu 14.04.5 LTS"

VERSION_ID="14.04"

HOME_URL="http://www.ubuntu.com/;

SUPPORT_URL="http://help.ubuntu.com/;

BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/;

laszlog@laszlog1:~$ *uname -a*

Linux laszlog1 3.19.0-69-generic #77~14.04.1-Ubuntu SMP Tue Aug 30 01:29:21
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

laszlog@laszlog1:~$ *cat
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq*

3301000

(probably true, the box is a DELL OptiPlex 7040 with a 3.3GHz Core i5-6600)

laszlog@laszlog1:~$ *head -30 /proc/cpuinfo*

processor : 0

vendor_id : GenuineIntel

cpu family : 6

model : 94

model name : Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz

stepping : 3

microcode : 0x74

cpu MHz : 800.000

cache size : 6144 KB

physical id : 0

siblings : 4

core id : 0

cpu cores : 4

apicid : 0

initial apicid : 0

fpu : yes

fpu_exception : yes

cpuid level : 22

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb
rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx
est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt
tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida
arat epb pln pts dtherm hwp hwp_noitfy hwp_act_window hwp_epp tpr_shadow
vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms
invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1

bugs :

bogomips : 6621.09

clflush size : 64

cache_alignment : 64

address sizes : 39 bits physical, 48 bits virtual

power management:

processor : 1

vendor_id : GenuineIntel

cpu family : 6







On Fri, Sep 23, 2016 at 7:35 PM, Lars Volker  wrote:

> For IMPALA-4193  I'm
> trying
> to see how widely supported the cpufreq subsystem of sysfs is. If would
> help a lot if you could run the following commands on your development
> machine and send the output to me. Thanks, Lars
>
> cat /etc/*release
> uname -a
> cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
> head -30 /proc/cpuinfo
>
> As an example, here's the output from my dev machine:
>
> ✔ ~$ cat /etc/*release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=14.04
> DISTRIB_CODENAME=trusty
> DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"
> NAME="Ubuntu"
> VERSION="14.04.4 LTS, Trusty Tahr"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu 14.04.4 LTS"
> VERSION_ID="14.04"
> HOME_URL="http://www.ubuntu.com/;
> SUPPORT_URL="http://help.ubuntu.com/;
> BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/;
> ✔ ~$ uname -a
> Linux lv-desktop 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22
> 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> ✔ ~$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
> 400
> ✔ ~$ head -30 /proc/cpuinfo
> processor   : 0
> vendor_id   : GenuineIntel
> cpu family  : 6
> model   : 60
> model name  : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
> stepping: 3
> microcode   : 0x1c
> cpu MHz : 799.875
> cache size  : 8192 KB
> physical id : 0
> siblings: 8
> core id : 0
> cpu cores   : 4
> apicid  : 0
> initial apicid  : 0
> fpu : yes
> fpu_exception   : yes
> cpuid level : 13
> wp  : yes
> flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
> pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
> xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor
> ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic
> movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida
> arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase
> tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt
> bugs:
> bogomips: 7183.48
> clflush size: 64
> cache_alignment : 64
> address sizes   : 39 bits physical, 48 bits virtual
> power management:
>
> processor   : 1
> vendor_id   : GenuineIntel
> cpu family  : 6
> ✔ ~$
>

Re: Error linking impalad with -build_shared_libs option

[Laszlo Gaal]

link.txt is generated by cmake based on the contents of CMakeLists.txt and
the project/environment settings. Most of the link-time dependencies seem
to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on settings
calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
$(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more into it
tomorrow.

Thanks,

- LaszloG

On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang 
wrote:

> I observed the same linking error and digged this into further. The below
> is what I've found.
>
> *1. how to reproduce*
>
> # assuming that impala-config.sh is sourced
> $ cd ${IMPALA_HOME}
> *$ make -j8 impalad*
> ...
> [100%] Built target CodeGen
> Linking CXX executable ../../build/debug/service/impalad
> ...
> ../../build/debug/util/libUtil.so: undefined reference to
> `google_breakpad::ExceptionHandler::WriteMinidump(std::string const&, bool
> (*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
> ../../build/debug/util/libUtil.so: undefined reference to
> `snappy::MaxCompressedLength(unsigned long)'
> ../../build/debug/util/libUtil.so: undefined reference to `my_strlen'
> ../../build/debug/util/libUtil.so: undefined reference to `LZ4_compress'
> ../../build/debug/util/libUtil.so: undefined reference to
> `BZ2_bzDecompress'
> ...
>
>
> *$ cd ${IMPALA_HOME}/be/src/service*
> *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> *cmake_link_script CMakeFiles/impalad.dir/link.txt --verbose=*
> ...
> ../../build/debug/util/libUtil.so: undefined reference to
> ...
> *../../build/debug/util/libUtil.so: undefined reference to `LZ4_compress'*
> *../../build/debug/util/libUtil.so: undefined reference to
> `BZ2_bzDecompress'*
> ../../build/debug/util/libUtil.so: undefined reference to
> *`snappy::RawCompress*(char const*,
> unsigned long, char*, unsigned long*)'
> ...
> boost::regex_traits >::do_assign(char
> const*, char
> const*, unsigned int)'
> ...
> ../../build/debug/util/libUtil.so: undefined reference to *`snappy::*
> RawUncompress(char
> const*, unsigned long, char*)'
>
> *2. cause of the failure*
>
> $ cd ${IMPALA_HOME}/be/src/service
> *$ cat CMakeFiles/impalad.dir/link.txt*
> ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++-Wall -Wno-sign-compare
> -Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
> -Wno-deprecated ...
> *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> ../../../toolchain/lz4-svn/lib/liblz4.a*
> ...
> *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> ...
> *../../build/debug/util/libUtil.so*
> -lpthread -ldl -lssl -lcrypto
>
> Apparently, libUtil.so __should__ show before libsnappy.a and *regex.a.
>
>
> Next step is to find out who creates CMakeFiles/impalad.dir/link.txt and
> why libUtil.so is placed after libsnappy.a, liblz4.a, and libboost_regex.a
>
> Thank you,
> Yonghyun
>
>
>
>
>
> On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson 
> wrote:
>
> > Only that it's IMPALA-3979. Does anyone have any theories yet?
> >
> > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > dtsirogian...@cloudera.com> wrote:
> >
> > > Hey guys,
> > >
> > > When I build from the latest master using the -build_shared_libs
> option I
> > > get linking errors (see [1]). Building with static works fine. Any
> ideas?
> > >
> > > Thanks
> > > Dimitris
> > >
> > >
> > > [1]
> > > Linking CXX executable ../../build/debug/service/impalad
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'BZ2_bzBuffToBuffCompress'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'snappy::MaxCompressedLength(unsigned long)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'snappy::RawCompress(char const*, unsigned long, char*, unsigned
> long*)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'LZ4_compress'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'BZ2_bzDecompressEnd'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'BZ2_bzDecompressInit'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'BZ2_bzBuffToBuffDecompress'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'BZ2_bzDecompress'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'snappy::GetUncompressedLength(char const*, unsigned long, unsigned
> > > long*)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'snappy::RawUncompress(char const*, unsigned long, char*)'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'LZ4_uncompress'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'my_strlen'
> > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > 'google_breakpad::ExceptionHandler::WriteMinidump(std::string const&,
> > bool
> > > (*)(google_breakpad::MinidumpDescriptor