Re: [VOTE] Graduate to a TLP

[Martin Grund (Das Grundprinzip.de)]

+1

On Wed, Oct 18, 2017 at 7:58 PM Shant Hovsepian 
wrote:

> +1
>
> woohoo
>
> On Wed, Oct 18, 2017 at 8:18 PM Bikramjeet Vig <
> bikramjeet@cloudera.com>
> wrote:
>
> > +1
> >
> > On Tue, Oct 17, 2017 at 7:06 PM, Jim Apple  wrote:
> >
> > > Following our discussion
> > > https://lists.apache.org/thread.html/2f5db4788aff9b0557354b9106c032
> > > 8a29c1f90c1a74a228163949d2@%3Cdev.impala.apache.org%3E
> > > , I propose that we graduate to a TLP. According to
> > > https://incubator.apache.org/guides/graduation.html#
> > > community_graduation_vote
> > > this is not required, and https://impala.apache.org/bylaws.html does
> not
> > > say whose votes are "binding" in a graduation vote, so all community
> > > members are welcome to vote.
> > >
> > > This will remain open 72 hours. I will be notifying general@incubator
> it
> > > is
> > > occurring.
> > >
> > > This is my +1.
> > >
> >
>
-- 
Martin Grund
1035 Dolores Dr
Lafayette, CA, 94549
USA

Re: run-tests.py on Ubuntu 16.04

[Martin Grund (Das Grundprinzip.de)]

It looks like it tries to load a function with a symbol that the Kudu
client library does not provide. If you haven't so far, I would nm the Kudu
client libary and grep for the KuduTablet symbol. According to the
demangler.com the symbol it's looking for uses the new C++11 ABI and I
might guess that you're still compiling with the old one? Without knowing
anything I might guess that the python package was compiled with a more
recent compiler (new ABI enabled by default) but the toolchain Kudu client
has the old ABI set.

kudu::client::KuduTablet::id[abi:cxx11]() const

Maybe this helps,
Martin

On Sat, Jan 14, 2017 at 12:47 PM Alex Behm  wrote:

> Anyone seen this and know how to fix?
>
> abehm@thinkpad:~/impala/tests$ ./run-tests.py
> query_test/test_avro_schema_resolution.py
> ...
> Traceback (most recent call last):
>   File
>
> "/home/abehm/impala/infra/python/env/local/lib/python2.7/site-packages/_pytest/config.py",
> line 543, in importconftest
> mod = conftestpath.pyimport()
>   File
>
> "/home/abehm/impala/infra/python/env/local/lib/python2.7/site-packages/py/_path/local.py",
> line 650, in pyimport
> __import__(modname)
>   File "/home/abehm/impala/tests/conftest.py", line 21, in 
> from kudu import connect as kudu_connect
>   File
>
> "/home/abehm/impala/infra/python/env/local/lib/python2.7/site-packages/kudu/__init__.py",
> line 18, in 
> from kudu.client import (Client, Table, Scanner, Session,  # noqa
> ImportError:
>
> /home/abehm/impala/infra/python/env/local/lib/python2.7/site-packages/kudu/client.so:
> undefined symbol: _ZNK4kudu6client10KuduTablet2idB5cxx11Ev
> ERROR: could not load /home/abehm/impala/tests/conftest.py
>

Re: Do we have something like git-clang-format for python/PEP8?

[Martin Grund (Das Grundprinzip.de)]

There is something called autoflake8 it can't solve all issues due to the
way that python only has limited cases for line continuations.

Martin

On Thu, Dec 15, 2016, 13:36 Lars Volker  wrote:

> Hi all,
>
> do we have a tool to format only the changed lines in python files
> according to PEP8, similar to what git-clang-format does for c++ files?
>
> Thanks, Lars
>

Re: Error linking impalad with -build_shared_libs option

[Martin Grund (Das Grundprinzip.de)]

What happened is that the Util library was linked to the error-utils-test
(which it is by default anyways because of the way how ADD_BE_TEST works).
Now, what happened is that if you dynamically link the order of the
libraries plays a role. The additional target-link-library would move the
libUtil.so to the end of the library list and if the linker picks this up
some of the symbols declared earlier where already dropped.

On Wed, Aug 24, 2016 at 11:00 AM Henry Robinson <he...@cloudera.com> wrote:

> Seems great to me - but do we know why this fixes the issue? Not that that
> should prevent us from merging the patch.
>
> On 24 August 2016 at 08:54, Tim Armstrong <tarmstr...@cloudera.com> wrote:
>
> > The fix works for me too, thanks Martin!
> >
> > I gave the patch a +1 but will upgrade to +2 if it seems everyone is
> happy
> > with it.
> >
> > Cheers,
> > Tim
> >
> > On Wed, Aug 24, 2016 at 8:15 AM, Yonghyun Hwang <yongh...@cloudera.com>
> > wrote:
> >
> > > Hello Martin,
> > >
> > > Your fix works great! :) As for the build failure, what I've found is
> > that
> > > cmake 3.2.3 causes a build failure w/o your fix while cmake 2.8.12
> works
> > > good.
> > >
> > > After the commit for "IMPALA-3924: Ubuntu16 support", cmake 3.2.3 got
> > > deployed under "toolchain", which reveals this build issue.
> > >
> > > So, any one using the newer cmake would require your fix to properly
> > build
> > > impala w/ -so.
> > >
> > > Thank you,
> > > Yonghyun
> > >
> > > On Tue, Aug 23, 2016 at 11:40 PM, Daniel Hecht <dhe...@cloudera.com>
> > > wrote:
> > >
> > > > Hi Martin,
> > > >
> > > > Nice to hear from you.
> https://issues.cloudera.org/browse/IMPALA-3979
> > > >
> > > > Thanks,
> > > > Dan
> > > >
> > > > On Tuesday, August 23, 2016, Martin Grund (Das Grundprinzip.de) <
> > > > grundprin...@gmail.com> wrote:
> > > >
> > > > > I've seen this in the past and propose a patch to fix it:
> > > > > https://gerrit.cloudera.org/#/c/4108/
> > > > >
> > > > > Let me know if you have JIRA to track this to update the commit
> > > message.
> > > > >
> > > > > Martin
> > > > >
> > > > > On Mon, Aug 22, 2016 at 10:15 PM Laszlo Gaal <
> > laszlo.g...@cloudera.com
> > > > > <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > link.txt is generated by cmake based on the contents of
> > > CMakeLists.txt
> > > > > and
> > > > > > the project/environment settings. Most of the link-time
> > dependencies
> > > > seem
> > > > > > to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on
> > settings
> > > > > > calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
> > > > > > $(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more
> > into
> > > it
> > > > > > tomorrow.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > - LaszloG
> > > > > >
> > > > > > On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang <
> > > yongh...@cloudera.com
> > > > > <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > I observed the same linking error and digged this into further.
> > The
> > > > > below
> > > > > > > is what I've found.
> > > > > > >
> > > > > > > *1. how to reproduce*
> > > > > > >
> > > > > > > # assuming that impala-config.sh is sourced
> > > > > > > $ cd ${IMPALA_HOME}
> > > > > > > *$ make -j8 impalad*
> > > > > > > ...
> > > > > > > [100%] Built target CodeGen
> > > > > > > Linking CXX executable ../../build/debug/service/impalad
> > > > > > > ...
> > > > > > > ../../build/debug/util/libUtil.so: undefined reference to
> > > > > > > `google_breakpad::ExceptionHandler::WriteMinidump(std::string
> > > > const&,
> > > > > > bool
> > > > > > > (*)(google_breakpad::MinidumpDescr

Re: Error linking impalad with -build_shared_libs option

[Martin Grund (Das Grundprinzip.de)]

I've seen this in the past and propose a patch to fix it:
https://gerrit.cloudera.org/#/c/4108/

Let me know if you have JIRA to track this to update the commit message.

Martin

On Mon, Aug 22, 2016 at 10:15 PM Laszlo Gaal 
wrote:

> link.txt is generated by cmake based on the contents of CMakeLists.txt and
> the project/environment settings. Most of the link-time dependencies seem
> to be set up in $(IMPALA_HOME)/be/CMakeLists.txt, based on settings
> calculated one level up in $(IMPALA_HOME)/CMakeLists.txt.
> $(IMPALA_HOME)/be/CMakeLists.txt looks complex; I'll look more into it
> tomorrow.
>
> Thanks,
>
> - LaszloG
>
> On Mon, Aug 22, 2016 at 6:07 PM, Yonghyun Hwang 
> wrote:
>
> > I observed the same linking error and digged this into further. The below
> > is what I've found.
> >
> > *1. how to reproduce*
> >
> > # assuming that impala-config.sh is sourced
> > $ cd ${IMPALA_HOME}
> > *$ make -j8 impalad*
> > ...
> > [100%] Built target CodeGen
> > Linking CXX executable ../../build/debug/service/impalad
> > ...
> > ../../build/debug/util/libUtil.so: undefined reference to
> > `google_breakpad::ExceptionHandler::WriteMinidump(std::string const&,
> bool
> > (*)(google_breakpad::MinidumpDescriptor const&, void*, bool), void*)'
> > ../../build/debug/util/libUtil.so: undefined reference to
> > `snappy::MaxCompressedLength(unsigned long)'
> > ../../build/debug/util/libUtil.so: undefined reference to `my_strlen'
> > ../../build/debug/util/libUtil.so: undefined reference to `LZ4_compress'
> > ../../build/debug/util/libUtil.so: undefined reference to
> > `BZ2_bzDecompress'
> > ...
> >
> >
> > *$ cd ${IMPALA_HOME}/be/src/service*
> > *$ ${IMPALA_HOME}/toolchain/cmake-3.2.3-p1/bin/cmake -E  *
> > *cmake_link_script CMakeFiles/impalad.dir/link.txt --verbose=*
> > ...
> > ../../build/debug/util/libUtil.so: undefined reference to
> > ...
> > *../../build/debug/util/libUtil.so: undefined reference to
> `LZ4_compress'*
> > *../../build/debug/util/libUtil.so: undefined reference to
> > `BZ2_bzDecompress'*
> > ../../build/debug/util/libUtil.so: undefined reference to
> > *`snappy::RawCompress*(char const*,
> > unsigned long, char*, unsigned long*)'
> > ...
> > boost::regex_traits
> >::do_assign(char
> > const*, char
> > const*, unsigned int)'
> > ...
> > ../../build/debug/util/libUtil.so: undefined reference to *`snappy::*
> > RawUncompress(char
> > const*, unsigned long, char*)'
> >
> > *2. cause of the failure*
> >
> > $ cd ${IMPALA_HOME}/be/src/service
> > *$ cat CMakeFiles/impalad.dir/link.txt*
> > ${IMPALA_HOME}/toolchain/gcc-4.9.2/bin/g++-Wall -Wno-sign-compare
> > -Wno-unknown-pragmas -pthread -fno-strict-aliasing -std=c++14
> > -Wno-deprecated ...
> > *../../../toolchain/snappy-1.1.3/lib/libsnappy.a
> > ../../../toolchain/lz4-svn/lib/liblz4.a*
> > ...
> > *../../../toolchain/boost-1.57.0/lib/libboost_regex.a*
> > ...
> > *../../build/debug/util/libUtil.so*
> > -lpthread -ldl -lssl -lcrypto
> >
> > Apparently, libUtil.so __should__ show before libsnappy.a and *regex.a.
> >
> >
> > Next step is to find out who creates CMakeFiles/impalad.dir/link.txt and
> > why libUtil.so is placed after libsnappy.a, liblz4.a, and
> libboost_regex.a
> >
> > Thank you,
> > Yonghyun
> >
> >
> >
> >
> >
> > On Wed, Aug 17, 2016 at 10:13 AM, Henry Robinson 
> > wrote:
> >
> > > Only that it's IMPALA-3979. Does anyone have any theories yet?
> > >
> > > On 17 August 2016 at 10:10, Dimitris Tsirogiannis <
> > > dtsirogian...@cloudera.com> wrote:
> > >
> > > > Hey guys,
> > > >
> > > > When I build from the latest master using the -build_shared_libs
> > option I
> > > > get linking errors (see [1]). Building with static works fine. Any
> > ideas?
> > > >
> > > > Thanks
> > > > Dimitris
> > > >
> > > >
> > > > [1]
> > > > Linking CXX executable ../../build/debug/service/impalad
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'BZ2_bzBuffToBuffCompress'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'snappy::MaxCompressedLength(unsigned long)'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'snappy::RawCompress(char const*, unsigned long, char*, unsigned
> > long*)'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'LZ4_compress'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'BZ2_bzDecompressEnd'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'BZ2_bzDecompressInit'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'BZ2_bzBuffToBuffDecompress'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'BZ2_bzDecompress'
> > > > ../../build/debug/util/libUtil.so: error: undefined reference to
> > > > 'snappy::GetUncompressedLength(char const*, unsigned long, unsigned
> > > > long*)'
> 

Re: HBase errors prevent run-all-tests.sh

[Martin Grund (Das Grundprinzip.de)]

One idea is to check your ulimit for file descriptors and run `lsof | grep
wc -l` to see if you for some reason exceeded the limit. Otherwise, a fresh
reboot might help to figure out if you somewhere have a spare process
hogging FDs.

On Sun, Jul 24, 2016 at 8:09 PM Jim Apple  wrote:

> The NN and DN logs are empty.
>
> I bin/kill-all.sh at the beginning of this, so I assume that nothing
> is taking them except for my little Impala work.
>
> On Sun, Jul 24, 2016 at 8:03 PM, Bharath Vissapragada
>  wrote:
> > Based on
> >
> > 16/07/24 18:36:08 WARN hdfs.BlockReaderFactory: I/O error constructing
> > remote block reader.
> > java.net.SocketException: Too many open files
> >
> > 16/07/24 18:36:08 WARN hdfs.DFSClient: Failed to connect to
> > /127.0.0.1:31000 for block, add to deadNodes and continue.
> > java.net.SocketException: Too many open files
> >
> > I'm guessing your hdfs instance might be overloaded (check the NN/DN
> logs).
> > HMaster is unable to connect to NN while opening regions and hence
> throwing
> > the error.
> >
> > On Mon, Jul 25, 2016 at 8:05 AM, Jim Apple  wrote:
> >
> >> Several thousand lines of things like
> >>
> >> WARN shortcircuit.ShortCircuitCache: ShortCircuitCache(0x419c7df4):
> >> failed to load 1073764575_BP-1490185442-127.0.0.1-1456935654337
> >>
> >> java.lang.NullPointerException at
> >>
> >>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitReplica.(ShortCircuitReplica.java:126)
> >> ...
> >>
> >> 16/07/24 18:36:08 WARN hdfs.BlockReaderFactory:
> >>
> >>
> BlockReaderFactory(fileName=/hbase/MasterProcWALs/state-3172.log,
> >> block=BP-1490185442-127.0.0.1-1456935654337:blk_1073764629_23805):
> >> error creating ShortCircuitReplica.
> >>
> >> java.io.EOFException: unexpected EOF while reading metadata file header
> >>
> >> 16/07/24 18:36:08 WARN hdfs.BlockReaderFactory: I/O error constructing
> >> remote block reader.
> >> java.net.SocketException: Too many open files
> >>
> >> 16/07/24 18:36:08 WARN hdfs.DFSClient: Failed to connect to
> >> /127.0.0.1:31000 for block, add to deadNodes and continue.
> >> java.net.SocketException: Too many open files
> >>
> >> 16/07/24 18:36:08 INFO hdfs.DFSClient: Could not obtain
> >> BP-1490185442-127.0.0.1-1456935654337:blk_1073764629_23805 from any
> >> node: java.io.IOException: No live nodes contain block
> >> BP-1490185442-127.0.0.1-1456935654337:blk_1073764629_23805 a
> >> fter checking nodes =
> >> [DatanodeInfoWithStorage[127.0.0.1:31000
> >> ,DS-0232508a-5512-4827-bcaf-c922f1e65eb1,DISK]],
> >> ignoredNodes = null No live nodes contain current block Block
> >> locations: DatanodeInfoWithStorage[127.0.0.1:31000,DS-0232508a-551
> >> 2-4827-bcaf-c922f1e65eb1,DISK] Dead nodes:
> >> DatanodeInfoWithStorage[127.0.0.1:31000
> >> ,DS-0232508a-5512-4827-bcaf-c922f1e65eb1,DISK].
> >> Will get new block locations from namenode and retry...
> >> 16/07/24 18:36:08 WARN hdfs.DFSClient: DFS chooseDataNode: got # 1
> >> IOException, will wait for 2772.7114628272548 msec.
> >> 16/07/24 18:36:11 WARN hdfs.BlockReaderFactory:
> >>
> >>
> BlockReaderFactory(fileName=/hbase/MasterProcWALs/state-3172.log,
> >> block=BP-1490185442-127.0.0.1-1456935654337:blk_1073764629_23805):
> >> error creating ShortCircuitReplica.
> >> java.io.IOException: Illegal seek
> >> at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
> >>
> >> On Sun, Jul 24, 2016 at 7:24 PM, Bharath Vissapragada
> >>  wrote:
> >> > Do you see something in the HMaster log? From the error it looks like
> the
> >> > Hbase master hasn't started properly for some reason.
> >> >
> >> > On Mon, Jul 25, 2016 at 6:08 AM, Jim Apple 
> wrote:
> >> >
> >> >> I tried reloading the data with
> >> >>
> >> >> ./bin/load-data.py --workloads functional-query
> >> >>
> >> >> but that gave errors like
> >> >>
> >> >> Executing HBase Command: hbase shell
> >> >> load-functional-query-core-hbase-generated.create
> >> >> 16/07/24 17:19:39 INFO Configuration.deprecation: hadoop.native.lib
> is
> >> >> deprecated. Instead, use io.native.lib.available
> >> >> SLF4J: Class path contains multiple SLF4J bindings.
> >> >> SLF4J: Found binding in
> >> >>
> >> >>
> >>
> [jar:file:/opt/Impala-Toolchain/cdh_components/hbase-1.2.0-cdh5.9.0-SNAPSHOT/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >> >> SLF4J: Found binding in
> >> >>
> >> >>
> >>
> [jar:file:/opt/Impala-Toolchain/cdh_components/hadoop-2.6.0-cdh5.9.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >> >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> >> >> explanation.
> >> >> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> >> >>
> >> >> ERROR: Can't get the locations
> >> >>
> >> >> Here is some help for this command:
> >> >> Start disable of named table:
> >> >>   hbase> disable 't1'