Isn't that already part of the script? https://github.com/awleblang/impala-setup/commit/56fa829c99e997585eb63fcd49cb65eb8357e679
https://git-wip-us.apache.org/repos/asf?p=incubator-impala.git;a=blob;f=bin/bootstrap_development.sh;h=8c4f742ae058f8017858d2a749e8824be58bd410;hb=HEAD#l68 On Tue, Nov 1, 2016 at 9:44 PM, Dimitris Tsirogiannis <[email protected]> wrote: > Hi Amos, > > You need to increase your limits (/etc/security/limits.conf) for max > number of open files (nofile). Use a pretty big number (e.g. 500K) for > both soft and hard. > > Hope that helps. > > Dimitris > > On Tue, Nov 1, 2016 at 8:57 PM, Amos Bird <[email protected]> wrote: >> >> Hi there, >> >> After days of efforts to make impala's local tests work on my Centos >> machine, I finally gave up and turns to Ubuntu. I followed this simple >> guide >> https://cwiki.apache.org/confluence/display/IMPALA/Bootstrapping+an+Impala+Development+Environment+From+Scratch >> on a fresh installed Ubuntu 14.04. Unfortunately there are still errors >> in loading data phase. Here is the error log, >> >> --------------------------------------------------------------------------------------------- >> Loading Kudu TPCH (logging to >> /home/amos/impala/logs/data_loading/load-kudu-tpch.log)... FAILED >> 'load-data tpch core kudu/none/none force' failed. Tail of log: >> distribute by hash (c_custkey) into 9 buckets stored as kudu >> >> (load-tpch-core-impala-generated-kudu-none-none.sql): >> >> >> Executing HBase Command: hbase shell load-tpch-core-hbase-generated.create >> 16/11/02 01:07:58 INFO Configuration.deprecation: hadoop.native.lib is >> deprecated. Instead, use io.native.lib.available >> SLF4J: Class path contains multiple SLF4J bindings. >> SLF4J: Found binding in >> [jar:file:/home/amos/impala/toolchain/cdh_components/hbase-1.2.0-cdh5.10.0-SNAPSHOT/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in >> [jar:file:/home/amos/impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.10.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >> explanation. >> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >> Executing HBase Command: hbase shell post-load-tpch-core-hbase-generated.sql >> 16/11/02 01:08:03 INFO Configuration.deprecation: hadoop.native.lib is >> deprecated. Instead, use io.native.lib.available >> SLF4J: Class path contains multiple SLF4J bindings. >> SLF4J: Found binding in >> [jar:file:/home/amos/impala/toolchain/cdh_components/hbase-1.2.0-cdh5.10.0-SNAPSHOT/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in >> [jar:file:/home/amos/impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.10.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >> explanation. >> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >> Invalidating Metadata >> (load-tpch-core-impala-load-generated-kudu-none-none.sql): >> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem >> >> Data Loading from Impala failed with error: ImpalaBeeswaxException: >> Query aborted: >> Kudu error(s) reported, first error: Timed out: Failed to write batch of >> 2708 ops to tablet 84aa134fb6c24916aa16cf50f48ec557 after 329 attempt(s): >> Failed to write to server: (no server available): Write(tablet: >> 84aa134fb6c24916aa16cf50f48ec557, num_ops: 2708, num_attempts: 329) passed >> its deadline: Network error: recv error: Connection reset by peer (error 104) >> >> >> >> Kudu error(s) reported, first error: Timed out: Failed to write batch of >> 2708 ops to tablet 84aa134fb6c24916aa16cf50f48ec557 after 329 attempt(s): >> Failed to write to server: (no server available): Write(tablet: >> 84aa134fb6c24916aa16cf50f48ec557, num_ops: 2708, num_attempts: 329) passed >> its deadline: Network error: recv error: Connection reset by peer (error 104) >> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to write >> batch of 2708 ops to tablet 84aa134fb6c24916aa16cf50f48ec557 after 329 >> attempt(s): Failed to write to server: (no server available): Write(tablet: >> 84aa134fb6c24916aa16cf50f48ec557, num_ops: 2708, num_attempts: 329) passed >> its deadline: Network error: recv error: Connection reset by peer (error >> 104) (1 of 2708 similar) >> >> Traceback (most recent call last): >> File "/home/amos/impala/bin/load-data.py", line 158, in >> exec_impala_query_from_file >> result = impala_client.execute(query) >> File "/home/amos/impala/tests/beeswax/impala_beeswax.py", line 173, in >> execute >> handle = self.__execute_query(query_string.strip(), user=user) >> File "/home/amos/impala/tests/beeswax/impala_beeswax.py", line 339, in >> __execute_query >> self.wait_for_completion(handle) >> File "/home/amos/impala/tests/beeswax/impala_beeswax.py", line 359, in >> wait_for_completion >> raise ImpalaBeeswaxException("Query aborted:" + error_log, None) >> ImpalaBeeswaxException: ImpalaBeeswaxException: >> Query aborted: >> Kudu error(s) reported, first error: Timed out: Failed to write batch of >> 2708 ops to tablet 84aa134fb6c24916aa16cf50f48ec557 after 329 attempt(s): >> Failed to write to server: (no server available): Write(tablet: >> 84aa134fb6c24916aa16cf50f48ec557, num_ops: 2708, num_attempts: 329) passed >> its deadline: Network error: recv error: Connection reset by peer (error 104) >> >> >> >> Kudu error(s) reported, first error: Timed out: Failed to write batch of >> 2708 ops to tablet 84aa134fb6c24916aa16cf50f48ec557 after 329 attempt(s): >> Failed to write to server: (no server available): Write(tablet: >> 84aa134fb6c24916aa16cf50f48ec557, num_ops: 2708, num_attempts: 329) passed >> its deadline: Network error: recv error: Connection reset by peer (error 104) >> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to write >> batch of 2708 ops to tablet 84aa134fb6c24916aa16cf50f48ec557 after 329 >> attempt(s): Failed to write to server: (no server available): Write(tablet: >> 84aa134fb6c24916aa16cf50f48ec557, num_ops: 2708, num_attempts: 329) passed >> its deadline: Network error: recv error: Connection reset by peer (error >> 104) (1 of 2708 similar) >> >> Error in /home/amos/impala/testdata/bin/create-load-data.sh at line 45: >> while [ -n "$*" ] >> + cleanup >> + rm -rf /tmp/tmp.hMzGwIcUo3 >> --------------------------------------------------------------------------------------------- >> >> This kinda blocks my patch's rebasing. Any help is much appreciated! >> >> regards, >> Amos
