Hi Joseph, I'm puzzled, but a few things to try:
* What's /etc/hosts look like? Is it possible that "localhost" is ambiguous in some way? * Googling found things like https://groups.google.com/forum/#!topic/kudu-user/k0DuoElPXiY for me. You may want to look at the Kudu server logs (and maybe the Kudu client logs in Impala) to see if something's up with SASL. * Try running "docker/test-with-docker.py --suite NOOP --no-cleanup-image" after pulling from Impala's git repo. (It's very recent.) That command will create a build image with the test data loaded. If it works, we can triangulate the difference between what you've been doing and what it does. If it doesn't work, well, that may tell us something too. -- Philip On Sat, Apr 7, 2018 at 7:27 PM, Jim Apple <jbap...@cloudera.com> wrote: > Are you able to bootstrap a kudu development environment inside the > container? I haven't tried this before, but maybe there's something going > on with the port number? > > On Sat, Apr 7, 2018 at 6:53 PM, Joseph Koshakow <kosh...@gmail.com> wrote: > > > Hi, > > > > *results of df -h:* > > impdev@c4fdb0931b2d:~/Impala$ df -h > > Filesystem Size Used Avail Use% Mounted on > > none 300G 107G 180G 38% / > > tmpfs 5.9G 0 5.9G 0% /dev > > tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup > > /dev/sda9 300G 107G 180G 38% /etc/hosts > > shm 64M 0 64M 0% /dev/shm > > > > > > *Here is the error I'm getting:* > > Starting Impala cluster (logging to > > /home/impdev/Impala/logs/data_loading/start-impala-cluster.log)... > > Starting Impala cluster OK (Took: 0 min 21 sec) > > Setting up HDFS environment (logging to > > /home/impdev/Impala/logs/data_loading/setup-hdfs-env.log)... > > Setting up HDFS environment OK (Took: 0 min 9 sec) > > Loading custom schemas (logging to > > /home/impdev/Impala/logs/data_loading/load-custom-schemas.log)... > > Loading custom schemas OK (Took: 0 min 36 sec) > > Started Loading functional-query data in background; pid 13490. > > Loading functional-query data (logging to > > /home/impdev/Impala/logs/data_loading/load-functional-query.log)... > > Started Loading TPC-H data in background; pid 13491. > > Started Loading TPC-DS data in background; pid 13492. > > Loading TPC-H data (logging to > > /home/impdev/Impala/logs/data_loading/load-tpch.log)... > > Loading TPC-DS data (logging to > > /home/impdev/Impala/logs/data_loading/load-tpcds.log)... > > FAILED (Took: 28 min 11 sec) > > 'load-data tpch core' failed. Tail of log: > > INVALIDATE METADATA tpch_kudu.partsupp > > > > (invalidate-tpch-core-impala-generated.sql): > > INVALIDATE METADATA tpch_kudu.supplier > > > > (invalidate-tpch-core-impala-generated.sql): > > INVALIDATE METADATA tpch_kudu.nation > > > > (invalidate-tpch-core-impala-generated.sql): > > INVALIDATE METADATA tpch_kudu.region > > > > (invalidate-tpch-core-impala-generated.sql): > > INVALIDATE METADATA tpch_kudu.orders > > > > (invalidate-tpch-core-impala-generated.sql): > > INVALIDATE METADATA tpch_kudu.customer > > > > (invalidate-tpch-core-impala-generated.sql): > > > > > > (load-tpch-core-impala-generated-kudu-none-none.sql): > > INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem > > > > (load-tpch-core-impala-generated-parquet-none-none.sql): > > INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM tpch.lineitem > > > > (load-tpch-core-impala-generated-parquet-none-none.sql): > > INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part > > > > Data Loading from Impala failed with error: ImpalaBeeswaxException: > > Query aborted:Kudu error(s) reported, first error: Timed out: Failed to > > write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce > after > > 4 > > attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem', > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client > > connection negotiation failed: client connection to 127.0.0.1:7051 > > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to > > write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce > after > > 4 > > attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem', > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client > > connection negotiation failed: client connection to 127.0.0.1:7051 (1 of > > 78325 similar) > > > > > > Traceback (most recent call last): > > File "/home/impdev/Impala/bin/load-data.py", line 179, in > > exec_impala_query_from_file > > result = impala_client.execute(query) > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 173, > in > > execute > > handle = self.__execute_query(query_string.strip(), user=user) > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 341, > in > > __execute_query > > self.wait_for_completion(handle) > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 361, > in > > wait_for_completion > > raise ImpalaBeeswaxException("Query aborted:" + error_log, None) > > ImpalaBeeswaxException: ImpalaBeeswaxException: > > Query aborted:Kudu error(s) reported, first error: Timed out: Failed to > > write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce > after > > 4 > > attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem', > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client > > connection negotiation failed: client connection to 127.0.0.1:7051 > > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to > > write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce > after > > 4 > > attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem', > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client > > connection negotiation failed: client connection to 127.0.0.1:7051 (1 of > > 78325 similar) > > > > > > If you'd like I can copy and paste the entire output. Thanks for the > help. > > > > > > On Sat, Apr 7, 2018 at 9:44 PM, Jim Apple <jbap...@cloudera.com> wrote: > > > > > The attachment didn't attach. This is a common issue on our mailing > > lists. > > > I've started another thread about it. > > > > > > I'll try to reproduce this. My first suspicions are disk space, > > networking > > > restrictions, and https://issues.apache.org/jira/browse/IMPALA-6737 > > > > > > Inside of the container, can you run df -h before exiting? > > > > > > On Sat, Apr 7, 2018 at 5:28 PM, Joseph Koshakow <kosh...@gmail.com> > > wrote: > > > > > > > Hi, > > > > > > > > Tried this again recently and still experiencing the same error. I've > > > > attached a copy of all my shell commands and output. > > > > > > > > On Sat, Feb 24, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com> > > > > wrote: > > > > > > > >> Though I just upgraded to the most recent version. It's still > failing > > > but > > > >> seems to be making it one step farther. I see from the emails that > > > there's > > > >> some issues with the current build so I'll just wait until that's > all > > > >> resolved. The output from my most recent attempt is below if you > have > > a > > > use > > > >> for it. > > > >> > > > >> Loading custom schemas OK (Took: 0 min 40 sec) > > > >> Started Loading functional-query data in background; pid 22229. > > > >> Loading functional-query data (logging to > > /home/impdev/Impala/logs/data_ > > > >> loading/load-functional-query.log)... > > > >> Started Loading TPC-H data in background; pid 22230. > > > >> Started Loading TPC-DS data in background; pid 22231. > > > >> Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_ > > > >> loading/load-tpcds.log)... > > > >> Loading TPC-H data (logging to /home/impdev/Impala/logs/data_ > > > >> loading/load-tpch.log)... > > > >> FAILED (Took: 25 min 51 sec) > > > >> 'load-data tpch core' failed. Tail of log: > > > >> INVALIDATE METADATA tpch_kudu.region > > > >> > > > >> (invalidate-tpch-core-impala-generated.sql): > > > >> INVALIDATE METADATA tpch_kudu.orders > > > >> > > > >> (invalidate-tpch-core-impala-generated.sql): > > > >> INVALIDATE METADATA tpch_kudu.customer > > > >> > > > >> (invalidate-tpch-core-impala-generated.sql): > > > >> > > > >> > > > >> (load-tpch-core-impala-generated-kudu-none-none.sql): > > > >> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem > > > >> > > > >> (load-tpch-core-impala-generated-parquet-none-none.sql): > > > >> INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM > > tpch.lineitem > > > >> > > > >> (load-tpch-core-impala-generated-parquet-none-none.sql): > > > >> INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part > > > >> > > > >> (load-tpch-core-impala-generated-parquet-none-none.sql): > > > >> INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM > > tpch.partsupp > > > >> > > > >> (load-tpch-core-impala-generated-parquet-none-none.sql): > > > >> INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM > > tpch.supplier > > > >> > > > >> Data Loading from Impala failed with error: ImpalaBeeswaxException: > > > >> Query aborted:Kudu error(s) reported, first error: Timed out: > Failed > > to > > > >> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9 > > > >> after 4 attempt(s): GetTableLocations { table: > > > >> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 } > > > >> failed: Client connection negotiation failed: client connection to > > > >> 127.0.0.1:7051 > > > >> > > > >> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed > to > > > >> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34 > > > >> after 4 attempt(s): GetTableLocations { table: > > > >> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2), > > > >> attempt: 1 } failed: Client connection negotiation failed: client > > > >> connection to 127.0.0.1:7051 (1 of 21494 similar) > > > >> > > > >> > > > >> Traceback (most recent call last): > > > >> File "/home/impdev/Impala/bin/load-data.py", line 179, in > > > >> exec_impala_query_from_file > > > >> (load-tpch-core-impala-generated-parquet-none-none.sql): > > > >> INSERT OVERWRITE TABLE tpch_parquet.nation SELECT * FROM tpch.nation > > > >> > > > >> result = impala_client.execute(query) > > > >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line > > 173, > > > >> in execute > > > >> handle = self.__execute_query(query_string.strip(), user=user) > > > >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line > > 341, > > > >> in __execute_query > > > >> self.wait_for_completion(handle) > > > >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line > > 361, > > > >> in wait_for_completion > > > >> raise ImpalaBeeswaxException("Query aborted:" + error_log, None) > > > >> ImpalaBeeswaxException: ImpalaBeeswaxException: > > > >> Query aborted:Kudu error(s) reported, first error: Timed out: > Failed > > to > > > >> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9 > > > >> after 4 attempt(s): GetTableLocations { table: > > > >> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 } > > > >> failed: Client connection negotiation failed: client connection to > > > >> 127.0.0.1:7051 > > > >> > > > >> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed > to > > > >> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34 > > > >> after 4 attempt(s): GetTableLocations { table: > > > >> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2), > > > >> attempt: 1 } failed: Client connection negotiation failed: client > > > >> connection to 127.0.0.1:7051 (1 of 21494 similar) > > > >> > > > >> > > > >> On Fri, Feb 23, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com > > > > > >> wrote: > > > >> > > > >>> 8302608bdff15d6e8af324fc885dd0a8c2b44a5d > > > >>> > > > >>> On Fri, Feb 23, 2018 at 12:41 AM, Jim Apple <jbap...@cloudera.com> > > > >>> wrote: > > > >>> > > > >>>> What commit hash is checked out? > > > >>>> > > > >>>> On Thu, Feb 22, 2018 at 8:23 PM Joseph Koshakow < > kosh...@gmail.com> > > > >>>> wrote: > > > >>>> > > > >>>> > Host OS is Ubunto 16.04.3 LTS. Docker OS is also Ubuntu 16.04 as > > per > > > >>>> > instructions. I have 202.61Gb of free space in the host > > environment > > > >>>> > (304.58Gb size total). I'm not sure how to check how much space > is > > > >>>> left in > > > >>>> > docker containers but the size of the one created for this is > > 97.81 > > > >>>> Gb. > > > >>>> > > > > >>>> > -Joe > > > >>>> > > > > >>>> > On Thu, Feb 22, 2018 at 11:15 PM, Jim Apple < > jbap...@cloudera.com > > > > > > >>>> wrote: > > > >>>> > > > > >>>> > > What is the host OS? What is the docker OS? How much space is > > left > > > >>>> on > > > >>>> > disk > > > >>>> > > in each environment? > > > >>>> > > > > > >>>> > > On Thu, Feb 22, 2018 at 8:05 PM, Joseph Koshakow < > > > kosh...@gmail.com > > > >>>> > > > > >>>> > > wrote: > > > >>>> > > > > > >>>> > > > Hi, > > > >>>> > > > > > > >>>> > > > I've been trying to set up my development environment using > > > >>>> docker by > > > >>>> > > > following the following instructions: https://cwiki.ap > > > >>>> > > > ache.org/confluence/display/IMPALA/Docker+for+Impala+ > > Developers > > > >>>> > > > I'm able to get to the step where you run `source > > > >>>> > > > ~/Impala/bin/bootsrap_development.sh`. Everything runs fine > > > >>>> until I get > > > >>>> > > > to this part: > > > >>>> > > > Loading TPC-DS data (logging to > /home/impdev/Impala/logs/data_ > > > >>>> > > > loading/load-tpcds.log)... > > > >>>> > > > > > > >>>> > > > That hangs for about 30 minutes and then prints the > following > > > >>>> (Also > > > >>>> > > > attached) before hanging again indefinitely until I quit > out. > > > I've > > > >>>> > tried > > > >>>> > > > multiple times and even deleted the docker image and started > > > >>>> over. Any > > > >>>> > > > ideas? > > > >>>> > > > > > > >>>> > > > FAILED (Took: 26 min 21 sec) > > > >>>> > > > 'load-data tpch core' failed. Tail of log: > > > >>>> > > > INVALIDATE METADATA tpch_kudu.nation > > > >>>> > > > > > > >>>> > > > (invalidate-tpch-core-impala-generated.sql): > > > >>>> > > > INVALIDATE METADATA tpch_kudu.region > > > >>>> > > > > > > >>>> > > > (invalidate-tpch-core-impala-generated.sql): > > > >>>> > > > INVALIDATE METADATA tpch_kudu.orders > > > >>>> > > > > > > >>>> > > > (invalidate-tpch-core-impala-generated.sql): > > > >>>> > > > INVALIDATE METADATA tpch_kudu.customer > > > >>>> > > > > > > >>>> > > > (invalidate-tpch-core-impala-generated.sql): > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): > > > >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM > > > >>>> > tpch.lineitem > > > >>>> > > > > > > >>>> > > > (load-tpch-core-impala-generated-kudu-none-none.sql): > > > >>>> > > > INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM > > tpch.lineitem > > > >>>> > > > > > > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): > > > >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM > > tpch.part > > > >>>> > > > > > > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): > > > >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM > > > >>>> > tpch.partsupp > > > >>>> > > > > > > >>>> > > > Data Loading from Impala failed with error: > > > >>>> ImpalaBeeswaxException: > > > >>>> > > > Query aborted:Kudu error(s) reported, first error: Timed > out: > > > >>>> Failed > > > >>>> > to > > > >>>> > > > write batch of 23532 ops to tablet > > > 3b2c6a27489d43028912a69ca7be8b > > > >>>> 71 > > > >>>> > > after > > > >>>> > > > 4 attempt(s): GetTableLocations { table: > > > >>>> 'impala::tpch_kudu.lineitem', > > > >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: > > > >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after > > 10.000s > > > >>>> (SENT) > > > >>>> > > > > > > >>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: > > > >>>> Failed to > > > >>>> > > > write batch of 23532 ops to tablet > > > 3b2c6a27489d43028912a69ca7be8b > > > >>>> 71 > > > >>>> > > after > > > >>>> > > > 4 attempt(s): GetTableLocations { table: > > > >>>> 'impala::tpch_kudu.lineitem', > > > >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: > > > >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after > > 10.000s > > > >>>> (SENT) > > > >>>> > > (1 > > > >>>> > > > of 23532 similar) > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > Traceback (most recent call last): > > > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): > > > >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM > > > >>>> > tpch.supplier > > > >>>> > > > File "/home/impdev/Impala/bin/load-data.py", line 179, in > > > >>>> > > > exec_impala_query_from_file > > > >>>> > > > > > > >>>> > > > result = impala_client.execute(query) > > > >>>> > > > File "/home/impdev/Impala/tests/ > beeswax/impala_beeswax.py", > > > >>>> line 173, > > > >>>> > > > in execute > > > >>>> > > > handle = self.__execute_query(query_string.strip(), > > > >>>> user=user) > > > >>>> > > > File "/home/impdev/Impala/tests/ > beeswax/impala_beeswax.py", > > > >>>> line 341, > > > >>>> > > > in __execute_query > > > >>>> > > > self.wait_for_completion(handle) > > > >>>> > > > File "/home/impdev/Impala/tests/ > beeswax/impala_beeswax.py", > > > >>>> line 361, > > > >>>> > > > in wait_for_completion > > > >>>> > > > raise ImpalaBeeswaxException("Query aborted:" + > error_log, > > > >>>> None) > > > >>>> > > > ImpalaBeeswaxException: ImpalaBeeswaxException: > > > >>>> > > > Query aborted:Kudu error(s) reported, first error: Timed > out: > > > >>>> Failed > > > >>>> > to > > > >>>> > > > write batch of 23532 ops to tablet > > > 3b2c6a27489d43028912a69ca7be8b > > > >>>> 71 > > > >>>> > > after > > > >>>> > > > 4 attempt(s): GetTableLocations { table: > > > >>>> 'impala::tpch_kudu.lineitem', > > > >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: > > > >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after > > 10.000s > > > >>>> (SENT) > > > >>>> > > > > > > >>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: > > > >>>> Failed to > > > >>>> > > > write batch of 23532 ops to tablet > > > 3b2c6a27489d43028912a69ca7be8b > > > >>>> 71 > > > >>>> > > after > > > >>>> > > > 4 attempt(s): GetTableLocations { table: > > > >>>> 'impala::tpch_kudu.lineitem', > > > >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: > > > >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after > > 10.000s > > > >>>> (SENT) > > > >>>> > > (1 > > > >>>> > > > of 23532 similar) > > > >>>> > > > > > > >>>> > > > Thanks, > > > >>>> > > > Joe Koshakow > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>> > > > >>> > > > >> > > > > > > > > > >
