The attachment didn't attach. This is a common issue on our mailing lists. I've started another thread about it.
I'll try to reproduce this. My first suspicions are disk space, networking restrictions, and https://issues.apache.org/jira/browse/IMPALA-6737 Inside of the container, can you run df -h before exiting? On Sat, Apr 7, 2018 at 5:28 PM, Joseph Koshakow <kosh...@gmail.com> wrote: > Hi, > > Tried this again recently and still experiencing the same error. I've > attached a copy of all my shell commands and output. > > On Sat, Feb 24, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com> > wrote: > >> Though I just upgraded to the most recent version. It's still failing but >> seems to be making it one step farther. I see from the emails that there's >> some issues with the current build so I'll just wait until that's all >> resolved. The output from my most recent attempt is below if you have a use >> for it. >> >> Loading custom schemas OK (Took: 0 min 40 sec) >> Started Loading functional-query data in background; pid 22229. >> Loading functional-query data (logging to /home/impdev/Impala/logs/data_ >> loading/load-functional-query.log)... >> Started Loading TPC-H data in background; pid 22230. >> Started Loading TPC-DS data in background; pid 22231. >> Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_ >> loading/load-tpcds.log)... >> Loading TPC-H data (logging to /home/impdev/Impala/logs/data_ >> loading/load-tpch.log)... >> FAILED (Took: 25 min 51 sec) >> 'load-data tpch core' failed. Tail of log: >> INVALIDATE METADATA tpch_kudu.region >> >> (invalidate-tpch-core-impala-generated.sql): >> INVALIDATE METADATA tpch_kudu.orders >> >> (invalidate-tpch-core-impala-generated.sql): >> INVALIDATE METADATA tpch_kudu.customer >> >> (invalidate-tpch-core-impala-generated.sql): >> >> >> (load-tpch-core-impala-generated-kudu-none-none.sql): >> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem >> >> (load-tpch-core-impala-generated-parquet-none-none.sql): >> INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM tpch.lineitem >> >> (load-tpch-core-impala-generated-parquet-none-none.sql): >> INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part >> >> (load-tpch-core-impala-generated-parquet-none-none.sql): >> INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM tpch.partsupp >> >> (load-tpch-core-impala-generated-parquet-none-none.sql): >> INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM tpch.supplier >> >> Data Loading from Impala failed with error: ImpalaBeeswaxException: >> Query aborted:Kudu error(s) reported, first error: Timed out: Failed to >> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9 >> after 4 attempt(s): GetTableLocations { table: >> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 } >> failed: Client connection negotiation failed: client connection to >> 127.0.0.1:7051 >> >> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to >> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34 >> after 4 attempt(s): GetTableLocations { table: >> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2), >> attempt: 1 } failed: Client connection negotiation failed: client >> connection to 127.0.0.1:7051 (1 of 21494 similar) >> >> >> Traceback (most recent call last): >> File "/home/impdev/Impala/bin/load-data.py", line 179, in >> exec_impala_query_from_file >> (load-tpch-core-impala-generated-parquet-none-none.sql): >> INSERT OVERWRITE TABLE tpch_parquet.nation SELECT * FROM tpch.nation >> >> result = impala_client.execute(query) >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 173, >> in execute >> handle = self.__execute_query(query_string.strip(), user=user) >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 341, >> in __execute_query >> self.wait_for_completion(handle) >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 361, >> in wait_for_completion >> raise ImpalaBeeswaxException("Query aborted:" + error_log, None) >> ImpalaBeeswaxException: ImpalaBeeswaxException: >> Query aborted:Kudu error(s) reported, first error: Timed out: Failed to >> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9 >> after 4 attempt(s): GetTableLocations { table: >> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 } >> failed: Client connection negotiation failed: client connection to >> 127.0.0.1:7051 >> >> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to >> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34 >> after 4 attempt(s): GetTableLocations { table: >> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2), >> attempt: 1 } failed: Client connection negotiation failed: client >> connection to 127.0.0.1:7051 (1 of 21494 similar) >> >> >> On Fri, Feb 23, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com> >> wrote: >> >>> 8302608bdff15d6e8af324fc885dd0a8c2b44a5d >>> >>> On Fri, Feb 23, 2018 at 12:41 AM, Jim Apple <jbap...@cloudera.com> >>> wrote: >>> >>>> What commit hash is checked out? >>>> >>>> On Thu, Feb 22, 2018 at 8:23 PM Joseph Koshakow <kosh...@gmail.com> >>>> wrote: >>>> >>>> > Host OS is Ubunto 16.04.3 LTS. Docker OS is also Ubuntu 16.04 as per >>>> > instructions. I have 202.61Gb of free space in the host environment >>>> > (304.58Gb size total). I'm not sure how to check how much space is >>>> left in >>>> > docker containers but the size of the one created for this is 97.81 >>>> Gb. >>>> > >>>> > -Joe >>>> > >>>> > On Thu, Feb 22, 2018 at 11:15 PM, Jim Apple <jbap...@cloudera.com> >>>> wrote: >>>> > >>>> > > What is the host OS? What is the docker OS? How much space is left >>>> on >>>> > disk >>>> > > in each environment? >>>> > > >>>> > > On Thu, Feb 22, 2018 at 8:05 PM, Joseph Koshakow <kosh...@gmail.com >>>> > >>>> > > wrote: >>>> > > >>>> > > > Hi, >>>> > > > >>>> > > > I've been trying to set up my development environment using >>>> docker by >>>> > > > following the following instructions: https://cwiki.ap >>>> > > > ache.org/confluence/display/IMPALA/Docker+for+Impala+Developers >>>> > > > I'm able to get to the step where you run `source >>>> > > > ~/Impala/bin/bootsrap_development.sh`. Everything runs fine >>>> until I get >>>> > > > to this part: >>>> > > > Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_ >>>> > > > loading/load-tpcds.log)... >>>> > > > >>>> > > > That hangs for about 30 minutes and then prints the following >>>> (Also >>>> > > > attached) before hanging again indefinitely until I quit out. I've >>>> > tried >>>> > > > multiple times and even deleted the docker image and started >>>> over. Any >>>> > > > ideas? >>>> > > > >>>> > > > FAILED (Took: 26 min 21 sec) >>>> > > > 'load-data tpch core' failed. Tail of log: >>>> > > > INVALIDATE METADATA tpch_kudu.nation >>>> > > > >>>> > > > (invalidate-tpch-core-impala-generated.sql): >>>> > > > INVALIDATE METADATA tpch_kudu.region >>>> > > > >>>> > > > (invalidate-tpch-core-impala-generated.sql): >>>> > > > INVALIDATE METADATA tpch_kudu.orders >>>> > > > >>>> > > > (invalidate-tpch-core-impala-generated.sql): >>>> > > > INVALIDATE METADATA tpch_kudu.customer >>>> > > > >>>> > > > (invalidate-tpch-core-impala-generated.sql): >>>> > > > >>>> > > > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM >>>> > tpch.lineitem >>>> > > > >>>> > > > (load-tpch-core-impala-generated-kudu-none-none.sql): >>>> > > > INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem >>>> > > > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part >>>> > > > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM >>>> > tpch.partsupp >>>> > > > >>>> > > > Data Loading from Impala failed with error: >>>> ImpalaBeeswaxException: >>>> > > > Query aborted:Kudu error(s) reported, first error: Timed out: >>>> Failed >>>> > to >>>> > > > write batch of 23532 ops to tablet 3b2c6a27489d43028912a69ca7be8b >>>> 71 >>>> > > after >>>> > > > 4 attempt(s): GetTableLocations { table: >>>> 'impala::tpch_kudu.lineitem', >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s >>>> (SENT) >>>> > > > >>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: >>>> Failed to >>>> > > > write batch of 23532 ops to tablet 3b2c6a27489d43028912a69ca7be8b >>>> 71 >>>> > > after >>>> > > > 4 attempt(s): GetTableLocations { table: >>>> 'impala::tpch_kudu.lineitem', >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s >>>> (SENT) >>>> > > (1 >>>> > > > of 23532 similar) >>>> > > > >>>> > > > >>>> > > > Traceback (most recent call last): >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM >>>> > tpch.supplier >>>> > > > File "/home/impdev/Impala/bin/load-data.py", line 179, in >>>> > > > exec_impala_query_from_file >>>> > > > >>>> > > > result = impala_client.execute(query) >>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", >>>> line 173, >>>> > > > in execute >>>> > > > handle = self.__execute_query(query_string.strip(), >>>> user=user) >>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", >>>> line 341, >>>> > > > in __execute_query >>>> > > > self.wait_for_completion(handle) >>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", >>>> line 361, >>>> > > > in wait_for_completion >>>> > > > raise ImpalaBeeswaxException("Query aborted:" + error_log, >>>> None) >>>> > > > ImpalaBeeswaxException: ImpalaBeeswaxException: >>>> > > > Query aborted:Kudu error(s) reported, first error: Timed out: >>>> Failed >>>> > to >>>> > > > write batch of 23532 ops to tablet 3b2c6a27489d43028912a69ca7be8b >>>> 71 >>>> > > after >>>> > > > 4 attempt(s): GetTableLocations { table: >>>> 'impala::tpch_kudu.lineitem', >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s >>>> (SENT) >>>> > > > >>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: >>>> Failed to >>>> > > > write batch of 23532 ops to tablet 3b2c6a27489d43028912a69ca7be8b >>>> 71 >>>> > > after >>>> > > > 4 attempt(s): GetTableLocations { table: >>>> 'impala::tpch_kudu.lineitem', >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s >>>> (SENT) >>>> > > (1 >>>> > > > of 23532 similar) >>>> > > > >>>> > > > Thanks, >>>> > > > Joe Koshakow >>>> > > > >>>> > > > >>>> > > >>>> > >>>> >>> >>> >> >
