I was unable to reproduce this in a fresh ec2 instance. I ran: sudo docker run --interactive --privileged --tty ubuntu:16.04 # inside the container apt-get update apt-get install -y sudo adduser --disabled-password --gecos '' impdev echo 'impdev ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers sudo su - impdev # as impdev sudo apt install -y git git clone https://git-wip-us.apache.org/repos/asf/impala.git ~/Impala
time ~/Impala/bin/bootstrap_development.sh On Sat, Apr 7, 2018 at 6:44 PM, Jim Apple <jbap...@cloudera.com> wrote: > The attachment didn't attach. This is a common issue on our mailing lists. > I've started another thread about it. > > I'll try to reproduce this. My first suspicions are disk space, networking > restrictions, and https://issues.apache.org/jira/browse/IMPALA-6737 > > Inside of the container, can you run df -h before exiting? > > On Sat, Apr 7, 2018 at 5:28 PM, Joseph Koshakow <kosh...@gmail.com> wrote: > >> Hi, >> >> Tried this again recently and still experiencing the same error. I've >> attached a copy of all my shell commands and output. >> >> On Sat, Feb 24, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com> >> wrote: >> >>> Though I just upgraded to the most recent version. It's still failing >>> but seems to be making it one step farther. I see from the emails that >>> there's some issues with the current build so I'll just wait until that's >>> all resolved. The output from my most recent attempt is below if you have a >>> use for it. >>> >>> Loading custom schemas OK (Took: 0 min 40 sec) >>> Started Loading functional-query data in background; pid 22229. >>> Loading functional-query data (logging to /home/impdev/Impala/logs/data_ >>> loading/load-functional-query.log)... >>> Started Loading TPC-H data in background; pid 22230. >>> Started Loading TPC-DS data in background; pid 22231. >>> Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_ >>> loading/load-tpcds.log)... >>> Loading TPC-H data (logging to /home/impdev/Impala/logs/data_ >>> loading/load-tpch.log)... >>> FAILED (Took: 25 min 51 sec) >>> 'load-data tpch core' failed. Tail of log: >>> INVALIDATE METADATA tpch_kudu.region >>> >>> (invalidate-tpch-core-impala-generated.sql): >>> INVALIDATE METADATA tpch_kudu.orders >>> >>> (invalidate-tpch-core-impala-generated.sql): >>> INVALIDATE METADATA tpch_kudu.customer >>> >>> (invalidate-tpch-core-impala-generated.sql): >>> >>> >>> (load-tpch-core-impala-generated-kudu-none-none.sql): >>> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem >>> >>> (load-tpch-core-impala-generated-parquet-none-none.sql): >>> INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM tpch.lineitem >>> >>> (load-tpch-core-impala-generated-parquet-none-none.sql): >>> INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part >>> >>> (load-tpch-core-impala-generated-parquet-none-none.sql): >>> INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM tpch.partsupp >>> >>> (load-tpch-core-impala-generated-parquet-none-none.sql): >>> INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM tpch.supplier >>> >>> Data Loading from Impala failed with error: ImpalaBeeswaxException: >>> Query aborted:Kudu error(s) reported, first error: Timed out: Failed to >>> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9 >>> after 4 attempt(s): GetTableLocations { table: >>> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 } >>> failed: Client connection negotiation failed: client connection to >>> 127.0.0.1:7051 >>> >>> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to >>> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34 >>> after 4 attempt(s): GetTableLocations { table: >>> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2), >>> attempt: 1 } failed: Client connection negotiation failed: client >>> connection to 127.0.0.1:7051 (1 of 21494 similar) >>> >>> >>> Traceback (most recent call last): >>> File "/home/impdev/Impala/bin/load-data.py", line 179, in >>> exec_impala_query_from_file >>> (load-tpch-core-impala-generated-parquet-none-none.sql): >>> INSERT OVERWRITE TABLE tpch_parquet.nation SELECT * FROM tpch.nation >>> >>> result = impala_client.execute(query) >>> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 173, >>> in execute >>> handle = self.__execute_query(query_string.strip(), user=user) >>> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 341, >>> in __execute_query >>> self.wait_for_completion(handle) >>> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 361, >>> in wait_for_completion >>> raise ImpalaBeeswaxException("Query aborted:" + error_log, None) >>> ImpalaBeeswaxException: ImpalaBeeswaxException: >>> Query aborted:Kudu error(s) reported, first error: Timed out: Failed to >>> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9 >>> after 4 attempt(s): GetTableLocations { table: >>> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 } >>> failed: Client connection negotiation failed: client connection to >>> 127.0.0.1:7051 >>> >>> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to >>> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34 >>> after 4 attempt(s): GetTableLocations { table: >>> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2), >>> attempt: 1 } failed: Client connection negotiation failed: client >>> connection to 127.0.0.1:7051 (1 of 21494 similar) >>> >>> >>> On Fri, Feb 23, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com> >>> wrote: >>> >>>> 8302608bdff15d6e8af324fc885dd0a8c2b44a5d >>>> >>>> On Fri, Feb 23, 2018 at 12:41 AM, Jim Apple <jbap...@cloudera.com> >>>> wrote: >>>> >>>>> What commit hash is checked out? >>>>> >>>>> On Thu, Feb 22, 2018 at 8:23 PM Joseph Koshakow <kosh...@gmail.com> >>>>> wrote: >>>>> >>>>> > Host OS is Ubunto 16.04.3 LTS. Docker OS is also Ubuntu 16.04 as per >>>>> > instructions. I have 202.61Gb of free space in the host environment >>>>> > (304.58Gb size total). I'm not sure how to check how much space is >>>>> left in >>>>> > docker containers but the size of the one created for this is 97.81 >>>>> Gb. >>>>> > >>>>> > -Joe >>>>> > >>>>> > On Thu, Feb 22, 2018 at 11:15 PM, Jim Apple <jbap...@cloudera.com> >>>>> wrote: >>>>> > >>>>> > > What is the host OS? What is the docker OS? How much space is left >>>>> on >>>>> > disk >>>>> > > in each environment? >>>>> > > >>>>> > > On Thu, Feb 22, 2018 at 8:05 PM, Joseph Koshakow < >>>>> kosh...@gmail.com> >>>>> > > wrote: >>>>> > > >>>>> > > > Hi, >>>>> > > > >>>>> > > > I've been trying to set up my development environment using >>>>> docker by >>>>> > > > following the following instructions: https://cwiki.ap >>>>> > > > ache.org/confluence/display/IMPALA/Docker+for+Impala+Developers >>>>> > > > I'm able to get to the step where you run `source >>>>> > > > ~/Impala/bin/bootsrap_development.sh`. Everything runs fine >>>>> until I get >>>>> > > > to this part: >>>>> > > > Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_ >>>>> > > > loading/load-tpcds.log)... >>>>> > > > >>>>> > > > That hangs for about 30 minutes and then prints the following >>>>> (Also >>>>> > > > attached) before hanging again indefinitely until I quit out. >>>>> I've >>>>> > tried >>>>> > > > multiple times and even deleted the docker image and started >>>>> over. Any >>>>> > > > ideas? >>>>> > > > >>>>> > > > FAILED (Took: 26 min 21 sec) >>>>> > > > 'load-data tpch core' failed. Tail of log: >>>>> > > > INVALIDATE METADATA tpch_kudu.nation >>>>> > > > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): >>>>> > > > INVALIDATE METADATA tpch_kudu.region >>>>> > > > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): >>>>> > > > INVALIDATE METADATA tpch_kudu.orders >>>>> > > > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): >>>>> > > > INVALIDATE METADATA tpch_kudu.customer >>>>> > > > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): >>>>> > > > >>>>> > > > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM >>>>> > tpch.lineitem >>>>> > > > >>>>> > > > (load-tpch-core-impala-generated-kudu-none-none.sql): >>>>> > > > INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem >>>>> > > > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part >>>>> > > > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM >>>>> > tpch.partsupp >>>>> > > > >>>>> > > > Data Loading from Impala failed with error: >>>>> ImpalaBeeswaxException: >>>>> > > > Query aborted:Kudu error(s) reported, first error: Timed out: >>>>> Failed >>>>> > to >>>>> > > > write batch of 23532 ops to tablet 3b2c6a27489d43028912a69ca7be8b >>>>> 71 >>>>> > > after >>>>> > > > 4 attempt(s): GetTableLocations { table: >>>>> 'impala::tpch_kudu.lineitem', >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s >>>>> (SENT) >>>>> > > > >>>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: >>>>> Failed to >>>>> > > > write batch of 23532 ops to tablet 3b2c6a27489d43028912a69ca7be8b >>>>> 71 >>>>> > > after >>>>> > > > 4 attempt(s): GetTableLocations { table: >>>>> 'impala::tpch_kudu.lineitem', >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s >>>>> (SENT) >>>>> > > (1 >>>>> > > > of 23532 similar) >>>>> > > > >>>>> > > > >>>>> > > > Traceback (most recent call last): >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM >>>>> > tpch.supplier >>>>> > > > File "/home/impdev/Impala/bin/load-data.py", line 179, in >>>>> > > > exec_impala_query_from_file >>>>> > > > >>>>> > > > result = impala_client.execute(query) >>>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", >>>>> line 173, >>>>> > > > in execute >>>>> > > > handle = self.__execute_query(query_string.strip(), >>>>> user=user) >>>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", >>>>> line 341, >>>>> > > > in __execute_query >>>>> > > > self.wait_for_completion(handle) >>>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", >>>>> line 361, >>>>> > > > in wait_for_completion >>>>> > > > raise ImpalaBeeswaxException("Query aborted:" + error_log, >>>>> None) >>>>> > > > ImpalaBeeswaxException: ImpalaBeeswaxException: >>>>> > > > Query aborted:Kudu error(s) reported, first error: Timed out: >>>>> Failed >>>>> > to >>>>> > > > write batch of 23532 ops to tablet 3b2c6a27489d43028912a69ca7be8b >>>>> 71 >>>>> > > after >>>>> > > > 4 attempt(s): GetTableLocations { table: >>>>> 'impala::tpch_kudu.lineitem', >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s >>>>> (SENT) >>>>> > > > >>>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: >>>>> Failed to >>>>> > > > write batch of 23532 ops to tablet 3b2c6a27489d43028912a69ca7be8b >>>>> 71 >>>>> > > after >>>>> > > > 4 attempt(s): GetTableLocations { table: >>>>> 'impala::tpch_kudu.lineitem', >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s >>>>> (SENT) >>>>> > > (1 >>>>> > > > of 23532 similar) >>>>> > > > >>>>> > > > Thanks, >>>>> > > > Joe Koshakow >>>>> > > > >>>>> > > > >>>>> > > >>>>> > >>>>> >>>> >>>> >>> >> >
