Hi,
*results of df -h:*
impdev@c4fdb0931b2d:~/Impala$ df -h
Filesystem Size Used Avail Use% Mounted on
none 300G 107G 180G 38% /
tmpfs 5.9G 0 5.9G 0% /dev
tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup
/dev/sda9 300G 107G 180G 38% /etc/hosts
shm 64M 0 64M 0% /dev/shm
*Here is the error I'm getting:*
Starting Impala cluster (logging to
/home/impdev/Impala/logs/data_loading/start-impala-cluster.log)...
Starting Impala cluster OK (Took: 0 min 21 sec)
Setting up HDFS environment (logging to
/home/impdev/Impala/logs/data_loading/setup-hdfs-env.log)...
Setting up HDFS environment OK (Took: 0 min 9 sec)
Loading custom schemas (logging to
/home/impdev/Impala/logs/data_loading/load-custom-schemas.log)...
Loading custom schemas OK (Took: 0 min 36 sec)
Started Loading functional-query data in background; pid 13490.
Loading functional-query data (logging to
/home/impdev/Impala/logs/data_loading/load-functional-query.log)...
Started Loading TPC-H data in background; pid 13491.
Started Loading TPC-DS data in background; pid 13492.
Loading TPC-H data (logging to
/home/impdev/Impala/logs/data_loading/load-tpch.log)...
Loading TPC-DS data (logging to
/home/impdev/Impala/logs/data_loading/load-tpcds.log)...
FAILED (Took: 28 min 11 sec)
'load-data tpch core' failed. Tail of log:
INVALIDATE METADATA tpch_kudu.partsupp
(invalidate-tpch-core-impala-generated.sql):
INVALIDATE METADATA tpch_kudu.supplier
(invalidate-tpch-core-impala-generated.sql):
INVALIDATE METADATA tpch_kudu.nation
(invalidate-tpch-core-impala-generated.sql):
INVALIDATE METADATA tpch_kudu.region
(invalidate-tpch-core-impala-generated.sql):
INVALIDATE METADATA tpch_kudu.orders
(invalidate-tpch-core-impala-generated.sql):
INVALIDATE METADATA tpch_kudu.customer
(invalidate-tpch-core-impala-generated.sql):
(load-tpch-core-impala-generated-kudu-none-none.sql):
INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem
(load-tpch-core-impala-generated-parquet-none-none.sql):
INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM tpch.lineitem
(load-tpch-core-impala-generated-parquet-none-none.sql):
INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part
Data Loading from Impala failed with error: ImpalaBeeswaxException:
Query aborted:Kudu error(s) reported, first error: Timed out: Failed to
write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce after 4
attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem',
partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client
connection negotiation failed: client connection to 127.0.0.1:7051
Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to
write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce after 4
attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem',
partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client
connection negotiation failed: client connection to 127.0.0.1:7051 (1 of
78325 similar)
Traceback (most recent call last):
File "/home/impdev/Impala/bin/load-data.py", line 179, in
exec_impala_query_from_file
result = impala_client.execute(query)
File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 173, in
execute
handle = self.__execute_query(query_string.strip(), user=user)
File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 341, in
__execute_query
self.wait_for_completion(handle)
File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 361, in
wait_for_completion
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
ImpalaBeeswaxException: ImpalaBeeswaxException:
Query aborted:Kudu error(s) reported, first error: Timed out: Failed to
write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce after 4
attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem',
partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client
connection negotiation failed: client connection to 127.0.0.1:7051
Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to
write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce after 4
attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem',
partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client
connection negotiation failed: client connection to 127.0.0.1:7051 (1 of
78325 similar)
If you'd like I can copy and paste the entire output. Thanks for the help.
On Sat, Apr 7, 2018 at 9:44 PM, Jim Apple <jbap...@cloudera.com> wrote:
> The attachment didn't attach. This is a common issue on our mailing lists.
> I've started another thread about it.
>
> I'll try to reproduce this. My first suspicions are disk space, networking
> restrictions, and https://issues.apache.org/jira/browse/IMPALA-6737
>
> Inside of the container, can you run df -h before exiting?
>
> On Sat, Apr 7, 2018 at 5:28 PM, Joseph Koshakow <kosh...@gmail.com> wrote:
>
> > Hi,
> >
> > Tried this again recently and still experiencing the same error. I've
> > attached a copy of all my shell commands and output.
> >
> > On Sat, Feb 24, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com>
> > wrote:
> >
> >> Though I just upgraded to the most recent version. It's still failing
> but
> >> seems to be making it one step farther. I see from the emails that
> there's
> >> some issues with the current build so I'll just wait until that's all
> >> resolved. The output from my most recent attempt is below if you have a
> use
> >> for it.
> >>
> >> Loading custom schemas OK (Took: 0 min 40 sec)
> >> Started Loading functional-query data in background; pid 22229.
> >> Loading functional-query data (logging to /home/impdev/Impala/logs/data_
> >> loading/load-functional-query.log)...
> >> Started Loading TPC-H data in background; pid 22230.
> >> Started Loading TPC-DS data in background; pid 22231.
> >> Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_
> >> loading/load-tpcds.log)...
> >> Loading TPC-H data (logging to /home/impdev/Impala/logs/data_
> >> loading/load-tpch.log)...
> >> FAILED (Took: 25 min 51 sec)
> >> 'load-data tpch core' failed. Tail of log:
> >> INVALIDATE METADATA tpch_kudu.region
> >>
> >> (invalidate-tpch-core-impala-generated.sql):
> >> INVALIDATE METADATA tpch_kudu.orders
> >>
> >> (invalidate-tpch-core-impala-generated.sql):
> >> INVALIDATE METADATA tpch_kudu.customer
> >>
> >> (invalidate-tpch-core-impala-generated.sql):
> >>
> >>
> >> (load-tpch-core-impala-generated-kudu-none-none.sql):
> >> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem
> >>
> >> (load-tpch-core-impala-generated-parquet-none-none.sql):
> >> INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM tpch.lineitem
> >>
> >> (load-tpch-core-impala-generated-parquet-none-none.sql):
> >> INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part
> >>
> >> (load-tpch-core-impala-generated-parquet-none-none.sql):
> >> INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM tpch.partsupp
> >>
> >> (load-tpch-core-impala-generated-parquet-none-none.sql):
> >> INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM tpch.supplier
> >>
> >> Data Loading from Impala failed with error: ImpalaBeeswaxException:
> >> Query aborted:Kudu error(s) reported, first error: Timed out: Failed to
> >> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9
> >> after 4 attempt(s): GetTableLocations { table:
> >> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 }
> >> failed: Client connection negotiation failed: client connection to
> >> 127.0.0.1:7051
> >>
> >> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to
> >> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34
> >> after 4 attempt(s): GetTableLocations { table:
> >> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2),
> >> attempt: 1 } failed: Client connection negotiation failed: client
> >> connection to 127.0.0.1:7051 (1 of 21494 similar)
> >>
> >>
> >> Traceback (most recent call last):
> >> File "/home/impdev/Impala/bin/load-data.py", line 179, in
> >> exec_impala_query_from_file
> >> (load-tpch-core-impala-generated-parquet-none-none.sql):
> >> INSERT OVERWRITE TABLE tpch_parquet.nation SELECT * FROM tpch.nation
> >>
> >> result = impala_client.execute(query)
> >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 173,
> >> in execute
> >> handle = self.__execute_query(query_string.strip(), user=user)
> >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 341,
> >> in __execute_query
> >> self.wait_for_completion(handle)
> >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 361,
> >> in wait_for_completion
> >> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> >> ImpalaBeeswaxException: ImpalaBeeswaxException:
> >> Query aborted:Kudu error(s) reported, first error: Timed out: Failed to
> >> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9
> >> after 4 attempt(s): GetTableLocations { table:
> >> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 }
> >> failed: Client connection negotiation failed: client connection to
> >> 127.0.0.1:7051
> >>
> >> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to
> >> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34
> >> after 4 attempt(s): GetTableLocations { table:
> >> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2),
> >> attempt: 1 } failed: Client connection negotiation failed: client
> >> connection to 127.0.0.1:7051 (1 of 21494 similar)
> >>
> >>
> >> On Fri, Feb 23, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com>
> >> wrote:
> >>
> >>> 8302608bdff15d6e8af324fc885dd0a8c2b44a5d
> >>>
> >>> On Fri, Feb 23, 2018 at 12:41 AM, Jim Apple <jbap...@cloudera.com>
> >>> wrote:
> >>>
> >>>> What commit hash is checked out?
> >>>>
> >>>> On Thu, Feb 22, 2018 at 8:23 PM Joseph Koshakow <kosh...@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > Host OS is Ubunto 16.04.3 LTS. Docker OS is also Ubuntu 16.04 as per
> >>>> > instructions. I have 202.61Gb of free space in the host environment
> >>>> > (304.58Gb size total). I'm not sure how to check how much space is
> >>>> left in
> >>>> > docker containers but the size of the one created for this is 97.81
> >>>> Gb.
> >>>> >
> >>>> > -Joe
> >>>> >
> >>>> > On Thu, Feb 22, 2018 at 11:15 PM, Jim Apple <jbap...@cloudera.com>
> >>>> wrote:
> >>>> >
> >>>> > > What is the host OS? What is the docker OS? How much space is left
> >>>> on
> >>>> > disk
> >>>> > > in each environment?
> >>>> > >
> >>>> > > On Thu, Feb 22, 2018 at 8:05 PM, Joseph Koshakow <
> kosh...@gmail.com
> >>>> >
> >>>> > > wrote:
> >>>> > >
> >>>> > > > Hi,
> >>>> > > >
> >>>> > > > I've been trying to set up my development environment using
> >>>> docker by
> >>>> > > > following the following instructions: https://cwiki.ap
> >>>> > > > ache.org/confluence/display/IMPALA/Docker+for+Impala+Developers
> >>>> > > > I'm able to get to the step where you run `source
> >>>> > > > ~/Impala/bin/bootsrap_development.sh`. Everything runs fine
> >>>> until I get
> >>>> > > > to this part:
> >>>> > > > Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_
> >>>> > > > loading/load-tpcds.log)...
> >>>> > > >
> >>>> > > > That hangs for about 30 minutes and then prints the following
> >>>> (Also
> >>>> > > > attached) before hanging again indefinitely until I quit out.
> I've
> >>>> > tried
> >>>> > > > multiple times and even deleted the docker image and started
> >>>> over. Any
> >>>> > > > ideas?
> >>>> > > >
> >>>> > > > FAILED (Took: 26 min 21 sec)
> >>>> > > > 'load-data tpch core' failed. Tail of log:
> >>>> > > > INVALIDATE METADATA tpch_kudu.nation
> >>>> > > >
> >>>> > > > (invalidate-tpch-core-impala-generated.sql):
> >>>> > > > INVALIDATE METADATA tpch_kudu.region
> >>>> > > >
> >>>> > > > (invalidate-tpch-core-impala-generated.sql):
> >>>> > > > INVALIDATE METADATA tpch_kudu.orders
> >>>> > > >
> >>>> > > > (invalidate-tpch-core-impala-generated.sql):
> >>>> > > > INVALIDATE METADATA tpch_kudu.customer
> >>>> > > >
> >>>> > > > (invalidate-tpch-core-impala-generated.sql):
> >>>> > > >
> >>>> > > >
> >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM
> >>>> > tpch.lineitem
> >>>> > > >
> >>>> > > > (load-tpch-core-impala-generated-kudu-none-none.sql):
> >>>> > > > INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem
> >>>> > > >
> >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part
> >>>> > > >
> >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM
> >>>> > tpch.partsupp
> >>>> > > >
> >>>> > > > Data Loading from Impala failed with error:
> >>>> ImpalaBeeswaxException:
> >>>> > > > Query aborted:Kudu error(s) reported, first error: Timed out:
> >>>> Failed
> >>>> > to
> >>>> > > > write batch of 23532 ops to tablet
> 3b2c6a27489d43028912a69ca7be8b
> >>>> 71
> >>>> > > after
> >>>> > > > 4 attempt(s): GetTableLocations { table:
> >>>> 'impala::tpch_kudu.lineitem',
> >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s
> >>>> (SENT)
> >>>> > > >
> >>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out:
> >>>> Failed to
> >>>> > > > write batch of 23532 ops to tablet
> 3b2c6a27489d43028912a69ca7be8b
> >>>> 71
> >>>> > > after
> >>>> > > > 4 attempt(s): GetTableLocations { table:
> >>>> 'impala::tpch_kudu.lineitem',
> >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s
> >>>> (SENT)
> >>>> > > (1
> >>>> > > > of 23532 similar)
> >>>> > > >
> >>>> > > >
> >>>> > > > Traceback (most recent call last):
> >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM
> >>>> > tpch.supplier
> >>>> > > > File "/home/impdev/Impala/bin/load-data.py", line 179, in
> >>>> > > > exec_impala_query_from_file
> >>>> > > >
> >>>> > > > result = impala_client.execute(query)
> >>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py",
> >>>> line 173,
> >>>> > > > in execute
> >>>> > > > handle = self.__execute_query(query_string.strip(),
> >>>> user=user)
> >>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py",
> >>>> line 341,
> >>>> > > > in __execute_query
> >>>> > > > self.wait_for_completion(handle)
> >>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py",
> >>>> line 361,
> >>>> > > > in wait_for_completion
> >>>> > > > raise ImpalaBeeswaxException("Query aborted:" + error_log,
> >>>> None)
> >>>> > > > ImpalaBeeswaxException: ImpalaBeeswaxException:
> >>>> > > > Query aborted:Kudu error(s) reported, first error: Timed out:
> >>>> Failed
> >>>> > to
> >>>> > > > write batch of 23532 ops to tablet
> 3b2c6a27489d43028912a69ca7be8b
> >>>> 71
> >>>> > > after
> >>>> > > > 4 attempt(s): GetTableLocations { table:
> >>>> 'impala::tpch_kudu.lineitem',
> >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s
> >>>> (SENT)
> >>>> > > >
> >>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out:
> >>>> Failed to
> >>>> > > > write batch of 23532 ops to tablet
> 3b2c6a27489d43028912a69ca7be8b
> >>>> 71
> >>>> > > after
> >>>> > > > 4 attempt(s): GetTableLocations { table:
> >>>> 'impala::tpch_kudu.lineitem',
> >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after 10.000s
> >>>> (SENT)
> >>>> > > (1
> >>>> > > > of 23532 similar)
> >>>> > > >
> >>>> > > > Thanks,
> >>>> > > > Joe Koshakow
> >>>> > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >
>
