Are you able to bootstrap a kudu development environment inside the
container? I haven't tried this before, but maybe there's something going
on with the port number?
On Sat, Apr 7, 2018 at 6:53 PM, Joseph Koshakow <kosh...@gmail.com> wrote:
> Hi,
>
> *results of df -h:*
> impdev@c4fdb0931b2d:~/Impala$ df -h
> Filesystem Size Used Avail Use% Mounted on
> none 300G 107G 180G 38% /
> tmpfs 5.9G 0 5.9G 0% /dev
> tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup
> /dev/sda9 300G 107G 180G 38% /etc/hosts
> shm 64M 0 64M 0% /dev/shm
>
>
> *Here is the error I'm getting:*
> Starting Impala cluster (logging to
> /home/impdev/Impala/logs/data_loading/start-impala-cluster.log)...
> Starting Impala cluster OK (Took: 0 min 21 sec)
> Setting up HDFS environment (logging to
> /home/impdev/Impala/logs/data_loading/setup-hdfs-env.log)...
> Setting up HDFS environment OK (Took: 0 min 9 sec)
> Loading custom schemas (logging to
> /home/impdev/Impala/logs/data_loading/load-custom-schemas.log)...
> Loading custom schemas OK (Took: 0 min 36 sec)
> Started Loading functional-query data in background; pid 13490.
> Loading functional-query data (logging to
> /home/impdev/Impala/logs/data_loading/load-functional-query.log)...
> Started Loading TPC-H data in background; pid 13491.
> Started Loading TPC-DS data in background; pid 13492.
> Loading TPC-H data (logging to
> /home/impdev/Impala/logs/data_loading/load-tpch.log)...
> Loading TPC-DS data (logging to
> /home/impdev/Impala/logs/data_loading/load-tpcds.log)...
> FAILED (Took: 28 min 11 sec)
> 'load-data tpch core' failed. Tail of log:
> INVALIDATE METADATA tpch_kudu.partsupp
>
> (invalidate-tpch-core-impala-generated.sql):
> INVALIDATE METADATA tpch_kudu.supplier
>
> (invalidate-tpch-core-impala-generated.sql):
> INVALIDATE METADATA tpch_kudu.nation
>
> (invalidate-tpch-core-impala-generated.sql):
> INVALIDATE METADATA tpch_kudu.region
>
> (invalidate-tpch-core-impala-generated.sql):
> INVALIDATE METADATA tpch_kudu.orders
>
> (invalidate-tpch-core-impala-generated.sql):
> INVALIDATE METADATA tpch_kudu.customer
>
> (invalidate-tpch-core-impala-generated.sql):
>
>
> (load-tpch-core-impala-generated-kudu-none-none.sql):
> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem
>
> (load-tpch-core-impala-generated-parquet-none-none.sql):
> INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM tpch.lineitem
>
> (load-tpch-core-impala-generated-parquet-none-none.sql):
> INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part
>
> Data Loading from Impala failed with error: ImpalaBeeswaxException:
> Query aborted:Kudu error(s) reported, first error: Timed out: Failed to
> write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce after
> 4
> attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem',
> partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client
> connection negotiation failed: client connection to 127.0.0.1:7051
>
> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to
> write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce after
> 4
> attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem',
> partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client
> connection negotiation failed: client connection to 127.0.0.1:7051 (1 of
> 78325 similar)
>
>
> Traceback (most recent call last):
> File "/home/impdev/Impala/bin/load-data.py", line 179, in
> exec_impala_query_from_file
> result = impala_client.execute(query)
> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 173, in
> execute
> handle = self.__execute_query(query_string.strip(), user=user)
> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 341, in
> __execute_query
> self.wait_for_completion(handle)
> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line 361, in
> wait_for_completion
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> ImpalaBeeswaxException: ImpalaBeeswaxException:
> Query aborted:Kudu error(s) reported, first error: Timed out: Failed to
> write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce after
> 4
> attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem',
> partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client
> connection negotiation failed: client connection to 127.0.0.1:7051
>
> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to
> write batch of 39157 ops to tablet d27af3fe9d6b4ff3b502ddc15c5ff4ce after
> 4
> attempt(s): GetTableLocations { table: 'impala::tpch_kudu.lineitem',
> partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: Client
> connection negotiation failed: client connection to 127.0.0.1:7051 (1 of
> 78325 similar)
>
>
> If you'd like I can copy and paste the entire output. Thanks for the help.
>
>
> On Sat, Apr 7, 2018 at 9:44 PM, Jim Apple <jbap...@cloudera.com> wrote:
>
> > The attachment didn't attach. This is a common issue on our mailing
> lists.
> > I've started another thread about it.
> >
> > I'll try to reproduce this. My first suspicions are disk space,
> networking
> > restrictions, and https://issues.apache.org/jira/browse/IMPALA-6737
> >
> > Inside of the container, can you run df -h before exiting?
> >
> > On Sat, Apr 7, 2018 at 5:28 PM, Joseph Koshakow <kosh...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > Tried this again recently and still experiencing the same error. I've
> > > attached a copy of all my shell commands and output.
> > >
> > > On Sat, Feb 24, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com>
> > > wrote:
> > >
> > >> Though I just upgraded to the most recent version. It's still failing
> > but
> > >> seems to be making it one step farther. I see from the emails that
> > there's
> > >> some issues with the current build so I'll just wait until that's all
> > >> resolved. The output from my most recent attempt is below if you have
> a
> > use
> > >> for it.
> > >>
> > >> Loading custom schemas OK (Took: 0 min 40 sec)
> > >> Started Loading functional-query data in background; pid 22229.
> > >> Loading functional-query data (logging to
> /home/impdev/Impala/logs/data_
> > >> loading/load-functional-query.log)...
> > >> Started Loading TPC-H data in background; pid 22230.
> > >> Started Loading TPC-DS data in background; pid 22231.
> > >> Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_
> > >> loading/load-tpcds.log)...
> > >> Loading TPC-H data (logging to /home/impdev/Impala/logs/data_
> > >> loading/load-tpch.log)...
> > >> FAILED (Took: 25 min 51 sec)
> > >> 'load-data tpch core' failed. Tail of log:
> > >> INVALIDATE METADATA tpch_kudu.region
> > >>
> > >> (invalidate-tpch-core-impala-generated.sql):
> > >> INVALIDATE METADATA tpch_kudu.orders
> > >>
> > >> (invalidate-tpch-core-impala-generated.sql):
> > >> INVALIDATE METADATA tpch_kudu.customer
> > >>
> > >> (invalidate-tpch-core-impala-generated.sql):
> > >>
> > >>
> > >> (load-tpch-core-impala-generated-kudu-none-none.sql):
> > >> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem
> > >>
> > >> (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >> INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM
> tpch.lineitem
> > >>
> > >> (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >> INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part
> > >>
> > >> (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >> INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM
> tpch.partsupp
> > >>
> > >> (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >> INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM
> tpch.supplier
> > >>
> > >> Data Loading from Impala failed with error: ImpalaBeeswaxException:
> > >> Query aborted:Kudu error(s) reported, first error: Timed out: Failed
> to
> > >> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9
> > >> after 4 attempt(s): GetTableLocations { table:
> > >> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 }
> > >> failed: Client connection negotiation failed: client connection to
> > >> 127.0.0.1:7051
> > >>
> > >> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to
> > >> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34
> > >> after 4 attempt(s): GetTableLocations { table:
> > >> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2),
> > >> attempt: 1 } failed: Client connection negotiation failed: client
> > >> connection to 127.0.0.1:7051 (1 of 21494 similar)
> > >>
> > >>
> > >> Traceback (most recent call last):
> > >> File "/home/impdev/Impala/bin/load-data.py", line 179, in
> > >> exec_impala_query_from_file
> > >> (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >> INSERT OVERWRITE TABLE tpch_parquet.nation SELECT * FROM tpch.nation
> > >>
> > >> result = impala_client.execute(query)
> > >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line
> 173,
> > >> in execute
> > >> handle = self.__execute_query(query_string.strip(), user=user)
> > >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line
> 341,
> > >> in __execute_query
> > >> self.wait_for_completion(handle)
> > >> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line
> 361,
> > >> in wait_for_completion
> > >> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> > >> ImpalaBeeswaxException: ImpalaBeeswaxException:
> > >> Query aborted:Kudu error(s) reported, first error: Timed out: Failed
> to
> > >> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9
> > >> after 4 attempt(s): GetTableLocations { table:
> > >> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 }
> > >> failed: Client connection negotiation failed: client connection to
> > >> 127.0.0.1:7051
> > >>
> > >> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed to
> > >> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34
> > >> after 4 attempt(s): GetTableLocations { table:
> > >> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2),
> > >> attempt: 1 } failed: Client connection negotiation failed: client
> > >> connection to 127.0.0.1:7051 (1 of 21494 similar)
> > >>
> > >>
> > >> On Fri, Feb 23, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com>
> > >> wrote:
> > >>
> > >>> 8302608bdff15d6e8af324fc885dd0a8c2b44a5d
> > >>>
> > >>> On Fri, Feb 23, 2018 at 12:41 AM, Jim Apple <jbap...@cloudera.com>
> > >>> wrote:
> > >>>
> > >>>> What commit hash is checked out?
> > >>>>
> > >>>> On Thu, Feb 22, 2018 at 8:23 PM Joseph Koshakow <kosh...@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>> > Host OS is Ubunto 16.04.3 LTS. Docker OS is also Ubuntu 16.04 as
> per
> > >>>> > instructions. I have 202.61Gb of free space in the host
> environment
> > >>>> > (304.58Gb size total). I'm not sure how to check how much space is
> > >>>> left in
> > >>>> > docker containers but the size of the one created for this is
> 97.81
> > >>>> Gb.
> > >>>> >
> > >>>> > -Joe
> > >>>> >
> > >>>> > On Thu, Feb 22, 2018 at 11:15 PM, Jim Apple <jbap...@cloudera.com
> >
> > >>>> wrote:
> > >>>> >
> > >>>> > > What is the host OS? What is the docker OS? How much space is
> left
> > >>>> on
> > >>>> > disk
> > >>>> > > in each environment?
> > >>>> > >
> > >>>> > > On Thu, Feb 22, 2018 at 8:05 PM, Joseph Koshakow <
> > kosh...@gmail.com
> > >>>> >
> > >>>> > > wrote:
> > >>>> > >
> > >>>> > > > Hi,
> > >>>> > > >
> > >>>> > > > I've been trying to set up my development environment using
> > >>>> docker by
> > >>>> > > > following the following instructions: https://cwiki.ap
> > >>>> > > > ache.org/confluence/display/IMPALA/Docker+for+Impala+
> Developers
> > >>>> > > > I'm able to get to the step where you run `source
> > >>>> > > > ~/Impala/bin/bootsrap_development.sh`. Everything runs fine
> > >>>> until I get
> > >>>> > > > to this part:
> > >>>> > > > Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_
> > >>>> > > > loading/load-tpcds.log)...
> > >>>> > > >
> > >>>> > > > That hangs for about 30 minutes and then prints the following
> > >>>> (Also
> > >>>> > > > attached) before hanging again indefinitely until I quit out.
> > I've
> > >>>> > tried
> > >>>> > > > multiple times and even deleted the docker image and started
> > >>>> over. Any
> > >>>> > > > ideas?
> > >>>> > > >
> > >>>> > > > FAILED (Took: 26 min 21 sec)
> > >>>> > > > 'load-data tpch core' failed. Tail of log:
> > >>>> > > > INVALIDATE METADATA tpch_kudu.nation
> > >>>> > > >
> > >>>> > > > (invalidate-tpch-core-impala-generated.sql):
> > >>>> > > > INVALIDATE METADATA tpch_kudu.region
> > >>>> > > >
> > >>>> > > > (invalidate-tpch-core-impala-generated.sql):
> > >>>> > > > INVALIDATE METADATA tpch_kudu.orders
> > >>>> > > >
> > >>>> > > > (invalidate-tpch-core-impala-generated.sql):
> > >>>> > > > INVALIDATE METADATA tpch_kudu.customer
> > >>>> > > >
> > >>>> > > > (invalidate-tpch-core-impala-generated.sql):
> > >>>> > > >
> > >>>> > > >
> > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM
> > >>>> > tpch.lineitem
> > >>>> > > >
> > >>>> > > > (load-tpch-core-impala-generated-kudu-none-none.sql):
> > >>>> > > > INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM
> tpch.lineitem
> > >>>> > > >
> > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM
> tpch.part
> > >>>> > > >
> > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM
> > >>>> > tpch.partsupp
> > >>>> > > >
> > >>>> > > > Data Loading from Impala failed with error:
> > >>>> ImpalaBeeswaxException:
> > >>>> > > > Query aborted:Kudu error(s) reported, first error: Timed out:
> > >>>> Failed
> > >>>> > to
> > >>>> > > > write batch of 23532 ops to tablet
> > 3b2c6a27489d43028912a69ca7be8b
> > >>>> 71
> > >>>> > > after
> > >>>> > > > 4 attempt(s): GetTableLocations { table:
> > >>>> 'impala::tpch_kudu.lineitem',
> > >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> > >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
> 10.000s
> > >>>> (SENT)
> > >>>> > > >
> > >>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out:
> > >>>> Failed to
> > >>>> > > > write batch of 23532 ops to tablet
> > 3b2c6a27489d43028912a69ca7be8b
> > >>>> 71
> > >>>> > > after
> > >>>> > > > 4 attempt(s): GetTableLocations { table:
> > >>>> 'impala::tpch_kudu.lineitem',
> > >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> > >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
> 10.000s
> > >>>> (SENT)
> > >>>> > > (1
> > >>>> > > > of 23532 similar)
> > >>>> > > >
> > >>>> > > >
> > >>>> > > > Traceback (most recent call last):
> > >>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>>> > > > INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM
> > >>>> > tpch.supplier
> > >>>> > > > File "/home/impdev/Impala/bin/load-data.py", line 179, in
> > >>>> > > > exec_impala_query_from_file
> > >>>> > > >
> > >>>> > > > result = impala_client.execute(query)
> > >>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py",
> > >>>> line 173,
> > >>>> > > > in execute
> > >>>> > > > handle = self.__execute_query(query_string.strip(),
> > >>>> user=user)
> > >>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py",
> > >>>> line 341,
> > >>>> > > > in __execute_query
> > >>>> > > > self.wait_for_completion(handle)
> > >>>> > > > File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py",
> > >>>> line 361,
> > >>>> > > > in wait_for_completion
> > >>>> > > > raise ImpalaBeeswaxException("Query aborted:" + error_log,
> > >>>> None)
> > >>>> > > > ImpalaBeeswaxException: ImpalaBeeswaxException:
> > >>>> > > > Query aborted:Kudu error(s) reported, first error: Timed out:
> > >>>> Failed
> > >>>> > to
> > >>>> > > > write batch of 23532 ops to tablet
> > 3b2c6a27489d43028912a69ca7be8b
> > >>>> 71
> > >>>> > > after
> > >>>> > > > 4 attempt(s): GetTableLocations { table:
> > >>>> 'impala::tpch_kudu.lineitem',
> > >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> > >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
> 10.000s
> > >>>> (SENT)
> > >>>> > > >
> > >>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out:
> > >>>> Failed to
> > >>>> > > > write batch of 23532 ops to tablet
> > 3b2c6a27489d43028912a69ca7be8b
> > >>>> 71
> > >>>> > > after
> > >>>> > > > 4 attempt(s): GetTableLocations { table:
> > >>>> 'impala::tpch_kudu.lineitem',
> > >>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> > >>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
> 10.000s
> > >>>> (SENT)
> > >>>> > > (1
> > >>>> > > > of 23532 similar)
> > >>>> > > >
> > >>>> > > > Thanks,
> > >>>> > > > Joe Koshakow
> > >>>> > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> > >>>
> > >>>
> > >>
> > >
> >
>
