It's certainly possible that this is an OOM issue. I'll try to repro that.

On Sun, Apr 8, 2018 at 12:54 PM, Joseph Koshakow <kosh...@gmail.com> wrote:

> Jim,
> I followed the commands from this page:
> https://cwiki.apache.org/confluence/display/IMPALA/
> Docker+for+Impala+Developers
> which
> is almost identical to the commands you listed.
>
> Philip,
> /etc/hosts contents:
> 127.0.0.1       localhost
> ::1     localhost ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
> 172.17.0.2      c4fdb0931b2d
>
>
> tests-with-docker:
> impdev@c4fdb0931b2d:~/Impala$ ./docker/test-with-docker.py --suite NOOP
> --no-cleanup-image
> 2018-04-08 19:46:22,638 MainThread: CPUs: 4 Memory (GB): 11.6096229553
> 2018-04-08 19:46:22,638 MainThread: This tool should be run on a machine
> with more memory.
> 2018-04-08 19:46:22,640 MainThread: Running: (['git', 'rev-parse',
> '--show-toplevel'],), {}; cmdline: git rev-parse --show-toplevel.
> 2018-04-08 19:46:22,642 MainThread: Arguments: Namespace(build_image=None,
> ccache_dir='/home/impdev/.ccache', cleanup_containers=True,
> cleanup_image=False, impalad_mem_limit_bytes=7516192768,
> name='i-20180408-194622', parallel_test_concurrency=4, suite=['NOOP'],
> suite_concurrency=1, test=False, timeout=120)
> 2018-04-08 19:46:22,643 MainThread: Running: (['docker', 'create',
> '--privileged', '--name', 'i-20180408-194622', '--hostname',
> 'i-20180408-194622', '--label=pwd=/home/impdev/Impala', '-e', 'LC_ALL=C',
> '-e', 'IMPALAD_MEM_LIMIT_BYTES=7516192768', '-v',
> '/home/impdev/Impala:/repo:ro', '-v', '/home/impdev/.ccache:/ccache',
> '-v',
> '/etc/localtime:/mnt/localtime', '-v',
> '/home/impdev/Impala/logs/docker/i-20180408-194622/build:/logs', '-v',
> '/home/impdev/Impala/docker:/mnt/base:ro', 'ubuntu:16.04',
> '/mnt/base/entrypoint.sh', 'build', '1000'],), {}; cmdline: docker create
> --privileged --name i-20180408-194622 --hostname i-20180408-194622
> --label=pwd=/home/impdev/Impala -e LC_ALL=C -e
> IMPALAD_MEM_LIMIT_BYTES=7516192768 -v /home/impdev/Impala:/repo:ro -v
> /home/impdev/.ccache:/ccache -v /etc/localtime:/mnt/localtime -v
> /home/impdev/Impala/logs/docker/i-20180408-194622/build:/logs -v
> /home/impdev/Impala/docker:/mnt/base:ro ubuntu:16.04
> /mnt/base/entrypoint.sh build 1000.
> 2018-04-08 19:46:22,649 Thread-1: Using cgroups: cpuacct
> /sys/fs/cgroup/cpu,cpuacct, memory /sys/fs/cgroup/memory
> 2018-04-08 19:46:22,649 MainThread: Memory usage: None GB min, None GB max
> Traceback (most recent call last):
>   File "./docker/test-with-docker.py", line 579, in <module>
>     main()
>   File "./docker/test-with-docker.py", line 198, in main
>     ret = t.run()
>   File "./docker/test-with-docker.py", line 494, in run
>     self._create_build_image()
>   File "./docker/test-with-docker.py", line 434, in _create_build_image
>     entrypoint=["/mnt/base/entrypoint.sh", "build", str(os.getuid())])
>   File "./docker/test-with-docker.py", line 376, in _create_container
>     + entrypoint).strip()
>   File "./docker/test-with-docker.py", line 279, in _check_output
>     return subprocess.check_output(*args, **kwargs)
>   File "/usr/lib/python2.7/subprocess.py", line 567, in check_output
>     process = Popen(stdout=PIPE, *popenargs, **kwargs)
>   File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
>     errread, errwrite)
>   File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
>     raise child_exception
> OSError: [Errno 2] No such file or directory
>
> Contents of /mnt:
> impdev@c4fdb0931b2d:~/Impala$ ls -lart /mnt/
> total 8
> drwxr-xr-x  2 root root 4096 Feb 28 19:13 .
> drwxr-xr-x 44 root root 4096 Apr  7 07:07 ..
>
>
> On Sun, Apr 8, 2018 at 12:13 AM, Jim Apple <jbap...@cloudera.com> wrote:
>
> > I was unable to reproduce this in a fresh ec2 instance. I ran:
> >
> > sudo docker run --interactive --privileged --tty ubuntu:16.04
> > # inside the container
> > apt-get update
> > apt-get install -y sudo
> > adduser --disabled-password --gecos '' impdev
> > echo 'impdev ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
> > sudo su - impdev
> > # as impdev
> > sudo apt install -y git
> > git clone https://git-wip-us.apache.org/repos/asf/impala.git ~/Impala
> >
> >
> > time ~/Impala/bin/bootstrap_development.sh
> >
> > On Sat, Apr 7, 2018 at 6:44 PM, Jim Apple <jbap...@cloudera.com> wrote:
> >
> > > The attachment didn't attach. This is a common issue on our mailing
> > lists.
> > > I've started another thread about it.
> > >
> > > I'll try to reproduce this. My first suspicions are disk space,
> > networking
> > > restrictions, and https://issues.apache.org/jira/browse/IMPALA-6737
> > >
> > > Inside of the container, can you run df -h before exiting?
> > >
> > > On Sat, Apr 7, 2018 at 5:28 PM, Joseph Koshakow <kosh...@gmail.com>
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> Tried this again recently and still experiencing the same error. I've
> > >> attached a copy of all my shell commands and output.
> > >>
> > >> On Sat, Feb 24, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com>
> > >> wrote:
> > >>
> > >>> Though I just upgraded to the most recent version. It's still failing
> > >>> but seems to be making it one step farther. I see from the emails
> that
> > >>> there's some issues with the current build so I'll just wait until
> > that's
> > >>> all resolved. The output from my most recent attempt is below if you
> > have a
> > >>> use for it.
> > >>>
> > >>> Loading custom schemas OK (Took: 0 min 40 sec)
> > >>> Started Loading functional-query data in background; pid 22229.
> > >>> Loading functional-query data (logging to
> > /home/impdev/Impala/logs/data_
> > >>> loading/load-functional-query.log)...
> > >>> Started Loading TPC-H data in background; pid 22230.
> > >>> Started Loading TPC-DS data in background; pid 22231.
> > >>> Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_
> > >>> loading/load-tpcds.log)...
> > >>> Loading TPC-H data (logging to /home/impdev/Impala/logs/data_
> > >>> loading/load-tpch.log)...
> > >>>     FAILED (Took: 25 min 51 sec)
> > >>>     'load-data tpch core' failed. Tail of log:
> > >>> INVALIDATE METADATA tpch_kudu.region
> > >>>
> > >>> (invalidate-tpch-core-impala-generated.sql):
> > >>> INVALIDATE METADATA tpch_kudu.orders
> > >>>
> > >>> (invalidate-tpch-core-impala-generated.sql):
> > >>> INVALIDATE METADATA tpch_kudu.customer
> > >>>
> > >>> (invalidate-tpch-core-impala-generated.sql):
> > >>>
> > >>>
> > >>> (load-tpch-core-impala-generated-kudu-none-none.sql):
> > >>> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem
> > >>>
> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>> INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM
> > tpch.lineitem
> > >>>
> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>> INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part
> > >>>
> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>> INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM
> > tpch.partsupp
> > >>>
> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>> INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM
> > tpch.supplier
> > >>>
> > >>> Data Loading from Impala failed with error: ImpalaBeeswaxException:
> > >>>  Query aborted:Kudu error(s) reported, first error: Timed out: Failed
> > to
> > >>> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9
> > >>> after 4 attempt(s): GetTableLocations { table:
> > >>> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 }
> > >>> failed: Client connection negotiation failed: client connection to
> > >>> 127.0.0.1:7051
> > >>>
> > >>> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed
> to
> > >>> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34
> > >>> after 4 attempt(s): GetTableLocations { table:
> > >>> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2),
> > >>> attempt: 1 } failed: Client connection negotiation failed: client
> > >>> connection to 127.0.0.1:7051 (1 of 21494 similar)
> > >>>
> > >>>
> > >>> Traceback (most recent call last):
> > >>>   File "/home/impdev/Impala/bin/load-data.py", line 179, in
> > >>> exec_impala_query_from_file
> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>> INSERT OVERWRITE TABLE tpch_parquet.nation SELECT * FROM tpch.nation
> > >>>
> > >>>     result = impala_client.execute(query)
> > >>>   File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line
> > 173,
> > >>> in execute
> > >>>     handle = self.__execute_query(query_string.strip(), user=user)
> > >>>   File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line
> > 341,
> > >>> in __execute_query
> > >>>     self.wait_for_completion(handle)
> > >>>   File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line
> > 361,
> > >>> in wait_for_completion
> > >>>     raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> > >>> ImpalaBeeswaxException: ImpalaBeeswaxException:
> > >>>  Query aborted:Kudu error(s) reported, first error: Timed out: Failed
> > to
> > >>> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9
> > >>> after 4 attempt(s): GetTableLocations { table:
> > >>> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 }
> > >>> failed: Client connection negotiation failed: client connection to
> > >>> 127.0.0.1:7051
> > >>>
> > >>> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed
> to
> > >>> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34
> > >>> after 4 attempt(s): GetTableLocations { table:
> > >>> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2),
> > >>> attempt: 1 } failed: Client connection negotiation failed: client
> > >>> connection to 127.0.0.1:7051 (1 of 21494 similar)
> > >>>
> > >>>
> > >>> On Fri, Feb 23, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> 8302608bdff15d6e8af324fc885dd0a8c2b44a5d
> > >>>>
> > >>>> On Fri, Feb 23, 2018 at 12:41 AM, Jim Apple <jbap...@cloudera.com>
> > >>>> wrote:
> > >>>>
> > >>>>> What commit hash is checked out?
> > >>>>>
> > >>>>> On Thu, Feb 22, 2018 at 8:23 PM Joseph Koshakow <kosh...@gmail.com
> >
> > >>>>> wrote:
> > >>>>>
> > >>>>> > Host OS is Ubunto 16.04.3 LTS. Docker OS is also Ubuntu 16.04 as
> > per
> > >>>>> > instructions. I have 202.61Gb of free space in the host
> environment
> > >>>>> > (304.58Gb size total). I'm not sure how to check how much space
> is
> > >>>>> left in
> > >>>>> > docker containers but the size of the one created for this is
> 97.81
> > >>>>> Gb.
> > >>>>> >
> > >>>>> > -Joe
> > >>>>> >
> > >>>>> > On Thu, Feb 22, 2018 at 11:15 PM, Jim Apple <
> jbap...@cloudera.com>
> > >>>>> wrote:
> > >>>>> >
> > >>>>> > > What is the host OS? What is the docker OS? How much space is
> > left
> > >>>>> on
> > >>>>> > disk
> > >>>>> > > in each environment?
> > >>>>> > >
> > >>>>> > > On Thu, Feb 22, 2018 at 8:05 PM, Joseph Koshakow <
> > >>>>> kosh...@gmail.com>
> > >>>>> > > wrote:
> > >>>>> > >
> > >>>>> > > > Hi,
> > >>>>> > > >
> > >>>>> > > > I've been trying to set up my development environment using
> > >>>>> docker by
> > >>>>> > > > following the following instructions: https://cwiki.ap
> > >>>>> > > > ache.org/confluence/display/IMPALA/Docker+for+Impala+
> > Developers
> > >>>>> > > > I'm able to get to the step where you run `source
> > >>>>> > > > ~/Impala/bin/bootsrap_development.sh`. Everything runs fine
> > >>>>> until I get
> > >>>>> > > > to this part:
> > >>>>> > > > Loading TPC-DS data (logging to
> /home/impdev/Impala/logs/data_
> > >>>>> > > > loading/load-tpcds.log)...
> > >>>>> > > >
> > >>>>> > > > That hangs for about 30 minutes and then prints the following
> > >>>>> (Also
> > >>>>> > > > attached) before hanging again indefinitely until I quit out.
> > >>>>> I've
> > >>>>> > tried
> > >>>>> > > > multiple times and even deleted the docker image and started
> > >>>>> over. Any
> > >>>>> > > > ideas?
> > >>>>> > > >
> > >>>>> > > > FAILED (Took: 26 min 21 sec)
> > >>>>> > > >     'load-data tpch core' failed. Tail of log:
> > >>>>> > > > INVALIDATE METADATA tpch_kudu.nation
> > >>>>> > > >
> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql):
> > >>>>> > > > INVALIDATE METADATA tpch_kudu.region
> > >>>>> > > >
> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql):
> > >>>>> > > > INVALIDATE METADATA tpch_kudu.orders
> > >>>>> > > >
> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql):
> > >>>>> > > > INVALIDATE METADATA tpch_kudu.customer
> > >>>>> > > >
> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql):
> > >>>>> > > >
> > >>>>> > > >
> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM
> > >>>>> > tpch.lineitem
> > >>>>> > > >
> > >>>>> > > > (load-tpch-core-impala-generated-kudu-none-none.sql):
> > >>>>> > > > INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM
> > tpch.lineitem
> > >>>>> > > >
> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM
> > tpch.part
> > >>>>> > > >
> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM
> > >>>>> > tpch.partsupp
> > >>>>> > > >
> > >>>>> > > > Data Loading from Impala failed with error:
> > >>>>> ImpalaBeeswaxException:
> > >>>>> > > >  Query aborted:Kudu error(s) reported, first error: Timed
> out:
> > >>>>> Failed
> > >>>>> > to
> > >>>>> > > > write batch of 23532 ops to tablet
> > 3b2c6a27489d43028912a69ca7be8b
> > >>>>> 71
> > >>>>> > > after
> > >>>>> > > > 4 attempt(s): GetTableLocations { table:
> > >>>>> 'impala::tpch_kudu.lineitem',
> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
> > 10.000s
> > >>>>> (SENT)
> > >>>>> > > >
> > >>>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out:
> > >>>>> Failed to
> > >>>>> > > > write batch of 23532 ops to tablet
> > 3b2c6a27489d43028912a69ca7be8b
> > >>>>> 71
> > >>>>> > > after
> > >>>>> > > > 4 attempt(s): GetTableLocations { table:
> > >>>>> 'impala::tpch_kudu.lineitem',
> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
> > 10.000s
> > >>>>> (SENT)
> > >>>>> > > (1
> > >>>>> > > > of 23532 similar)
> > >>>>> > > >
> > >>>>> > > >
> > >>>>> > > > Traceback (most recent call last):
> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM
> > >>>>> > tpch.supplier
> > >>>>> > > >   File "/home/impdev/Impala/bin/load-data.py", line 179, in
> > >>>>> > > > exec_impala_query_from_file
> > >>>>> > > >
> > >>>>> > > >     result = impala_client.execute(query)
> > >>>>> > > >   File "/home/impdev/Impala/tests/
> beeswax/impala_beeswax.py",
> > >>>>> line 173,
> > >>>>> > > > in execute
> > >>>>> > > >     handle = self.__execute_query(query_string.strip(),
> > >>>>> user=user)
> > >>>>> > > >   File "/home/impdev/Impala/tests/
> beeswax/impala_beeswax.py",
> > >>>>> line 341,
> > >>>>> > > > in __execute_query
> > >>>>> > > >     self.wait_for_completion(handle)
> > >>>>> > > >   File "/home/impdev/Impala/tests/
> beeswax/impala_beeswax.py",
> > >>>>> line 361,
> > >>>>> > > > in wait_for_completion
> > >>>>> > > >     raise ImpalaBeeswaxException("Query aborted:" +
> error_log,
> > >>>>> None)
> > >>>>> > > > ImpalaBeeswaxException: ImpalaBeeswaxException:
> > >>>>> > > >  Query aborted:Kudu error(s) reported, first error: Timed
> out:
> > >>>>> Failed
> > >>>>> > to
> > >>>>> > > > write batch of 23532 ops to tablet
> > 3b2c6a27489d43028912a69ca7be8b
> > >>>>> 71
> > >>>>> > > after
> > >>>>> > > > 4 attempt(s): GetTableLocations { table:
> > >>>>> 'impala::tpch_kudu.lineitem',
> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
> > 10.000s
> > >>>>> (SENT)
> > >>>>> > > >
> > >>>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out:
> > >>>>> Failed to
> > >>>>> > > > write batch of 23532 ops to tablet
> > 3b2c6a27489d43028912a69ca7be8b
> > >>>>> 71
> > >>>>> > > after
> > >>>>> > > > 4 attempt(s): GetTableLocations { table:
> > >>>>> 'impala::tpch_kudu.lineitem',
> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
> > 10.000s
> > >>>>> (SENT)
> > >>>>> > > (1
> > >>>>> > > > of 23532 similar)
> > >>>>> > > >
> > >>>>> > > > Thanks,
> > >>>>> > > > Joe Koshakow
> > >>>>> > > >
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> >
>

Reply via email to