It's certainly possible that this is an OOM issue. I'll try to repro that. On Sun, Apr 8, 2018 at 12:54 PM, Joseph Koshakow <kosh...@gmail.com> wrote:
> Jim, > I followed the commands from this page: > https://cwiki.apache.org/confluence/display/IMPALA/ > Docker+for+Impala+Developers > which > is almost identical to the commands you listed. > > Philip, > /etc/hosts contents: > 127.0.0.1 localhost > ::1 localhost ip6-localhost ip6-loopback > fe00::0 ip6-localnet > ff00::0 ip6-mcastprefix > ff02::1 ip6-allnodes > ff02::2 ip6-allrouters > 172.17.0.2 c4fdb0931b2d > > > tests-with-docker: > impdev@c4fdb0931b2d:~/Impala$ ./docker/test-with-docker.py --suite NOOP > --no-cleanup-image > 2018-04-08 19:46:22,638 MainThread: CPUs: 4 Memory (GB): 11.6096229553 > 2018-04-08 19:46:22,638 MainThread: This tool should be run on a machine > with more memory. > 2018-04-08 19:46:22,640 MainThread: Running: (['git', 'rev-parse', > '--show-toplevel'],), {}; cmdline: git rev-parse --show-toplevel. > 2018-04-08 19:46:22,642 MainThread: Arguments: Namespace(build_image=None, > ccache_dir='/home/impdev/.ccache', cleanup_containers=True, > cleanup_image=False, impalad_mem_limit_bytes=7516192768, > name='i-20180408-194622', parallel_test_concurrency=4, suite=['NOOP'], > suite_concurrency=1, test=False, timeout=120) > 2018-04-08 19:46:22,643 MainThread: Running: (['docker', 'create', > '--privileged', '--name', 'i-20180408-194622', '--hostname', > 'i-20180408-194622', '--label=pwd=/home/impdev/Impala', '-e', 'LC_ALL=C', > '-e', 'IMPALAD_MEM_LIMIT_BYTES=7516192768', '-v', > '/home/impdev/Impala:/repo:ro', '-v', '/home/impdev/.ccache:/ccache', > '-v', > '/etc/localtime:/mnt/localtime', '-v', > '/home/impdev/Impala/logs/docker/i-20180408-194622/build:/logs', '-v', > '/home/impdev/Impala/docker:/mnt/base:ro', 'ubuntu:16.04', > '/mnt/base/entrypoint.sh', 'build', '1000'],), {}; cmdline: docker create > --privileged --name i-20180408-194622 --hostname i-20180408-194622 > --label=pwd=/home/impdev/Impala -e LC_ALL=C -e > IMPALAD_MEM_LIMIT_BYTES=7516192768 -v /home/impdev/Impala:/repo:ro -v > /home/impdev/.ccache:/ccache -v /etc/localtime:/mnt/localtime -v > /home/impdev/Impala/logs/docker/i-20180408-194622/build:/logs -v > /home/impdev/Impala/docker:/mnt/base:ro ubuntu:16.04 > /mnt/base/entrypoint.sh build 1000. > 2018-04-08 19:46:22,649 Thread-1: Using cgroups: cpuacct > /sys/fs/cgroup/cpu,cpuacct, memory /sys/fs/cgroup/memory > 2018-04-08 19:46:22,649 MainThread: Memory usage: None GB min, None GB max > Traceback (most recent call last): > File "./docker/test-with-docker.py", line 579, in <module> > main() > File "./docker/test-with-docker.py", line 198, in main > ret = t.run() > File "./docker/test-with-docker.py", line 494, in run > self._create_build_image() > File "./docker/test-with-docker.py", line 434, in _create_build_image > entrypoint=["/mnt/base/entrypoint.sh", "build", str(os.getuid())]) > File "./docker/test-with-docker.py", line 376, in _create_container > + entrypoint).strip() > File "./docker/test-with-docker.py", line 279, in _check_output > return subprocess.check_output(*args, **kwargs) > File "/usr/lib/python2.7/subprocess.py", line 567, in check_output > process = Popen(stdout=PIPE, *popenargs, **kwargs) > File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ > errread, errwrite) > File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child > raise child_exception > OSError: [Errno 2] No such file or directory > > Contents of /mnt: > impdev@c4fdb0931b2d:~/Impala$ ls -lart /mnt/ > total 8 > drwxr-xr-x 2 root root 4096 Feb 28 19:13 . > drwxr-xr-x 44 root root 4096 Apr 7 07:07 .. > > > On Sun, Apr 8, 2018 at 12:13 AM, Jim Apple <jbap...@cloudera.com> wrote: > > > I was unable to reproduce this in a fresh ec2 instance. I ran: > > > > sudo docker run --interactive --privileged --tty ubuntu:16.04 > > # inside the container > > apt-get update > > apt-get install -y sudo > > adduser --disabled-password --gecos '' impdev > > echo 'impdev ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers > > sudo su - impdev > > # as impdev > > sudo apt install -y git > > git clone https://git-wip-us.apache.org/repos/asf/impala.git ~/Impala > > > > > > time ~/Impala/bin/bootstrap_development.sh > > > > On Sat, Apr 7, 2018 at 6:44 PM, Jim Apple <jbap...@cloudera.com> wrote: > > > > > The attachment didn't attach. This is a common issue on our mailing > > lists. > > > I've started another thread about it. > > > > > > I'll try to reproduce this. My first suspicions are disk space, > > networking > > > restrictions, and https://issues.apache.org/jira/browse/IMPALA-6737 > > > > > > Inside of the container, can you run df -h before exiting? > > > > > > On Sat, Apr 7, 2018 at 5:28 PM, Joseph Koshakow <kosh...@gmail.com> > > wrote: > > > > > >> Hi, > > >> > > >> Tried this again recently and still experiencing the same error. I've > > >> attached a copy of all my shell commands and output. > > >> > > >> On Sat, Feb 24, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com> > > >> wrote: > > >> > > >>> Though I just upgraded to the most recent version. It's still failing > > >>> but seems to be making it one step farther. I see from the emails > that > > >>> there's some issues with the current build so I'll just wait until > > that's > > >>> all resolved. The output from my most recent attempt is below if you > > have a > > >>> use for it. > > >>> > > >>> Loading custom schemas OK (Took: 0 min 40 sec) > > >>> Started Loading functional-query data in background; pid 22229. > > >>> Loading functional-query data (logging to > > /home/impdev/Impala/logs/data_ > > >>> loading/load-functional-query.log)... > > >>> Started Loading TPC-H data in background; pid 22230. > > >>> Started Loading TPC-DS data in background; pid 22231. > > >>> Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_ > > >>> loading/load-tpcds.log)... > > >>> Loading TPC-H data (logging to /home/impdev/Impala/logs/data_ > > >>> loading/load-tpch.log)... > > >>> FAILED (Took: 25 min 51 sec) > > >>> 'load-data tpch core' failed. Tail of log: > > >>> INVALIDATE METADATA tpch_kudu.region > > >>> > > >>> (invalidate-tpch-core-impala-generated.sql): > > >>> INVALIDATE METADATA tpch_kudu.orders > > >>> > > >>> (invalidate-tpch-core-impala-generated.sql): > > >>> INVALIDATE METADATA tpch_kudu.customer > > >>> > > >>> (invalidate-tpch-core-impala-generated.sql): > > >>> > > >>> > > >>> (load-tpch-core-impala-generated-kudu-none-none.sql): > > >>> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem > > >>> > > >>> (load-tpch-core-impala-generated-parquet-none-none.sql): > > >>> INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM > > tpch.lineitem > > >>> > > >>> (load-tpch-core-impala-generated-parquet-none-none.sql): > > >>> INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part > > >>> > > >>> (load-tpch-core-impala-generated-parquet-none-none.sql): > > >>> INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM > > tpch.partsupp > > >>> > > >>> (load-tpch-core-impala-generated-parquet-none-none.sql): > > >>> INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM > > tpch.supplier > > >>> > > >>> Data Loading from Impala failed with error: ImpalaBeeswaxException: > > >>> Query aborted:Kudu error(s) reported, first error: Timed out: Failed > > to > > >>> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9 > > >>> after 4 attempt(s): GetTableLocations { table: > > >>> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 } > > >>> failed: Client connection negotiation failed: client connection to > > >>> 127.0.0.1:7051 > > >>> > > >>> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed > to > > >>> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34 > > >>> after 4 attempt(s): GetTableLocations { table: > > >>> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2), > > >>> attempt: 1 } failed: Client connection negotiation failed: client > > >>> connection to 127.0.0.1:7051 (1 of 21494 similar) > > >>> > > >>> > > >>> Traceback (most recent call last): > > >>> File "/home/impdev/Impala/bin/load-data.py", line 179, in > > >>> exec_impala_query_from_file > > >>> (load-tpch-core-impala-generated-parquet-none-none.sql): > > >>> INSERT OVERWRITE TABLE tpch_parquet.nation SELECT * FROM tpch.nation > > >>> > > >>> result = impala_client.execute(query) > > >>> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line > > 173, > > >>> in execute > > >>> handle = self.__execute_query(query_string.strip(), user=user) > > >>> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line > > 341, > > >>> in __execute_query > > >>> self.wait_for_completion(handle) > > >>> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line > > 361, > > >>> in wait_for_completion > > >>> raise ImpalaBeeswaxException("Query aborted:" + error_log, None) > > >>> ImpalaBeeswaxException: ImpalaBeeswaxException: > > >>> Query aborted:Kudu error(s) reported, first error: Timed out: Failed > > to > > >>> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9 > > >>> after 4 attempt(s): GetTableLocations { table: > > >>> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 } > > >>> failed: Client connection negotiation failed: client connection to > > >>> 127.0.0.1:7051 > > >>> > > >>> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed > to > > >>> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34 > > >>> after 4 attempt(s): GetTableLocations { table: > > >>> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2), > > >>> attempt: 1 } failed: Client connection negotiation failed: client > > >>> connection to 127.0.0.1:7051 (1 of 21494 similar) > > >>> > > >>> > > >>> On Fri, Feb 23, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com> > > >>> wrote: > > >>> > > >>>> 8302608bdff15d6e8af324fc885dd0a8c2b44a5d > > >>>> > > >>>> On Fri, Feb 23, 2018 at 12:41 AM, Jim Apple <jbap...@cloudera.com> > > >>>> wrote: > > >>>> > > >>>>> What commit hash is checked out? > > >>>>> > > >>>>> On Thu, Feb 22, 2018 at 8:23 PM Joseph Koshakow <kosh...@gmail.com > > > > >>>>> wrote: > > >>>>> > > >>>>> > Host OS is Ubunto 16.04.3 LTS. Docker OS is also Ubuntu 16.04 as > > per > > >>>>> > instructions. I have 202.61Gb of free space in the host > environment > > >>>>> > (304.58Gb size total). I'm not sure how to check how much space > is > > >>>>> left in > > >>>>> > docker containers but the size of the one created for this is > 97.81 > > >>>>> Gb. > > >>>>> > > > >>>>> > -Joe > > >>>>> > > > >>>>> > On Thu, Feb 22, 2018 at 11:15 PM, Jim Apple < > jbap...@cloudera.com> > > >>>>> wrote: > > >>>>> > > > >>>>> > > What is the host OS? What is the docker OS? How much space is > > left > > >>>>> on > > >>>>> > disk > > >>>>> > > in each environment? > > >>>>> > > > > >>>>> > > On Thu, Feb 22, 2018 at 8:05 PM, Joseph Koshakow < > > >>>>> kosh...@gmail.com> > > >>>>> > > wrote: > > >>>>> > > > > >>>>> > > > Hi, > > >>>>> > > > > > >>>>> > > > I've been trying to set up my development environment using > > >>>>> docker by > > >>>>> > > > following the following instructions: https://cwiki.ap > > >>>>> > > > ache.org/confluence/display/IMPALA/Docker+for+Impala+ > > Developers > > >>>>> > > > I'm able to get to the step where you run `source > > >>>>> > > > ~/Impala/bin/bootsrap_development.sh`. Everything runs fine > > >>>>> until I get > > >>>>> > > > to this part: > > >>>>> > > > Loading TPC-DS data (logging to > /home/impdev/Impala/logs/data_ > > >>>>> > > > loading/load-tpcds.log)... > > >>>>> > > > > > >>>>> > > > That hangs for about 30 minutes and then prints the following > > >>>>> (Also > > >>>>> > > > attached) before hanging again indefinitely until I quit out. > > >>>>> I've > > >>>>> > tried > > >>>>> > > > multiple times and even deleted the docker image and started > > >>>>> over. Any > > >>>>> > > > ideas? > > >>>>> > > > > > >>>>> > > > FAILED (Took: 26 min 21 sec) > > >>>>> > > > 'load-data tpch core' failed. Tail of log: > > >>>>> > > > INVALIDATE METADATA tpch_kudu.nation > > >>>>> > > > > > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): > > >>>>> > > > INVALIDATE METADATA tpch_kudu.region > > >>>>> > > > > > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): > > >>>>> > > > INVALIDATE METADATA tpch_kudu.orders > > >>>>> > > > > > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): > > >>>>> > > > INVALIDATE METADATA tpch_kudu.customer > > >>>>> > > > > > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): > > >>>>> > > > > > >>>>> > > > > > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): > > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM > > >>>>> > tpch.lineitem > > >>>>> > > > > > >>>>> > > > (load-tpch-core-impala-generated-kudu-none-none.sql): > > >>>>> > > > INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM > > tpch.lineitem > > >>>>> > > > > > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): > > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM > > tpch.part > > >>>>> > > > > > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): > > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM > > >>>>> > tpch.partsupp > > >>>>> > > > > > >>>>> > > > Data Loading from Impala failed with error: > > >>>>> ImpalaBeeswaxException: > > >>>>> > > > Query aborted:Kudu error(s) reported, first error: Timed > out: > > >>>>> Failed > > >>>>> > to > > >>>>> > > > write batch of 23532 ops to tablet > > 3b2c6a27489d43028912a69ca7be8b > > >>>>> 71 > > >>>>> > > after > > >>>>> > > > 4 attempt(s): GetTableLocations { table: > > >>>>> 'impala::tpch_kudu.lineitem', > > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: > > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after > > 10.000s > > >>>>> (SENT) > > >>>>> > > > > > >>>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: > > >>>>> Failed to > > >>>>> > > > write batch of 23532 ops to tablet > > 3b2c6a27489d43028912a69ca7be8b > > >>>>> 71 > > >>>>> > > after > > >>>>> > > > 4 attempt(s): GetTableLocations { table: > > >>>>> 'impala::tpch_kudu.lineitem', > > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: > > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after > > 10.000s > > >>>>> (SENT) > > >>>>> > > (1 > > >>>>> > > > of 23532 similar) > > >>>>> > > > > > >>>>> > > > > > >>>>> > > > Traceback (most recent call last): > > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): > > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM > > >>>>> > tpch.supplier > > >>>>> > > > File "/home/impdev/Impala/bin/load-data.py", line 179, in > > >>>>> > > > exec_impala_query_from_file > > >>>>> > > > > > >>>>> > > > result = impala_client.execute(query) > > >>>>> > > > File "/home/impdev/Impala/tests/ > beeswax/impala_beeswax.py", > > >>>>> line 173, > > >>>>> > > > in execute > > >>>>> > > > handle = self.__execute_query(query_string.strip(), > > >>>>> user=user) > > >>>>> > > > File "/home/impdev/Impala/tests/ > beeswax/impala_beeswax.py", > > >>>>> line 341, > > >>>>> > > > in __execute_query > > >>>>> > > > self.wait_for_completion(handle) > > >>>>> > > > File "/home/impdev/Impala/tests/ > beeswax/impala_beeswax.py", > > >>>>> line 361, > > >>>>> > > > in wait_for_completion > > >>>>> > > > raise ImpalaBeeswaxException("Query aborted:" + > error_log, > > >>>>> None) > > >>>>> > > > ImpalaBeeswaxException: ImpalaBeeswaxException: > > >>>>> > > > Query aborted:Kudu error(s) reported, first error: Timed > out: > > >>>>> Failed > > >>>>> > to > > >>>>> > > > write batch of 23532 ops to tablet > > 3b2c6a27489d43028912a69ca7be8b > > >>>>> 71 > > >>>>> > > after > > >>>>> > > > 4 attempt(s): GetTableLocations { table: > > >>>>> 'impala::tpch_kudu.lineitem', > > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: > > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after > > 10.000s > > >>>>> (SENT) > > >>>>> > > > > > >>>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: > > >>>>> Failed to > > >>>>> > > > write batch of 23532 ops to tablet > > 3b2c6a27489d43028912a69ca7be8b > > >>>>> 71 > > >>>>> > > after > > >>>>> > > > 4 attempt(s): GetTableLocations { table: > > >>>>> 'impala::tpch_kudu.lineitem', > > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: > > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after > > 10.000s > > >>>>> (SENT) > > >>>>> > > (1 > > >>>>> > > > of 23532 similar) > > >>>>> > > > > > >>>>> > > > Thanks, > > >>>>> > > > Joe Koshakow > > >>>>> > > > > > >>>>> > > > > > >>>>> > > > > >>>>> > > > >>>>> > > >>>> > > >>>> > > >>> > > >> > > > > > >