I was able to repro a failed data load with 8GB of memory. You have more than that, but it's still possible that's the issue.
On Sun, Apr 8, 2018 at 7:58 PM, Jim Apple <jbap...@cloudera.com> wrote: > It's certainly possible that this is an OOM issue. I'll try to repro that. > > On Sun, Apr 8, 2018 at 12:54 PM, Joseph Koshakow <kosh...@gmail.com> > wrote: > >> Jim, >> I followed the commands from this page: >> https://cwiki.apache.org/confluence/display/IMPALA/Docker+ >> for+Impala+Developers >> which >> is almost identical to the commands you listed. >> >> Philip, >> /etc/hosts contents: >> 127.0.0.1 localhost >> ::1 localhost ip6-localhost ip6-loopback >> fe00::0 ip6-localnet >> ff00::0 ip6-mcastprefix >> ff02::1 ip6-allnodes >> ff02::2 ip6-allrouters >> 172.17.0.2 c4fdb0931b2d >> >> >> tests-with-docker: >> impdev@c4fdb0931b2d:~/Impala$ ./docker/test-with-docker.py --suite NOOP >> --no-cleanup-image >> 2018-04-08 19:46:22,638 MainThread: CPUs: 4 Memory (GB): 11.6096229553 >> 2018-04-08 19:46:22,638 MainThread: This tool should be run on a machine >> with more memory. >> 2018-04-08 19:46:22,640 MainThread: Running: (['git', 'rev-parse', >> '--show-toplevel'],), {}; cmdline: git rev-parse --show-toplevel. >> 2018-04-08 19:46:22,642 MainThread: Arguments: Namespace(build_image=None, >> ccache_dir='/home/impdev/.ccache', cleanup_containers=True, >> cleanup_image=False, impalad_mem_limit_bytes=7516192768, >> name='i-20180408-194622', parallel_test_concurrency=4, suite=['NOOP'], >> suite_concurrency=1, test=False, timeout=120) >> 2018-04-08 19:46:22,643 MainThread: Running: (['docker', 'create', >> '--privileged', '--name', 'i-20180408-194622', '--hostname', >> 'i-20180408-194622', '--label=pwd=/home/impdev/Impala', '-e', 'LC_ALL=C', >> '-e', 'IMPALAD_MEM_LIMIT_BYTES=7516192768', '-v', >> '/home/impdev/Impala:/repo:ro', '-v', '/home/impdev/.ccache:/ccache', >> '-v', >> '/etc/localtime:/mnt/localtime', '-v', >> '/home/impdev/Impala/logs/docker/i-20180408-194622/build:/logs', '-v', >> '/home/impdev/Impala/docker:/mnt/base:ro', 'ubuntu:16.04', >> '/mnt/base/entrypoint.sh', 'build', '1000'],), {}; cmdline: docker create >> --privileged --name i-20180408-194622 --hostname i-20180408-194622 >> --label=pwd=/home/impdev/Impala -e LC_ALL=C -e >> IMPALAD_MEM_LIMIT_BYTES=7516192768 -v /home/impdev/Impala:/repo:ro -v >> /home/impdev/.ccache:/ccache -v /etc/localtime:/mnt/localtime -v >> /home/impdev/Impala/logs/docker/i-20180408-194622/build:/logs -v >> /home/impdev/Impala/docker:/mnt/base:ro ubuntu:16.04 >> /mnt/base/entrypoint.sh build 1000. >> 2018-04-08 19:46:22,649 Thread-1: Using cgroups: cpuacct >> /sys/fs/cgroup/cpu,cpuacct, memory /sys/fs/cgroup/memory >> 2018-04-08 19:46:22,649 MainThread: Memory usage: None GB min, None GB max >> Traceback (most recent call last): >> File "./docker/test-with-docker.py", line 579, in <module> >> main() >> File "./docker/test-with-docker.py", line 198, in main >> ret = t.run() >> File "./docker/test-with-docker.py", line 494, in run >> self._create_build_image() >> File "./docker/test-with-docker.py", line 434, in _create_build_image >> entrypoint=["/mnt/base/entrypoint.sh", "build", str(os.getuid())]) >> File "./docker/test-with-docker.py", line 376, in _create_container >> + entrypoint).strip() >> File "./docker/test-with-docker.py", line 279, in _check_output >> return subprocess.check_output(*args, **kwargs) >> File "/usr/lib/python2.7/subprocess.py", line 567, in check_output >> process = Popen(stdout=PIPE, *popenargs, **kwargs) >> File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ >> errread, errwrite) >> File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child >> raise child_exception >> OSError: [Errno 2] No such file or directory >> >> Contents of /mnt: >> impdev@c4fdb0931b2d:~/Impala$ ls -lart /mnt/ >> total 8 >> drwxr-xr-x 2 root root 4096 Feb 28 19:13 . >> drwxr-xr-x 44 root root 4096 Apr 7 07:07 .. >> >> >> On Sun, Apr 8, 2018 at 12:13 AM, Jim Apple <jbap...@cloudera.com> wrote: >> >> > I was unable to reproduce this in a fresh ec2 instance. I ran: >> > >> > sudo docker run --interactive --privileged --tty ubuntu:16.04 >> > # inside the container >> > apt-get update >> > apt-get install -y sudo >> > adduser --disabled-password --gecos '' impdev >> > echo 'impdev ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers >> > sudo su - impdev >> > # as impdev >> > sudo apt install -y git >> > git clone https://git-wip-us.apache.org/repos/asf/impala.git ~/Impala >> > >> > >> > time ~/Impala/bin/bootstrap_development.sh >> > >> > On Sat, Apr 7, 2018 at 6:44 PM, Jim Apple <jbap...@cloudera.com> wrote: >> > >> > > The attachment didn't attach. This is a common issue on our mailing >> > lists. >> > > I've started another thread about it. >> > > >> > > I'll try to reproduce this. My first suspicions are disk space, >> > networking >> > > restrictions, and https://issues.apache.org/jira/browse/IMPALA-6737 >> > > >> > > Inside of the container, can you run df -h before exiting? >> > > >> > > On Sat, Apr 7, 2018 at 5:28 PM, Joseph Koshakow <kosh...@gmail.com> >> > wrote: >> > > >> > >> Hi, >> > >> >> > >> Tried this again recently and still experiencing the same error. I've >> > >> attached a copy of all my shell commands and output. >> > >> >> > >> On Sat, Feb 24, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com> >> > >> wrote: >> > >> >> > >>> Though I just upgraded to the most recent version. It's still >> failing >> > >>> but seems to be making it one step farther. I see from the emails >> that >> > >>> there's some issues with the current build so I'll just wait until >> > that's >> > >>> all resolved. The output from my most recent attempt is below if you >> > have a >> > >>> use for it. >> > >>> >> > >>> Loading custom schemas OK (Took: 0 min 40 sec) >> > >>> Started Loading functional-query data in background; pid 22229. >> > >>> Loading functional-query data (logging to >> > /home/impdev/Impala/logs/data_ >> > >>> loading/load-functional-query.log)... >> > >>> Started Loading TPC-H data in background; pid 22230. >> > >>> Started Loading TPC-DS data in background; pid 22231. >> > >>> Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_ >> > >>> loading/load-tpcds.log)... >> > >>> Loading TPC-H data (logging to /home/impdev/Impala/logs/data_ >> > >>> loading/load-tpch.log)... >> > >>> FAILED (Took: 25 min 51 sec) >> > >>> 'load-data tpch core' failed. Tail of log: >> > >>> INVALIDATE METADATA tpch_kudu.region >> > >>> >> > >>> (invalidate-tpch-core-impala-generated.sql): >> > >>> INVALIDATE METADATA tpch_kudu.orders >> > >>> >> > >>> (invalidate-tpch-core-impala-generated.sql): >> > >>> INVALIDATE METADATA tpch_kudu.customer >> > >>> >> > >>> (invalidate-tpch-core-impala-generated.sql): >> > >>> >> > >>> >> > >>> (load-tpch-core-impala-generated-kudu-none-none.sql): >> > >>> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem >> > >>> >> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql): >> > >>> INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM >> > tpch.lineitem >> > >>> >> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql): >> > >>> INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part >> > >>> >> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql): >> > >>> INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM >> > tpch.partsupp >> > >>> >> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql): >> > >>> INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM >> > tpch.supplier >> > >>> >> > >>> Data Loading from Impala failed with error: ImpalaBeeswaxException: >> > >>> Query aborted:Kudu error(s) reported, first error: Timed out: >> Failed >> > to >> > >>> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9 >> > >>> after 4 attempt(s): GetTableLocations { table: >> > >>> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 } >> > >>> failed: Client connection negotiation failed: client connection to >> > >>> 127.0.0.1:7051 >> > >>> >> > >>> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed >> to >> > >>> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34 >> > >>> after 4 attempt(s): GetTableLocations { table: >> > >>> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2), >> > >>> attempt: 1 } failed: Client connection negotiation failed: client >> > >>> connection to 127.0.0.1:7051 (1 of 21494 similar) >> > >>> >> > >>> >> > >>> Traceback (most recent call last): >> > >>> File "/home/impdev/Impala/bin/load-data.py", line 179, in >> > >>> exec_impala_query_from_file >> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql): >> > >>> INSERT OVERWRITE TABLE tpch_parquet.nation SELECT * FROM tpch.nation >> > >>> >> > >>> result = impala_client.execute(query) >> > >>> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line >> > 173, >> > >>> in execute >> > >>> handle = self.__execute_query(query_string.strip(), user=user) >> > >>> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line >> > 341, >> > >>> in __execute_query >> > >>> self.wait_for_completion(handle) >> > >>> File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line >> > 361, >> > >>> in wait_for_completion >> > >>> raise ImpalaBeeswaxException("Query aborted:" + error_log, None) >> > >>> ImpalaBeeswaxException: ImpalaBeeswaxException: >> > >>> Query aborted:Kudu error(s) reported, first error: Timed out: >> Failed >> > to >> > >>> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9 >> > >>> after 4 attempt(s): GetTableLocations { table: >> > >>> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 } >> > >>> failed: Client connection negotiation failed: client connection to >> > >>> 127.0.0.1:7051 >> > >>> >> > >>> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed >> to >> > >>> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34 >> > >>> after 4 attempt(s): GetTableLocations { table: >> > >>> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2), >> > >>> attempt: 1 } failed: Client connection negotiation failed: client >> > >>> connection to 127.0.0.1:7051 (1 of 21494 similar) >> > >>> >> > >>> >> > >>> On Fri, Feb 23, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com >> > >> > >>> wrote: >> > >>> >> > >>>> 8302608bdff15d6e8af324fc885dd0a8c2b44a5d >> > >>>> >> > >>>> On Fri, Feb 23, 2018 at 12:41 AM, Jim Apple <jbap...@cloudera.com> >> > >>>> wrote: >> > >>>> >> > >>>>> What commit hash is checked out? >> > >>>>> >> > >>>>> On Thu, Feb 22, 2018 at 8:23 PM Joseph Koshakow < >> kosh...@gmail.com> >> > >>>>> wrote: >> > >>>>> >> > >>>>> > Host OS is Ubunto 16.04.3 LTS. Docker OS is also Ubuntu 16.04 as >> > per >> > >>>>> > instructions. I have 202.61Gb of free space in the host >> environment >> > >>>>> > (304.58Gb size total). I'm not sure how to check how much space >> is >> > >>>>> left in >> > >>>>> > docker containers but the size of the one created for this is >> 97.81 >> > >>>>> Gb. >> > >>>>> > >> > >>>>> > -Joe >> > >>>>> > >> > >>>>> > On Thu, Feb 22, 2018 at 11:15 PM, Jim Apple < >> jbap...@cloudera.com> >> > >>>>> wrote: >> > >>>>> > >> > >>>>> > > What is the host OS? What is the docker OS? How much space is >> > left >> > >>>>> on >> > >>>>> > disk >> > >>>>> > > in each environment? >> > >>>>> > > >> > >>>>> > > On Thu, Feb 22, 2018 at 8:05 PM, Joseph Koshakow < >> > >>>>> kosh...@gmail.com> >> > >>>>> > > wrote: >> > >>>>> > > >> > >>>>> > > > Hi, >> > >>>>> > > > >> > >>>>> > > > I've been trying to set up my development environment using >> > >>>>> docker by >> > >>>>> > > > following the following instructions: https://cwiki.ap >> > >>>>> > > > ache.org/confluence/display/IMPALA/Docker+for+Impala+ >> > Developers >> > >>>>> > > > I'm able to get to the step where you run `source >> > >>>>> > > > ~/Impala/bin/bootsrap_development.sh`. Everything runs fine >> > >>>>> until I get >> > >>>>> > > > to this part: >> > >>>>> > > > Loading TPC-DS data (logging to >> /home/impdev/Impala/logs/data_ >> > >>>>> > > > loading/load-tpcds.log)... >> > >>>>> > > > >> > >>>>> > > > That hangs for about 30 minutes and then prints the >> following >> > >>>>> (Also >> > >>>>> > > > attached) before hanging again indefinitely until I quit >> out. >> > >>>>> I've >> > >>>>> > tried >> > >>>>> > > > multiple times and even deleted the docker image and started >> > >>>>> over. Any >> > >>>>> > > > ideas? >> > >>>>> > > > >> > >>>>> > > > FAILED (Took: 26 min 21 sec) >> > >>>>> > > > 'load-data tpch core' failed. Tail of log: >> > >>>>> > > > INVALIDATE METADATA tpch_kudu.nation >> > >>>>> > > > >> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): >> > >>>>> > > > INVALIDATE METADATA tpch_kudu.region >> > >>>>> > > > >> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): >> > >>>>> > > > INVALIDATE METADATA tpch_kudu.orders >> > >>>>> > > > >> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): >> > >>>>> > > > INVALIDATE METADATA tpch_kudu.customer >> > >>>>> > > > >> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql): >> > >>>>> > > > >> > >>>>> > > > >> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM >> > >>>>> > tpch.lineitem >> > >>>>> > > > >> > >>>>> > > > (load-tpch-core-impala-generated-kudu-none-none.sql): >> > >>>>> > > > INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM >> > tpch.lineitem >> > >>>>> > > > >> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM >> > tpch.part >> > >>>>> > > > >> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM >> > >>>>> > tpch.partsupp >> > >>>>> > > > >> > >>>>> > > > Data Loading from Impala failed with error: >> > >>>>> ImpalaBeeswaxException: >> > >>>>> > > > Query aborted:Kudu error(s) reported, first error: Timed >> out: >> > >>>>> Failed >> > >>>>> > to >> > >>>>> > > > write batch of 23532 ops to tablet >> > 3b2c6a27489d43028912a69ca7be8b >> > >>>>> 71 >> > >>>>> > > after >> > >>>>> > > > 4 attempt(s): GetTableLocations { table: >> > >>>>> 'impala::tpch_kudu.lineitem', >> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after >> > 10.000s >> > >>>>> (SENT) >> > >>>>> > > > >> > >>>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: >> > >>>>> Failed to >> > >>>>> > > > write batch of 23532 ops to tablet >> > 3b2c6a27489d43028912a69ca7be8b >> > >>>>> 71 >> > >>>>> > > after >> > >>>>> > > > 4 attempt(s): GetTableLocations { table: >> > >>>>> 'impala::tpch_kudu.lineitem', >> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after >> > 10.000s >> > >>>>> (SENT) >> > >>>>> > > (1 >> > >>>>> > > > of 23532 similar) >> > >>>>> > > > >> > >>>>> > > > >> > >>>>> > > > Traceback (most recent call last): >> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql): >> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM >> > >>>>> > tpch.supplier >> > >>>>> > > > File "/home/impdev/Impala/bin/load-data.py", line 179, in >> > >>>>> > > > exec_impala_query_from_file >> > >>>>> > > > >> > >>>>> > > > result = impala_client.execute(query) >> > >>>>> > > > File "/home/impdev/Impala/tests/bee >> swax/impala_beeswax.py", >> > >>>>> line 173, >> > >>>>> > > > in execute >> > >>>>> > > > handle = self.__execute_query(query_string.strip(), >> > >>>>> user=user) >> > >>>>> > > > File "/home/impdev/Impala/tests/bee >> swax/impala_beeswax.py", >> > >>>>> line 341, >> > >>>>> > > > in __execute_query >> > >>>>> > > > self.wait_for_completion(handle) >> > >>>>> > > > File "/home/impdev/Impala/tests/bee >> swax/impala_beeswax.py", >> > >>>>> line 361, >> > >>>>> > > > in wait_for_completion >> > >>>>> > > > raise ImpalaBeeswaxException("Query aborted:" + >> error_log, >> > >>>>> None) >> > >>>>> > > > ImpalaBeeswaxException: ImpalaBeeswaxException: >> > >>>>> > > > Query aborted:Kudu error(s) reported, first error: Timed >> out: >> > >>>>> Failed >> > >>>>> > to >> > >>>>> > > > write batch of 23532 ops to tablet >> > 3b2c6a27489d43028912a69ca7be8b >> > >>>>> 71 >> > >>>>> > > after >> > >>>>> > > > 4 attempt(s): GetTableLocations { table: >> > >>>>> 'impala::tpch_kudu.lineitem', >> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after >> > 10.000s >> > >>>>> (SENT) >> > >>>>> > > > >> > >>>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: >> > >>>>> Failed to >> > >>>>> > > > write batch of 23532 ops to tablet >> > 3b2c6a27489d43028912a69ca7be8b >> > >>>>> 71 >> > >>>>> > > after >> > >>>>> > > > 4 attempt(s): GetTableLocations { table: >> > >>>>> 'impala::tpch_kudu.lineitem', >> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed: >> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after >> > 10.000s >> > >>>>> (SENT) >> > >>>>> > > (1 >> > >>>>> > > > of 23532 similar) >> > >>>>> > > > >> > >>>>> > > > Thanks, >> > >>>>> > > > Joe Koshakow >> > >>>>> > > > >> > >>>>> > > > >> > >>>>> > > >> > >>>>> > >> > >>>>> >> > >>>> >> > >>>> >> > >>> >> > >> >> > > >> > >> > >
