I was able to repro a failed data load with 8GB of memory. You have more
than that, but it's still possible that's the issue.

On Sun, Apr 8, 2018 at 7:58 PM, Jim Apple <jbap...@cloudera.com> wrote:

> It's certainly possible that this is an OOM issue. I'll try to repro that.
>
> On Sun, Apr 8, 2018 at 12:54 PM, Joseph Koshakow <kosh...@gmail.com>
> wrote:
>
>> Jim,
>> I followed the commands from this page:
>> https://cwiki.apache.org/confluence/display/IMPALA/Docker+
>> for+Impala+Developers
>> which
>> is almost identical to the commands you listed.
>>
>> Philip,
>> /etc/hosts contents:
>> 127.0.0.1       localhost
>> ::1     localhost ip6-localhost ip6-loopback
>> fe00::0 ip6-localnet
>> ff00::0 ip6-mcastprefix
>> ff02::1 ip6-allnodes
>> ff02::2 ip6-allrouters
>> 172.17.0.2      c4fdb0931b2d
>>
>>
>> tests-with-docker:
>> impdev@c4fdb0931b2d:~/Impala$ ./docker/test-with-docker.py --suite NOOP
>> --no-cleanup-image
>> 2018-04-08 19:46:22,638 MainThread: CPUs: 4 Memory (GB): 11.6096229553
>> 2018-04-08 19:46:22,638 MainThread: This tool should be run on a machine
>> with more memory.
>> 2018-04-08 19:46:22,640 MainThread: Running: (['git', 'rev-parse',
>> '--show-toplevel'],), {}; cmdline: git rev-parse --show-toplevel.
>> 2018-04-08 19:46:22,642 MainThread: Arguments: Namespace(build_image=None,
>> ccache_dir='/home/impdev/.ccache', cleanup_containers=True,
>> cleanup_image=False, impalad_mem_limit_bytes=7516192768,
>> name='i-20180408-194622', parallel_test_concurrency=4, suite=['NOOP'],
>> suite_concurrency=1, test=False, timeout=120)
>> 2018-04-08 19:46:22,643 MainThread: Running: (['docker', 'create',
>> '--privileged', '--name', 'i-20180408-194622', '--hostname',
>> 'i-20180408-194622', '--label=pwd=/home/impdev/Impala', '-e', 'LC_ALL=C',
>> '-e', 'IMPALAD_MEM_LIMIT_BYTES=7516192768', '-v',
>> '/home/impdev/Impala:/repo:ro', '-v', '/home/impdev/.ccache:/ccache',
>> '-v',
>> '/etc/localtime:/mnt/localtime', '-v',
>> '/home/impdev/Impala/logs/docker/i-20180408-194622/build:/logs', '-v',
>> '/home/impdev/Impala/docker:/mnt/base:ro', 'ubuntu:16.04',
>> '/mnt/base/entrypoint.sh', 'build', '1000'],), {}; cmdline: docker create
>> --privileged --name i-20180408-194622 --hostname i-20180408-194622
>> --label=pwd=/home/impdev/Impala -e LC_ALL=C -e
>> IMPALAD_MEM_LIMIT_BYTES=7516192768 -v /home/impdev/Impala:/repo:ro -v
>> /home/impdev/.ccache:/ccache -v /etc/localtime:/mnt/localtime -v
>> /home/impdev/Impala/logs/docker/i-20180408-194622/build:/logs -v
>> /home/impdev/Impala/docker:/mnt/base:ro ubuntu:16.04
>> /mnt/base/entrypoint.sh build 1000.
>> 2018-04-08 19:46:22,649 Thread-1: Using cgroups: cpuacct
>> /sys/fs/cgroup/cpu,cpuacct, memory /sys/fs/cgroup/memory
>> 2018-04-08 19:46:22,649 MainThread: Memory usage: None GB min, None GB max
>> Traceback (most recent call last):
>>   File "./docker/test-with-docker.py", line 579, in <module>
>>     main()
>>   File "./docker/test-with-docker.py", line 198, in main
>>     ret = t.run()
>>   File "./docker/test-with-docker.py", line 494, in run
>>     self._create_build_image()
>>   File "./docker/test-with-docker.py", line 434, in _create_build_image
>>     entrypoint=["/mnt/base/entrypoint.sh", "build", str(os.getuid())])
>>   File "./docker/test-with-docker.py", line 376, in _create_container
>>     + entrypoint).strip()
>>   File "./docker/test-with-docker.py", line 279, in _check_output
>>     return subprocess.check_output(*args, **kwargs)
>>   File "/usr/lib/python2.7/subprocess.py", line 567, in check_output
>>     process = Popen(stdout=PIPE, *popenargs, **kwargs)
>>   File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
>>     errread, errwrite)
>>   File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
>>     raise child_exception
>> OSError: [Errno 2] No such file or directory
>>
>> Contents of /mnt:
>> impdev@c4fdb0931b2d:~/Impala$ ls -lart /mnt/
>> total 8
>> drwxr-xr-x  2 root root 4096 Feb 28 19:13 .
>> drwxr-xr-x 44 root root 4096 Apr  7 07:07 ..
>>
>>
>> On Sun, Apr 8, 2018 at 12:13 AM, Jim Apple <jbap...@cloudera.com> wrote:
>>
>> > I was unable to reproduce this in a fresh ec2 instance. I ran:
>> >
>> > sudo docker run --interactive --privileged --tty ubuntu:16.04
>> > # inside the container
>> > apt-get update
>> > apt-get install -y sudo
>> > adduser --disabled-password --gecos '' impdev
>> > echo 'impdev ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
>> > sudo su - impdev
>> > # as impdev
>> > sudo apt install -y git
>> > git clone https://git-wip-us.apache.org/repos/asf/impala.git ~/Impala
>> >
>> >
>> > time ~/Impala/bin/bootstrap_development.sh
>> >
>> > On Sat, Apr 7, 2018 at 6:44 PM, Jim Apple <jbap...@cloudera.com> wrote:
>> >
>> > > The attachment didn't attach. This is a common issue on our mailing
>> > lists.
>> > > I've started another thread about it.
>> > >
>> > > I'll try to reproduce this. My first suspicions are disk space,
>> > networking
>> > > restrictions, and https://issues.apache.org/jira/browse/IMPALA-6737
>> > >
>> > > Inside of the container, can you run df -h before exiting?
>> > >
>> > > On Sat, Apr 7, 2018 at 5:28 PM, Joseph Koshakow <kosh...@gmail.com>
>> > wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> Tried this again recently and still experiencing the same error. I've
>> > >> attached a copy of all my shell commands and output.
>> > >>
>> > >> On Sat, Feb 24, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> Though I just upgraded to the most recent version. It's still
>> failing
>> > >>> but seems to be making it one step farther. I see from the emails
>> that
>> > >>> there's some issues with the current build so I'll just wait until
>> > that's
>> > >>> all resolved. The output from my most recent attempt is below if you
>> > have a
>> > >>> use for it.
>> > >>>
>> > >>> Loading custom schemas OK (Took: 0 min 40 sec)
>> > >>> Started Loading functional-query data in background; pid 22229.
>> > >>> Loading functional-query data (logging to
>> > /home/impdev/Impala/logs/data_
>> > >>> loading/load-functional-query.log)...
>> > >>> Started Loading TPC-H data in background; pid 22230.
>> > >>> Started Loading TPC-DS data in background; pid 22231.
>> > >>> Loading TPC-DS data (logging to /home/impdev/Impala/logs/data_
>> > >>> loading/load-tpcds.log)...
>> > >>> Loading TPC-H data (logging to /home/impdev/Impala/logs/data_
>> > >>> loading/load-tpch.log)...
>> > >>>     FAILED (Took: 25 min 51 sec)
>> > >>>     'load-data tpch core' failed. Tail of log:
>> > >>> INVALIDATE METADATA tpch_kudu.region
>> > >>>
>> > >>> (invalidate-tpch-core-impala-generated.sql):
>> > >>> INVALIDATE METADATA tpch_kudu.orders
>> > >>>
>> > >>> (invalidate-tpch-core-impala-generated.sql):
>> > >>> INVALIDATE METADATA tpch_kudu.customer
>> > >>>
>> > >>> (invalidate-tpch-core-impala-generated.sql):
>> > >>>
>> > >>>
>> > >>> (load-tpch-core-impala-generated-kudu-none-none.sql):
>> > >>> INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM tpch.lineitem
>> > >>>
>> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql):
>> > >>> INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM
>> > tpch.lineitem
>> > >>>
>> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql):
>> > >>> INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM tpch.part
>> > >>>
>> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql):
>> > >>> INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM
>> > tpch.partsupp
>> > >>>
>> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql):
>> > >>> INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM
>> > tpch.supplier
>> > >>>
>> > >>> Data Loading from Impala failed with error: ImpalaBeeswaxException:
>> > >>>  Query aborted:Kudu error(s) reported, first error: Timed out:
>> Failed
>> > to
>> > >>> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9
>> > >>> after 4 attempt(s): GetTableLocations { table:
>> > >>> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 }
>> > >>> failed: Client connection negotiation failed: client connection to
>> > >>> 127.0.0.1:7051
>> > >>>
>> > >>> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed
>> to
>> > >>> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34
>> > >>> after 4 attempt(s): GetTableLocations { table:
>> > >>> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2),
>> > >>> attempt: 1 } failed: Client connection negotiation failed: client
>> > >>> connection to 127.0.0.1:7051 (1 of 21494 similar)
>> > >>>
>> > >>>
>> > >>> Traceback (most recent call last):
>> > >>>   File "/home/impdev/Impala/bin/load-data.py", line 179, in
>> > >>> exec_impala_query_from_file
>> > >>> (load-tpch-core-impala-generated-parquet-none-none.sql):
>> > >>> INSERT OVERWRITE TABLE tpch_parquet.nation SELECT * FROM tpch.nation
>> > >>>
>> > >>>     result = impala_client.execute(query)
>> > >>>   File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line
>> > 173,
>> > >>> in execute
>> > >>>     handle = self.__execute_query(query_string.strip(), user=user)
>> > >>>   File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line
>> > 341,
>> > >>> in __execute_query
>> > >>>     self.wait_for_completion(handle)
>> > >>>   File "/home/impdev/Impala/tests/beeswax/impala_beeswax.py", line
>> > 361,
>> > >>> in wait_for_completion
>> > >>>     raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
>> > >>> ImpalaBeeswaxException: ImpalaBeeswaxException:
>> > >>>  Query aborted:Kudu error(s) reported, first error: Timed out:
>> Failed
>> > to
>> > >>> write batch of 21493 ops to tablet 5c5c917295d642edb9cdbe152f3283d9
>> > >>> after 4 attempt(s): GetTableLocations { table:
>> > >>> 'impala::tpch_kudu.lineitem', partition-key: (<start>), attempt: 1 }
>> > >>> failed: Client connection negotiation failed: client connection to
>> > >>> 127.0.0.1:7051
>> > >>>
>> > >>> Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out: Failed
>> to
>> > >>> write batch of 21537 ops to tablet e9f68f0ee8e945ebb7630947169a6c34
>> > >>> after 4 attempt(s): GetTableLocations { table:
>> > >>> 'impala::tpch_kudu.lineitem', partition-key: (HASH (l_orderkey): 2),
>> > >>> attempt: 1 } failed: Client connection negotiation failed: client
>> > >>> connection to 127.0.0.1:7051 (1 of 21494 similar)
>> > >>>
>> > >>>
>> > >>> On Fri, Feb 23, 2018 at 6:34 PM, Joseph Koshakow <kosh...@gmail.com
>> >
>> > >>> wrote:
>> > >>>
>> > >>>> 8302608bdff15d6e8af324fc885dd0a8c2b44a5d
>> > >>>>
>> > >>>> On Fri, Feb 23, 2018 at 12:41 AM, Jim Apple <jbap...@cloudera.com>
>> > >>>> wrote:
>> > >>>>
>> > >>>>> What commit hash is checked out?
>> > >>>>>
>> > >>>>> On Thu, Feb 22, 2018 at 8:23 PM Joseph Koshakow <
>> kosh...@gmail.com>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>> > Host OS is Ubunto 16.04.3 LTS. Docker OS is also Ubuntu 16.04 as
>> > per
>> > >>>>> > instructions. I have 202.61Gb of free space in the host
>> environment
>> > >>>>> > (304.58Gb size total). I'm not sure how to check how much space
>> is
>> > >>>>> left in
>> > >>>>> > docker containers but the size of the one created for this is
>> 97.81
>> > >>>>> Gb.
>> > >>>>> >
>> > >>>>> > -Joe
>> > >>>>> >
>> > >>>>> > On Thu, Feb 22, 2018 at 11:15 PM, Jim Apple <
>> jbap...@cloudera.com>
>> > >>>>> wrote:
>> > >>>>> >
>> > >>>>> > > What is the host OS? What is the docker OS? How much space is
>> > left
>> > >>>>> on
>> > >>>>> > disk
>> > >>>>> > > in each environment?
>> > >>>>> > >
>> > >>>>> > > On Thu, Feb 22, 2018 at 8:05 PM, Joseph Koshakow <
>> > >>>>> kosh...@gmail.com>
>> > >>>>> > > wrote:
>> > >>>>> > >
>> > >>>>> > > > Hi,
>> > >>>>> > > >
>> > >>>>> > > > I've been trying to set up my development environment using
>> > >>>>> docker by
>> > >>>>> > > > following the following instructions: https://cwiki.ap
>> > >>>>> > > > ache.org/confluence/display/IMPALA/Docker+for+Impala+
>> > Developers
>> > >>>>> > > > I'm able to get to the step where you run `source
>> > >>>>> > > > ~/Impala/bin/bootsrap_development.sh`. Everything runs fine
>> > >>>>> until I get
>> > >>>>> > > > to this part:
>> > >>>>> > > > Loading TPC-DS data (logging to
>> /home/impdev/Impala/logs/data_
>> > >>>>> > > > loading/load-tpcds.log)...
>> > >>>>> > > >
>> > >>>>> > > > That hangs for about 30 minutes and then prints the
>> following
>> > >>>>> (Also
>> > >>>>> > > > attached) before hanging again indefinitely until I quit
>> out.
>> > >>>>> I've
>> > >>>>> > tried
>> > >>>>> > > > multiple times and even deleted the docker image and started
>> > >>>>> over. Any
>> > >>>>> > > > ideas?
>> > >>>>> > > >
>> > >>>>> > > > FAILED (Took: 26 min 21 sec)
>> > >>>>> > > >     'load-data tpch core' failed. Tail of log:
>> > >>>>> > > > INVALIDATE METADATA tpch_kudu.nation
>> > >>>>> > > >
>> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql):
>> > >>>>> > > > INVALIDATE METADATA tpch_kudu.region
>> > >>>>> > > >
>> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql):
>> > >>>>> > > > INVALIDATE METADATA tpch_kudu.orders
>> > >>>>> > > >
>> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql):
>> > >>>>> > > > INVALIDATE METADATA tpch_kudu.customer
>> > >>>>> > > >
>> > >>>>> > > > (invalidate-tpch-core-impala-generated.sql):
>> > >>>>> > > >
>> > >>>>> > > >
>> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
>> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.lineitem SELECT * FROM
>> > >>>>> > tpch.lineitem
>> > >>>>> > > >
>> > >>>>> > > > (load-tpch-core-impala-generated-kudu-none-none.sql):
>> > >>>>> > > > INSERT INTO TABLE tpch_kudu.lineitem SELECT * FROM
>> > tpch.lineitem
>> > >>>>> > > >
>> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
>> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.part SELECT * FROM
>> > tpch.part
>> > >>>>> > > >
>> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
>> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.partsupp SELECT * FROM
>> > >>>>> > tpch.partsupp
>> > >>>>> > > >
>> > >>>>> > > > Data Loading from Impala failed with error:
>> > >>>>> ImpalaBeeswaxException:
>> > >>>>> > > >  Query aborted:Kudu error(s) reported, first error: Timed
>> out:
>> > >>>>> Failed
>> > >>>>> > to
>> > >>>>> > > > write batch of 23532 ops to tablet
>> > 3b2c6a27489d43028912a69ca7be8b
>> > >>>>> 71
>> > >>>>> > > after
>> > >>>>> > > > 4 attempt(s): GetTableLocations { table:
>> > >>>>> 'impala::tpch_kudu.lineitem',
>> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
>> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
>> > 10.000s
>> > >>>>> (SENT)
>> > >>>>> > > >
>> > >>>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out:
>> > >>>>> Failed to
>> > >>>>> > > > write batch of 23532 ops to tablet
>> > 3b2c6a27489d43028912a69ca7be8b
>> > >>>>> 71
>> > >>>>> > > after
>> > >>>>> > > > 4 attempt(s): GetTableLocations { table:
>> > >>>>> 'impala::tpch_kudu.lineitem',
>> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
>> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
>> > 10.000s
>> > >>>>> (SENT)
>> > >>>>> > > (1
>> > >>>>> > > > of 23532 similar)
>> > >>>>> > > >
>> > >>>>> > > >
>> > >>>>> > > > Traceback (most recent call last):
>> > >>>>> > > > (load-tpch-core-impala-generated-parquet-none-none.sql):
>> > >>>>> > > > INSERT OVERWRITE TABLE tpch_parquet.supplier SELECT * FROM
>> > >>>>> > tpch.supplier
>> > >>>>> > > >   File "/home/impdev/Impala/bin/load-data.py", line 179, in
>> > >>>>> > > > exec_impala_query_from_file
>> > >>>>> > > >
>> > >>>>> > > >     result = impala_client.execute(query)
>> > >>>>> > > >   File "/home/impdev/Impala/tests/bee
>> swax/impala_beeswax.py",
>> > >>>>> line 173,
>> > >>>>> > > > in execute
>> > >>>>> > > >     handle = self.__execute_query(query_string.strip(),
>> > >>>>> user=user)
>> > >>>>> > > >   File "/home/impdev/Impala/tests/bee
>> swax/impala_beeswax.py",
>> > >>>>> line 341,
>> > >>>>> > > > in __execute_query
>> > >>>>> > > >     self.wait_for_completion(handle)
>> > >>>>> > > >   File "/home/impdev/Impala/tests/bee
>> swax/impala_beeswax.py",
>> > >>>>> line 361,
>> > >>>>> > > > in wait_for_completion
>> > >>>>> > > >     raise ImpalaBeeswaxException("Query aborted:" +
>> error_log,
>> > >>>>> None)
>> > >>>>> > > > ImpalaBeeswaxException: ImpalaBeeswaxException:
>> > >>>>> > > >  Query aborted:Kudu error(s) reported, first error: Timed
>> out:
>> > >>>>> Failed
>> > >>>>> > to
>> > >>>>> > > > write batch of 23532 ops to tablet
>> > 3b2c6a27489d43028912a69ca7be8b
>> > >>>>> 71
>> > >>>>> > > after
>> > >>>>> > > > 4 attempt(s): GetTableLocations { table:
>> > >>>>> 'impala::tpch_kudu.lineitem',
>> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
>> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
>> > 10.000s
>> > >>>>> (SENT)
>> > >>>>> > > >
>> > >>>>> > > > Error in Kudu table 'impala::tpch_kudu.lineitem': Timed out:
>> > >>>>> Failed to
>> > >>>>> > > > write batch of 23532 ops to tablet
>> > 3b2c6a27489d43028912a69ca7be8b
>> > >>>>> 71
>> > >>>>> > > after
>> > >>>>> > > > 4 attempt(s): GetTableLocations { table:
>> > >>>>> 'impala::tpch_kudu.lineitem',
>> > >>>>> > > > partition-key: (HASH (l_orderkey): 1), attempt: 1 } failed:
>> > >>>>> > > > GetTableLocations RPC to 127.0.0.1:7051 timed out after
>> > 10.000s
>> > >>>>> (SENT)
>> > >>>>> > > (1
>> > >>>>> > > > of 23532 similar)
>> > >>>>> > > >
>> > >>>>> > > > Thanks,
>> > >>>>> > > > Joe Koshakow
>> > >>>>> > > >
>> > >>>>> > > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> > >>>>
>> > >>>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>

Reply via email to