CPU and IO went to near zero on the master and all nodes after about 1
hour. I am do not know if the bulk of rows were written within that hour
or after.
Is there any way you can read the table and try to validate if all of
the data was written?
A simple join will show me where it stopped, and if that was at a
specific point in scanning the source file top to bottom.
While we certainly want to look into this more to find the issue in
your case, you might have all of the data you need to start running
queries against the parquet files.
Simple row count comparison tells me about 5% of the rows are missing in
the destination, but I will be confirming that.
On 28 May 2015, at 13:24, Jason Altekruse wrote:
He mentioned in his original post that he saw CPU and IO on all of the
nodes for a while when the query was active, but it suddenly dropped
down
to low CPU usage and stopped producing files. It seems like we are
failing
to detect an error an cancel the query.
It is possible that the failure happened when we were finalizing the
query,
cleanup resources/file handles/ etc. Is there any way you can read the
table and try to validate if all of the data was written? You can try
to
run a few of the same queries against the tab delimited files and
resulting
parquet files to see if all of the records were written. While we
certainly
want to look into this more to find the issue in your case, you might
have
all of the data you need to start running queries against the parquet
files.
On Thu, May 28, 2015 at 10:06 AM, Andries Engelbrecht <
[email protected]> wrote:
The time seems pretty long for that file size. What type of file is
it?
Is the CTAS running single threaded?
—Andries
On May 28, 2015, at 9:37 AM, Matt <[email protected]> wrote:
How large is the data set you are working with, and your
cluster/nodes?
Just testing with that single 44GB source file currently, and my
test
cluster is made from 4 nodes, each with 8 CPU cores, 32GB RAM, a 6TB
Ext4
volume (RAID-10).
Drill defaults left as come in v1.0. I will be adjusting memory and
retrying the CTAS.
I know I can / should assign individual disks to HDFS, but as a test
cluster there are apps that expect data volumes to work on. A
dedicated
Hadoop production cluster would have a disk layout specific to the
task.
On 28 May 2015, at 12:26, Andries Engelbrecht wrote:
Just check the drillbit.log and drillbit.out files in the log
directory.
Before adjusting memory, see if that is an issue first. It was for
me,
but as Jason mentioned there can be other causes as well.
You adjust memory allocation in the drill-env.sh files, and have to
restart the drill bits.
How large is the data set you are working with, and your
cluster/nodes?
—Andries
On May 28, 2015, at 9:17 AM, Matt <[email protected]> wrote:
To make sure I am adjusting the correct config, these are heap
parameters within the Drill configure path, not for Hadoop or
Zookeeper?
On May 28, 2015, at 12:08 PM, Jason Altekruse <
[email protected]> wrote:
There should be no upper limit on the size of the tables you can
create
with Drill. Be advised that Drill does currently operate entirely
optimistically in regards to available resources. If a network
connection
between two drillbits fails during a query, we will not currently
re-schedule the work to make use of remaining nodes and network
connections
that are still live. While we have had a good amount of success
using
Drill
for data conversion, be aware that these conditions could cause
long
running queries to fail.
That being said, it isn't the only possible cause for such a
failure.
In
the case of a network failure we would expect to see a message
returned to
you that part of the query was unsuccessful and that it had been
cancelled.
Andries has a good suggestion in regards to checking the heap
memory,
this
should also be detected and reported back to you at the CLI, but
we
may be
failing to propagate the error back to the head node for the
query. I
believe writing parquet may still be the most heap-intensive
operation in
Drill, despite our efforts to refactor the write path to use
direct
memory
instead of on-heap for large buffers needed in the process of
creating
parquet files.
On Thu, May 28, 2015 at 8:43 AM, Matt <[email protected]> wrote:
Is 300MM records too much to do in a single CTAS statement?
After almost 23 hours I killed the query (^c) and it returned:
~~~
+-----------+----------------------------+
| Fragment | Number of records written |
+-----------+----------------------------+
| 1_20 | 13568824 |
| 1_15 | 12411822 |
| 1_7 | 12470329 |
| 1_12 | 13693867 |
| 1_5 | 13292136 |
| 1_18 | 13874321 |
| 1_16 | 13303094 |
| 1_9 | 13639049 |
| 1_10 | 13698380 |
| 1_22 | 13501073 |
| 1_8 | 13533736 |
| 1_2 | 13549402 |
| 1_21 | 13665183 |
| 1_0 | 13544745 |
| 1_4 | 13532957 |
| 1_19 | 12767473 |
| 1_17 | 13670687 |
| 1_13 | 13469515 |
| 1_23 | 12517632 |
| 1_6 | 13634338 |
| 1_14 | 13611322 |
| 1_3 | 13061900 |
| 1_11 | 12760978 |
+-----------+----------------------------+
23 rows selected (82294.854 seconds)
~~~
The sum of those record counts is 306,772,763 which is close to
the
320,843,454 in the source file:
~~~
0: jdbc:drill:zk=es05:2181> select count(*) FROM
root.`sample_201501.dat`;
+------------+
| EXPR$0 |
+------------+
| 320843454 |
+------------+
1 row selected (384.665 seconds)
~~~
It represents one month of data, 4 key columns and 38 numeric
measure
columns, which could also be partitioned daily. The test here
was to
create
monthly Parquet files to see how the min/max stats on Parquet
chunks
help
with range select performance.
Instead of a small number of large monthly RDBMS tables, I am
attempting
to determine how many Parquet files should be used with Drill /
HDFS.
On 27 May 2015, at 15:17, Matt wrote:
Attempting to create a Parquet backed table with a CTAS from an
44GB
tab
delimited file in HDFS. The process seemed to be running, as
CPU
and IO was
seen on all 4 nodes in this cluster, and .parquet files being
created in
the expected path.
In however in the last two hours or so, all nodes show near
zero
CPU or
IO, and the Last Modified date on the .parquet have not
changed.
Same time
delay shown in the Last Progress column in the active fragment
profile.
What approach can I take to determine what is happening (or
not)?
