[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264326#comment-16264326
]
Arina Ielchiieva commented on DRILL-5941:
-
Merged into Apache master with commit id
f8bbb7591235627a58a1f10584d7902d8251d221
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263508#comment-16263508
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/1030
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263123#comment-16263123
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user parthchandra commented on the issue:
https://github.com/apache/drill/pull/1030
+1
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261609#comment-16261609
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user ppadma commented on the issue:
https://github.com/apache/drill/pull/1030
+1. LGTM.
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260214#comment-16260214
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user ppadma commented on the issue:
https://github.com/apache/drill/pull/1030
@arina-ielchiieva Ideally, it would be better to do this properly first
time so we don't have to revisit. But, since there is a correctness issue, it
is ok to go with the approach you have and address the performance problem
later. I will finish review and post my comments soon.
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259349#comment-16259349
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1030
@ppadma performance impact will be only on those tables that have skip
header / footer functionality (one reader per file), for other tables
processing will be the same (one reader per input). Currently for tables that
have skip header / footer functionality and have multiple input splits, these
rows are skipped for each input, thus leaving user with incorrect data set.
Regarding your suggestion, as I have mentioned in my previous comment
(please see links to the code there as well) we use hadoop reader for the data
and don't have information about number of rows in input split (I wish we did,
it would make life much easier).
I have an idea how to parallelize readers when we have only header (though
still all readers will be on the same node) but when we have footer we'll have
to read one file per reader. Also we can consider Drill text reader usage
instead of hadoop one (as we do for parquet files).
Anyway, I suggest we commit these changes and create new Jira for future
improvement of skip header / footer performance. Will this be acceptable?
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258863#comment-16258863
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user ppadma commented on the issue:
https://github.com/apache/drill/pull/1030
@arina-ielchiieva I am concerned about performance impact by grouping all
splits in a single reader (essentially, not parallelizing at all).
Wondering if it is possible to do this way:
During planning, in HiveScan, if it is text file and has header/footer,
get the number of rows to skip. Read the header/footer rows and based on that,
adjust the first/last split and offset within them. The splits which have only
header/footer rows can be removed from inputSplits. In HiveSubScan, change
hiveReadEntry to be a list (one entry for each split). Add an entry in
hiveReadEntry, numRowsToSkip (or offsetToStart) which can be passed to the
recordReaders in getBatch for each subScan. This is fairly complicated and I am
sure I might be missing some details :-)
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255372#comment-16255372
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1030
@ppadma pull request is updated to handle header / footer logic in
distributed environment. Comment with PR description in the beginning of the
pull request is updated. Please review when possible.
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251801#comment-16251801
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1030
@ppadma
To create reader for each input split and maintain skip header / footer
functionality we need to know how many rows are in input split. Unfortunately,
input split does not hold such information, only number of bytes. [1] We can't
apply skip header functionality for the first input split and skip footer for
the last input either since we don't know how many rows will be skipped, it can
be the situation that we need to skip the whole first input split and partially
second.
@paul-rogers
To read from hive we actually use Hadoop reader [2, 3] so if I am not
mistaken unfortunately the described above approach can be applied.
[1]
https://hadoop.apache.org/docs/r2.7.0/api/org/apache/hadoop/mapred/FileSplit.html
[2]
https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveAbstractReader.java#L234
[3]
https://hadoop.apache.org/docs/r2.7.0/api/org/apache/hadoop/mapred/RecordReader.html
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250120#comment-16250120
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1030
For FWIW, the native CSV reader does the following:
* To read the header, it seeks to offset 0 in the file, regardless of the
block being read, then reads the header, which may be a remote read.
* The reader then seeks to the start of its block. If this is the first
block, it skips the header, else it searches for the start of the next record.
Since Hive has the same challenges, Hive must have solved this, we have
only to research that existing solution.
One simple solution is:
* If block number is 0, skip the header.
* If block number is 1 or larger, look for the next record separator.
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250007#comment-16250007
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user ppadma commented on the issue:
https://github.com/apache/drill/pull/1030
@arina-ielchiieva That will have performance impact. Better way would be
to keep one reader per split and see if we can figure out a way to tell readers
how many rows they should skip (for header or footer).
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249774#comment-16249774
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1030
@ppadma really good point! I did not realize that when creating readers
input splits can be already distributed among several drillbits. So I guess we
should make sure that when we have skip header / footer logic in table input
splits of the same file reside in one `HiveSubScan` during planning stage. I'll
have to make some changes and let you know once done.
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248259#comment-16248259
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user ppadma commented on the issue:
https://github.com/apache/drill/pull/1030
@arina-ielchiieva I have a question. If the splits for a file are spread
across multiple fragments, does this logic work ?
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245778#comment-16245778
]
ASF GitHub Bot commented on DRILL-5941:
---
Github user priteshm commented on the issue:
https://github.com/apache/drill/pull/1030
@ppadma can you review this?
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5941) Skip header / footer logic works incorrectly for Hive tables when file has several input splits
[
https://issues.apache.org/jira/browse/DRILL-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245686#comment-16245686
]
ASF GitHub Bot commented on DRILL-5941:
---
GitHub user arina-ielchiieva opened a pull request:
https://github.com/apache/drill/pull/1030
DRILL-5941: Skip header / footer improvements for Hive storage plugin
Overview:
1. When table has header / footer process input splits fo the same file in
one reader (bug fix for DRILL-5941).
2. Apply skip header logic during reader initialization only once to avoid
checks during reading the data (DRILL-5106).
3. Apply skip footer logic only when footer is more then 0, otherwise
default processing will be done without buffering data in queue (DRIL-5106).
Code changes:
1. AbstractReadersInitializer was introduced to factor out common logic
during readers intialization.
It will have three implementations:
a. Default (each input split gets its own reader);
b. Empty (for empty tables);
c. InputSplitGroups (applied when table has header / footer and input
splits of the same file should be processed together).
2. AbstractRecordsInspector was introduced to improve performance when
table has footer is less or equals to 0.
It will have two implementations:
a. Default (records will be processed one by one without buffering);
b. SkipFooter (queue will be used to buffer N records that should be
skipped in the end of file processing).
3. Allow HiveAbstractReader to have multiple input splits.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/arina-ielchiieva/drill DRILL-5941
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1030.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1030
> Skip header / footer logic works incorrectly for Hive tables when file has
> several input splits
> ---
>
> Key: DRILL-5941
> URL: https://issues.apache.org/jira/browse/DRILL-5941
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: 1.12.0
>
>
> *To reproduce*
> 1. Create csv file with two columns (key, value) for 329 rows, where
> first row is a header.
> The data file has size of should be greater than chunk size of 256 MB. Copy
> file to the distributed file system.
> 2. Create table in Hive:
> {noformat}
> CREATE EXTERNAL TABLE `h_table`(
> `key` bigint,
> `value` string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'maprfs:/tmp/h_table'
> TBLPROPERTIES (
> 'skip.header.line.count'='1');
> {noformat}
> 3. Execute query {{select * from hive.h_table}} in Drill (query data using
> Hive plugin). The result will return less rows then expected. Expected result
> is 328 (total count minus one row as header).
> *The root cause*
> Since file is greater than default chunk size, it's split into several
> fragments, known as input splits. For example:
> {noformat}
> maprfs:/tmp/h_table/h_table.csv:0+268435456
> maprfs:/tmp/h_table/h_table.csv:268435457+492782112
> {noformat}
> TextHiveReader is responsible for handling skip header and / or footer logic.
> Currently Drill creates reader [for each input
> split|https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScanBatchCreator.java#L84]
> and skip header and /or footer logic is applied for each input splits,
> though ideally the above mentioned input splits should have been read by one
> reader, so skip / header footer logic was applied correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
