[ https://issues.apache.org/jira/browse/DRILL-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arina Ielchiieva updated DRILL-3688: ------------------------------------ Description: Currently Drill does not honor the "skip.header.line.count" attribute of Hive table. It may cause some other format conversion issue. Reproduce: 1. Create a Hive table {code} create table h1db.testheader(col0 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE tblproperties("skip.header.line.count"="1"); {code} 2. Prepare a sample data: {code} # cat test.data col0 2015-01-01 {code} 3. Load sample data into Hive {code} LOAD DATA LOCAL INPATH '/xxx/test.data' OVERWRITE INTO TABLE h1db.testheader; {code} 4. Hive {code} hive> select * from h1db.testheader ; OK 2015-01-01 Time taken: 0.254 seconds, Fetched: 1 row(s) {code} 5. Drill {code} > select * from hive.h1db.testheader ; +-------------+ | col0 | +-------------+ | col0 | | 2015-01-01 | +-------------+ 2 rows selected (0.257 seconds) > select cast(col0 as date) from hive.h1db.testheader ; Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must be in the range [1,12] Fragment 0:0 [Error Id: 34353702-ca27-440b-a4f4-0c9f79fc8ccd on h1.poc.com:31010] (org.joda.time.IllegalFieldValueException) Value 0 for monthOfYear must be in the range [1,12] org.joda.time.field.FieldUtils.verifyValueBounds():236 org.joda.time.chrono.BasicChronology.getDateMidnightMillis():613 org.joda.time.chrono.BasicChronology.getDateTimeMillis():159 org.joda.time.chrono.AssembledChronology.getDateTimeMillis():120 org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.memGetDate():261 org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getDate():218 org.apache.drill.exec.test.generated.ProjectorGen0.doEval():67 org.apache.drill.exec.test.generated.ProjectorGen0.projectRecords():62 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork():172 org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129 org.apache.drill.exec.record.AbstractRecordBatch.next():147 org.apache.drill.exec.physical.impl.BaseRootExec.next():83 org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():79 org.apache.drill.exec.physical.impl.BaseRootExec.next():73 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():261 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():255 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():422 org.apache.hadoop.security.UserGroupInformation.doAs():1566 org.apache.drill.exec.work.fragment.FragmentExecutor.run():255 org.apache.drill.common.SelfCleaningRunnable.run():38 java.util.concurrent.ThreadPoolExecutor.runWorker():1142 java.util.concurrent.ThreadPoolExecutor$Worker.run():617 java.lang.Thread.run():745 (state=,code=0) {code} Also "skip.footer.line.count" should be taken into account. If "skip.header.line.count" or "skip.footer.line.count" has incorrect value in Hive, throw appropriate exception in Drill. Ex: Hive table property skip.header.line.count value 'someValue' is non-numeric was: Currently Drill does not honor the "skip.header.line.count" attribute of Hive table. It may cause some other format conversion issue. Reproduce: 1. Create a Hive table {code} create table h1db.testheader(col0 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE tblproperties("skip.header.line.count"="1"); {code} 2. Prepare a sample data: {code} # cat test.data col0 2015-01-01 {code} 3. Load sample data into Hive {code} LOAD DATA LOCAL INPATH '/xxx/test.data' OVERWRITE INTO TABLE h1db.testheader; {code} 4. Hive {code} hive> select * from h1db.testheader ; OK 2015-01-01 Time taken: 0.254 seconds, Fetched: 1 row(s) {code} 5. Drill {code} > select * from hive.h1db.testheader ; +-------------+ | col0 | +-------------+ | col0 | | 2015-01-01 | +-------------+ 2 rows selected (0.257 seconds) > select cast(col0 as date) from hive.h1db.testheader ; Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must be in the range [1,12] Fragment 0:0 [Error Id: 34353702-ca27-440b-a4f4-0c9f79fc8ccd on h1.poc.com:31010] (org.joda.time.IllegalFieldValueException) Value 0 for monthOfYear must be in the range [1,12] org.joda.time.field.FieldUtils.verifyValueBounds():236 org.joda.time.chrono.BasicChronology.getDateMidnightMillis():613 org.joda.time.chrono.BasicChronology.getDateTimeMillis():159 org.joda.time.chrono.AssembledChronology.getDateTimeMillis():120 org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.memGetDate():261 org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getDate():218 org.apache.drill.exec.test.generated.ProjectorGen0.doEval():67 org.apache.drill.exec.test.generated.ProjectorGen0.projectRecords():62 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork():172 org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129 org.apache.drill.exec.record.AbstractRecordBatch.next():147 org.apache.drill.exec.physical.impl.BaseRootExec.next():83 org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():79 org.apache.drill.exec.physical.impl.BaseRootExec.next():73 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():261 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():255 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():422 org.apache.hadoop.security.UserGroupInformation.doAs():1566 org.apache.drill.exec.work.fragment.FragmentExecutor.run():255 org.apache.drill.common.SelfCleaningRunnable.run():38 java.util.concurrent.ThreadPoolExecutor.runWorker():1142 java.util.concurrent.ThreadPoolExecutor$Worker.run():617 java.lang.Thread.run():745 (state=,code=0) {code} Summary: Drill should honor "skip.header.line.count" and "skip.footer.line.count" attributes of Hive table (was: Drill should honor "skip.header.line.count" attribute of Hive table) > Drill should honor "skip.header.line.count" and "skip.footer.line.count" > attributes of Hive table > ------------------------------------------------------------------------------------------------- > > Key: DRILL-3688 > URL: https://issues.apache.org/jira/browse/DRILL-3688 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive > Affects Versions: 1.1.0 > Environment: 1.1 > Reporter: Hao Zhu > Assignee: Arina Ielchiieva > Fix For: Future > > > Currently Drill does not honor the "skip.header.line.count" attribute of Hive > table. > It may cause some other format conversion issue. > Reproduce: > 1. Create a Hive table > {code} > create table h1db.testheader(col0 string) > ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' > STORED AS TEXTFILE > tblproperties("skip.header.line.count"="1"); > {code} > 2. Prepare a sample data: > {code} > # cat test.data > col0 > 2015-01-01 > {code} > 3. Load sample data into Hive > {code} > LOAD DATA LOCAL INPATH '/xxx/test.data' OVERWRITE INTO TABLE h1db.testheader; > {code} > 4. Hive > {code} > hive> select * from h1db.testheader ; > OK > 2015-01-01 > Time taken: 0.254 seconds, Fetched: 1 row(s) > {code} > 5. Drill > {code} > > select * from hive.h1db.testheader ; > +-------------+ > | col0 | > +-------------+ > | col0 | > | 2015-01-01 | > +-------------+ > 2 rows selected (0.257 seconds) > > select cast(col0 as date) from hive.h1db.testheader ; > Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must > be in the range [1,12] > Fragment 0:0 > [Error Id: 34353702-ca27-440b-a4f4-0c9f79fc8ccd on h1.poc.com:31010] > (org.joda.time.IllegalFieldValueException) Value 0 for monthOfYear must be > in the range [1,12] > org.joda.time.field.FieldUtils.verifyValueBounds():236 > org.joda.time.chrono.BasicChronology.getDateMidnightMillis():613 > org.joda.time.chrono.BasicChronology.getDateTimeMillis():159 > org.joda.time.chrono.AssembledChronology.getDateTimeMillis():120 > org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.memGetDate():261 > org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getDate():218 > org.apache.drill.exec.test.generated.ProjectorGen0.doEval():67 > org.apache.drill.exec.test.generated.ProjectorGen0.projectRecords():62 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork():172 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129 > org.apache.drill.exec.record.AbstractRecordBatch.next():147 > org.apache.drill.exec.physical.impl.BaseRootExec.next():83 > > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():79 > org.apache.drill.exec.physical.impl.BaseRootExec.next():73 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():261 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():255 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():422 > org.apache.hadoop.security.UserGroupInformation.doAs():1566 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():255 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1142 > java.util.concurrent.ThreadPoolExecutor$Worker.run():617 > java.lang.Thread.run():745 (state=,code=0) > {code} > Also "skip.footer.line.count" should be taken into account. > If "skip.header.line.count" or "skip.footer.line.count" has incorrect value > in Hive, throw appropriate exception in Drill. > Ex: Hive table property skip.header.line.count value 'someValue' is > non-numeric -- This message was sent by Atlassian JIRA (v6.3.4#6332)