[jira] [Commented] (DRILL-4203) Parquet File : Date is stored wrongly

[ASF GitHub Bot (JIRA)]

    [ 
https://issues.apache.org/jira/browse/DRILL-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534229#comment-15534229
 ] 

ASF GitHub Bot commented on DRILL-4203:
---------------------------------------

Github user parthchandra commented on a diff in the pull request:

    https://github.com/apache/drill/pull/595#discussion_r81240013
  
    --- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderUtility.java
 ---
    @@ -18,16 +18,62 @@
     package org.apache.drill.exec.store.parquet;
     
     import org.apache.drill.common.exceptions.UserException;
    +import org.apache.drill.common.expression.PathSegment;
    +import org.apache.drill.common.expression.SchemaPath;
     import org.apache.drill.exec.planner.physical.PlannerSettings;
     import org.apache.drill.exec.server.options.OptionManager;
    +import org.apache.drill.exec.store.AbstractRecordReader;
    +import org.apache.drill.exec.store.ParquetOutputRecordWriter;
     import org.apache.drill.exec.work.ExecErrorConstants;
    +import org.apache.parquet.SemanticVersion;
    +import org.apache.parquet.VersionParser;
    +import org.apache.parquet.column.ColumnDescriptor;
    +import org.apache.parquet.column.statistics.Statistics;
    +import org.apache.parquet.format.ConvertedType;
    +import org.apache.parquet.format.FileMetaData;
    +import org.apache.parquet.format.SchemaElement;
    +import org.apache.parquet.format.converter.ParquetMetadataConverter;
    +import org.apache.parquet.hadoop.ParquetFileWriter;
    +import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
    +import org.apache.parquet.hadoop.metadata.ColumnPath;
    +import org.apache.parquet.hadoop.metadata.ParquetMetadata;
    +import org.apache.parquet.schema.OriginalType;
    +import org.joda.time.Chronology;
    +import org.joda.time.DateTimeConstants;
    +
    +import java.util.Arrays;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
     
     /*
      * Utility class where we can capture common logic between the two parquet 
readers
      */
     public class ParquetReaderUtility {
       private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(ParquetReaderUtility.class);
     
    +  // Note the negation symbol in the beginning
    +  public static final double CORRECT_CORRUPT_DATE_SHIFT = 
-ParquetOutputRecordWriter.JULIAN_DAY_EPOC - 0.5;
    --- End diff --
    
    Is the the number by which you must correct the incorrect values? If so, 
please add a comment. This had me very confused for a while.



> Parquet File : Date is stored wrongly
> -------------------------------------
>
>                 Key: DRILL-4203
>                 URL: https://issues.apache.org/jira/browse/DRILL-4203
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.4.0
>            Reporter: Stéphane Trou
>            Assignee: Vitalii Diravka
>            Priority: Critical
>
> Hello,
> I have some problems when i try to read parquet files produce by drill with  
> Spark,  all dates are corrupted.
> I think the problem come from drill :)
> {code}
> cat /tmp/date_parquet.csv 
> Epoch,1970-01-01
> {code}
> {code}
> 0: jdbc:drill:zk=local> select columns[0] as name, cast(columns[1] as date) 
> as epoch_date from dfs.tmp.`date_parquet.csv`;
> +--------+-------------+
> |  name  | epoch_date  |
> +--------+-------------+
> | Epoch  | 1970-01-01  |
> +--------+-------------+
> {code}
> {code}
> 0: jdbc:drill:zk=local> create table dfs.tmp.`buggy_parquet`as select 
> columns[0] as name, cast(columns[1] as date) as epoch_date from 
> dfs.tmp.`date_parquet.csv`;
> +-----------+----------------------------+
> | Fragment  | Number of records written  |
> +-----------+----------------------------+
> | 0_0       | 1                          |
> +-----------+----------------------------+
> {code}
> When I read the file with parquet tools, i found  
> {code}
> java -jar parquet-tools-1.8.1.jar head /tmp/buggy_parquet/
> name = Epoch
> epoch_date = 4881176
> {code}
> According to 
> [https://github.com/Parquet/parquet-format/blob/master/LogicalTypes.md#date], 
> epoch_date should be equals to 0.
> Meta : 
> {code}
> java -jar parquet-tools-1.8.1.jar meta /tmp/buggy_parquet/
> file:        file:/tmp/buggy_parquet/0_0_0.parquet 
> creator:     parquet-mr version 1.8.1-drill-r0 (build 
> 6b605a4ea05b66e1a6bf843353abcb4834a4ced8) 
> extra:       drill.version = 1.4.0 
> file schema: root 
> --------------------------------------------------------------------------------
> name:        OPTIONAL BINARY O:UTF8 R:0 D:1
> epoch_date:  OPTIONAL INT32 O:DATE R:0 D:1
> row group 1: RC:1 TS:93 OFFSET:4 
> --------------------------------------------------------------------------------
> name:         BINARY SNAPPY DO:0 FPO:4 SZ:52/50/0,96 VC:1 
> ENC:RLE,BIT_PACKED,PLAIN
> epoch_date:   INT32 SNAPPY DO:0 FPO:56 SZ:45/43/0,96 VC:1 
> ENC:RLE,BIT_PACKED,PLAIN
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)