[GitHub] [incubator-hudi] cdmikechen commented on issue #770: remove com.databricks:spark-avro to build spark avro schema by itself

[GitBox]

cdmikechen commented on issue #770: remove com.databricks:spark-avro to build 
spark avro schema by itself
URL: https://github.com/apache/incubator-hudi/pull/770#issuecomment-532470332
 
 
   @umehrot2 
   In addition to the decimal problem, I also modified a timestamp conversion 
problem. 
   On spark dataset, this PR get the right result. But there are still some 
problems on Hive and sparksql. Hive 2.3 does not correctly identify the 
logical-type in parquet-avro file, timestamp type may be cast to long type in 
Hive 2.3.
   I modified some of Hive's source in 
`org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector`
 code to solve this problem.
   ```
   package org.apache.hadoop.hive.serde2.objectinspector.primitive;
   
   import java.sql.Timestamp;
   
   import org.apache.hadoop.hive.serde2.io.TimestampWritable;
   import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
   import org.apache.hadoop.io.LongWritable;
   
   public class WritableTimestampObjectInspector extends
       AbstractPrimitiveWritableObjectInspector implements
       SettableTimestampObjectInspector {
   
     public WritableTimestampObjectInspector() {
       super(TypeInfoFactory.timestampTypeInfo);
     }
   
     @Override
     public TimestampWritable getPrimitiveWritableObject(Object o) {
       if (o instanceof LongWritable) {
         return (TimestampWritable) 
PrimitiveObjectInspectorFactory.writableTimestampObjectInspector
                 .create(new Timestamp(((LongWritable) o).get()));
       }
       return o == null ? null : (TimestampWritable) o;
     }
   
     public Timestamp getPrimitiveJavaObject(Object o) {
       if (o instanceof LongWritable) {
           return new Timestamp(((LongWritable) o).get());
       }
       return o == null ? null : ((TimestampWritable) o).getTimestamp();
     }
   
     public Object copyObject(Object o) {
       if (o instanceof LongWritable) {
           return new TimestampWritable(new Timestamp(((LongWritable) 
o).get()));
       }
       return o == null ? null : new TimestampWritable((TimestampWritable) o);
     }
   
     public Object set(Object o, byte[] bytes, int offset) {
       if (o instanceof LongWritable) {
         o = PrimitiveObjectInspectorFactory.writableTimestampObjectInspector
                 .create(new Timestamp(((LongWritable) o).get()));
       } else
       ((TimestampWritable) o).set(bytes, offset);
       return o;
     }
   
     public Object set(Object o, Timestamp t) {
       if (t == null) {
         return null;
       }
       if (o instanceof LongWritable) {
         o = 
PrimitiveObjectInspectorFactory.writableTimestampObjectInspector.create(t);
       } else
       ((TimestampWritable) o).set(t);
       return o;
     }
   
     public Object set(Object o, TimestampWritable t) {
       if (t == null) {
         return null;
       }
       if (o instanceof LongWritable) {
         o = PrimitiveObjectInspectorFactory.writableTimestampObjectInspector
                 .create(new Timestamp(((LongWritable) o).get()));
       } else
       ((TimestampWritable) o).set(t);
       return o;
     }
   
     public Object create(byte[] bytes, int offset) {
       return new TimestampWritable(bytes, offset);
     }
   
     public Object create(Timestamp t) {
       return new TimestampWritable(t);
     }
   }
   ```
   I'm looking for a solution that doesn't need to modify the hive source code. 
See if you can come up with any good ideas.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services