Hello Ryan:
Using "parquet-avro-1.8.2" (that I have built from the pull-request#
https://github.com/apache/parquet-mr/pull/318), I have tried creating a
"logical type for date" to write to parquet-file, using the code-block
below:
-----------------------------------------------------------------------
34 LogicalType AvroDate = new LogicalType("AvroDate") ;
35 Schema Pdate = AvroDate.addToSchema(Schema.create(Type.INT)) ;
36 fields.add(new Field("doj", Pdate, null, null));
-----------------------------------------------------------------------
The program is built successfully. But, when I try to run the program, I
got the below exeception:
----------------------------------------------------------------------------------------------------------------------
Exception in thread "main" java.lang.NoSuchMethodError:
org/apache/avro/Schema.setLogicalType(Lorg/apache/avro/LogicalType;)V
at org.apache.avro.LogicalType.addToSchema(LogicalType.java:72)
at pqtw.makeSchema(pqtw.java:35)
at pqtw.main(pqtw.java:63)
----------------------------------------------------------------------------------------------------------------------
I am using "parquet-avro-1.8.2.jar" & "avro-1.8.0.jar". The error is
indicating that the method: "org/apache/avro/Schema.setLogicalType" is NOT
found. From the trace, it looks like "addToSchema" function is in turn
calling "setLogicalType" in schema class, which is where it is failing
with "NoMethodFound" exception.
Hence, I am trying to understand, whether it is a correct way to create a
"LogicalType" (or) if there is any other approach (or if I should use a
"higher version" of "avro.jar", if any...) ?
Could you please let me know your inputs in this regard.. (or do you
suppose, this question should go to "AVRO-mailing-list" ?) Pl. let me know
your thoughts.
Thanks,
Ravi
NOTE:
Pl. find below the full-code of the test-program, if you wish to have a
look. FYI only.
======================================================================================
1 import java.io.IOException;
2 import java.util.*;
3
4 import org.apache.hadoop.conf.Configuration;
5 import org.apache.hadoop.fs.FileSystem;
6 import org.apache.hadoop.fs.Path;
7 import org.apache.hadoop.io.Text;
8
9 import org.apache.parquet.avro.*;
10
11 import org.apache.avro.Schema;
12 import org.apache.avro.Schema.Type;
13 import org.apache.avro.Schema.Field;
14 import org.apache.avro.LogicalType;
15 import org.apache.avro.LogicalTypes;
16
17 import org.apache.parquet.column.ParquetProperties.WriterVersion;
18
19 import org.apache.parquet.hadoop.api.WriteSupport;
20 import org.apache.parquet.hadoop.ParquetWriter;
21 import org.apache.parquet.hadoop.ParquetWriter.*;
22 import org.apache.parquet.hadoop.metadata.CompressionCodecName;
23
24 import org.apache.avro.generic.* ;
25
26 public class pqtw {
27
28 public static Schema makeSchema() {
29 List<Field> fields = new ArrayList<Field>();
30 fields.add(new Field("name", Schema.create(Type.STRING), null,
null));
31 fields.add(new Field("age", Schema.create(Type.INT), null,
null));
32 //fields.add(new Field("doj", Schema.create(Type.INT), null,
null));
33
34 LogicalType AvroDate = new LogicalType("AvroDate") ;
35 Schema Pdate = AvroDate.addToSchema(Schema.create(Type.INT)) ;
36 fields.add(new Field("doj", Pdate, null, null));
37
38 Schema schema = Schema.createRecord("filecc", null, "parquet",
false);
39 schema.setFields(fields);
40
41 return(schema);
42 }
43
44 public static GenericData.Record makeRecord (Schema schema, String
name, int age, int doj) {
45 GenericData.Record record = new GenericData.Record(schema);
46 record.put("name", name);
47 record.put("age", age);
48 record.put("doj", doj);
49 return(record);
50 }
51
52 public static void main(String[] args) throws IOException,
53
54 InterruptedException, ClassNotFoundException {
55
56 String pqfile = "/tmp/pqtfile2";
57
58 try {
59
60 Configuration conf = new Configuration();
61 FileSystem fs = FileSystem.getLocal(conf);
62
63 Schema schema = makeSchema() ;
64 GenericData.Record rec = makeRecord(schema,"abcd", 5,15000) ;
65 AvroParquetWriter writer = new AvroParquetWriter(new
Path(pqfile), schema) ;
66 writer.write(rec);
67 writer.close();
68 }
69 catch (Exception e)
70 {
71 e.printStackTrace();
72 }
73 }
74 }
======================================================================================
