Hi Manisha,
The main recommendation I have is to not use the
org.apache.parquet.example.* classes. Those are an example of how to
implement an object model, not classes that can or should be used in an
application that reads or writes Parquet data.
The best thing is to use one of the real object models, like Avro or
Thrift. That way you get the option of using row-oriented or
column-oriented storage in your application without translating between
object models.
rb
On 10/29/2015 01:46 AM, Manisha Sethi wrote:
Hi All,
I am trying to write a list in parquet using below code, but something is going
wrong..
MessageType schema = MessageTypeParser.parseMessageType("message basket { required
group myList (LIST) { repeated group list { required float listfloat;} } }");
ParquetWriter<Group> writer=new ParquetWriter<Group>(outDirPath,new
GroupWriteSupport() {
@Override
public WriteContext init(Configuration configuration) {
if (configuration.get(GroupWriteSupport.PARQUET_EXAMPLE_SCHEMA) == null) {
configuration.set(GroupWriteSupport.PARQUET_EXAMPLE_SCHEMA,
schema.toString());
}
return super.init(configuration);
}},CompressionCodecName.SNAPPY, 256*1024*1024, 100*1024);
GroupWriteSupport.setSchema(schema,config);
SimpleGroupFactory f=new SimpleGroupFactory(schema);
writer.write(f.newGroup().append("listfloat", ( float)2.8).append("listfloat",
3.3f));
Its not working....exception :
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.conf.Configuration.deprecation).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Exception in thread "main" parquet.io.InvalidRecordException: listfloat not
found in message basket {
required group myList (LIST) {
repeated group list {
required float listfloat;
}
}
}
at parquet.schema.GroupType.getFieldIndex(GroupType.java:147)
at parquet.example.data.Group.add(Group.java:39)
at parquet.example.data.Group.append(Group.java:107)
at ParquetTestWriter.main(ParquetTestWriter.java:90)
Appreciate the response!!.
Manisha
________________________________
--
Ryan Blue
Software Engineer
Cloudera, Inc.
