Adding on Ryan's comment.Sometime it seems confusing to understand how parquet
and object model works.Some relevant link :
On Thursday, October 29, 2015 10:02 AM, Ryan Blue <[email protected]>
wrote:
Hi Manisha,
The main recommendation I have is to not use the
org.apache.parquet.example.* classes. Those are an example of how to
implement an object model, not classes that can or should be used in an
application that reads or writes Parquet data.
The best thing is to use one of the real object models, like Avro or
Thrift. That way you get the option of using row-oriented or
column-oriented storage in your application without translating between
object models.
rb
On 10/29/2015 01:46 AM, Manisha Sethi wrote:
> Hi All,
>
> I am trying to write a list in parquet using below code, but something is
> going wrong..
>
> MessageType schema = MessageTypeParser.parseMessageType("message basket {
> required group myList (LIST) { repeated group list { required float
> listfloat;} } }");
> ParquetWriter<Group> writer=new ParquetWriter<Group>(outDirPath,new
> GroupWriteSupport() {
> @Override
> public WriteContext init(Configuration configuration) {
> if (configuration.get(GroupWriteSupport.PARQUET_EXAMPLE_SCHEMA) == null) {
> configuration.set(GroupWriteSupport.PARQUET_EXAMPLE_SCHEMA,
>schema.toString());
> }
> return super.init(configuration);
> }},CompressionCodecName.SNAPPY, 256*1024*1024, 100*1024);
> GroupWriteSupport.setSchema(schema,config);
> SimpleGroupFactory f=new SimpleGroupFactory(schema);
> writer.write(f.newGroup().append("listfloat", (
> float)2.8).append("listfloat", 3.3f));
>
>
> Its not working....exception :
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.conf.Configuration.deprecation).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main" parquet.io.InvalidRecordException: listfloat not
> found in message basket {
> required group myList (LIST) {
> repeated group list {
> required float listfloat;
> }
> }
> }
>
> at parquet.schema.GroupType.getFieldIndex(GroupType.java:147)
> at parquet.example.data.Group.add(Group.java:39)
> at parquet.example.data.Group.append(Group.java:107)
> at ParquetTestWriter.main(ParquetTestWriter.java:90)
>
>
>
> Appreciate the response!!.
>
> Manisha
>
> ________________________________
>
>
--
Ryan Blue
Software Engineer
Cloudera, Inc.
