Fachang Yang created PARQUET-2446:
-------------------------------------
Summary: ProtoParquetWriter Not Support DynamicMessage
Key: PARQUET-2446
URL: https://issues.apache.org/jira/browse/PARQUET-2446
Project: Parquet
Issue Type: Improvement
Components: parquet-protobuf
Affects Versions: 1.13.1
Environment: Parquet-Protobuf 1.13.1
Reporter: Fachang Yang
when i write DynamicMessages to parquet by protoParquetWriter,
{code:java}
pbwriter = ProtoParquetWriter.<DynamicMessage>builder(s3ParquetOutputFile)
.withMessage(DynamicMessage.class)
.withConf(hadoopConf)
.withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
.withDictionaryEncoding(true)
.withCompressionCodec(storage.conf().parquetCompressionCodecName())
.withPageSize(PAGE_SIZE)
.build(); {code}
I got an exception like this:
{code:java}
[ERROR] - com.twitter.elephantbird.util.Protobufs -Protobufs.java(126) -Could
not find method getDescriptor in class
com.google.protobuf.DynamicMessagecom.google.protobuf.DynamicMessage
java.lang.NoSuchMethodException:
com.google.protobuf.DynamicMessage.getDescriptor()
at java.lang.Class.getMethod(Class.java:1786)
at
com.twitter.elephantbird.util.Protobufs.getMessageDescriptor(Protobufs.java:123)
at
org.apache.parquet.proto.ProtoWriteSupport.init(ProtoWriteSupport.java:136)
at
org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:279)
at
org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:702)
{code}
I believe that ProtoParquetWriter should support DynamicMessage and be capable
of creating ProtoWriteSupport using descriptors same with PBMessage class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
