[
https://issues.apache.org/jira/browse/AVRO-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17134042#comment-17134042
]
Ryan Skraba commented on AVRO-2799:
-----------------------------------
OK! I can duplicate this error with your code. Thanks for the detail!
It looks like the supported and tested protobuf field types are described here:
https://avro.apache.org/docs/1.9.2/api/java/org/apache/avro/protobuf/package-summary.html
and {{map}} isn't in the list.
Experimentally, when there is a map, the code can correctly serialize map types
to binary, and it can deserialize the binary back into Avro GenericRecords, but
not back into the protobuf generated classes. Apparently, the record names
used internally for the map type from the protobuf descriptors (in your case
{{com.example.RevenueEntry.PayloadEvent}}) don't correspond to any generated
protobuf classes.
I'm not sure how difficult the fix for this would be, but it's unlikely to be
fixed for Avro 1.10.0 (which is already underway).
> [Java] protobuf map types are not supported
> -------------------------------------------
>
> Key: AVRO-2799
> URL: https://issues.apache.org/jira/browse/AVRO-2799
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.9.2
> Reporter: Allen Wang
> Priority: Major
>
> I have a protobuf message defined as this:
>
> {code:java}
> syntax = "proto3";
> package ExampleProtobuf;
> option java_package = "com.example";
> import "google/protobuf/any.proto";
> import "google/protobuf/timestamp.proto";
> import "google/protobuf/wrappers.proto";
> enum EventType {
> TYPE_UNKNOWN = 0;
> TYPE_ORDER_RECEIVED = 1;
> TYPE_PAYMENT_AUTHORIZED = 2;
> TYPE_PAYMENT_CAPTURED = 3;
> TYPE_ORDER_SUBMITTED = 4;
> TYPE_ORDER_DELIVERY_CREATED = 5;
> TYPE_MERCHANT_CONFIRM_ORDER = 6;
> }
> message RevenueEvent {
> google.protobuf.StringValue id = 1;
> EventType type = 2;
> google.protobuf.Timestamp change_time = 3;
> google.protobuf.StringValue country = 4;
> map <string, google.protobuf.Any> payload = 5;
> google.protobuf.StringValue source = 6;
> bool is_test = 7;
> google.protobuf.StringValue version = 8;
> google.protobuf.StringValue checksum = 9;
> }
> {code}
>
>
> I wrote a program to encode such Protobuf message with Avro schema and
> convert it back to Protobuf message:
>
>
> {code:java}
> public class EventTest {
> public static void main(String[] args) throws Exception {
> Example.RevenueEvent event = RevenueEvent.newBuilder()
> .setId(StringValue.newBuilder()
> .setValue("12345")
> .build())
> .setChangeTime(Timestamp.newBuilder()
> .setSeconds(System.currentTimeMillis())
> .build())
> .setChecksum(StringValue.newBuilder()
> .setValue("xyz")
> .build())
> .setCountry(StringValue.newBuilder()
> .setValue("US")
> .build())
> .setSource(StringValue.newBuilder()
> .setValue("unknown")
> .build())
> .setType(EventType.TYPE_MERCHANT_CONFIRM_ORDER)
> .putPayload("key",
> Any.newBuilder().setValue(ByteString.copyFromUtf8("xyz")).build())
> .build();
> ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
> Encoder encoder = EncoderFactory.get().directBinaryEncoder(outputStream,
> null);
> GenericDatumWriter<Example.RevenueEvent> datumWriter = new
> ProtobufDatumWriter<Example.RevenueEvent>(RevenueEvent.class);
> datumWriter.write(event, encoder);
> encoder.flush();
> byte[] bytes = outputStream.toByteArray();
> DatumReader<RevenueEvent> datumReader = new
> ProtobufDatumReader<>(RevenueEvent.class);
> InputStream inputStream = new ByteArrayInputStream(bytes);
> Decoder decoder = DecoderFactory.get().binaryDecoder(inputStream, null);
> datumReader.read(null, decoder);
> }
> }
> {code}
>
>
> The program crashed with java.lang.StackOverflowError:
>
> Exception in thread "main" java.lang.StackOverflowErrorException in thread
> "main" java.lang.StackOverflowError at
> java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at org.apache.avro.specific.SpecificData.getClass(SpecificData.java:250) at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:141) at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143) at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143) at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> ...
>
> Looking into ProtoData.java, there seems to be a situation where the
> recursion would not end:
>
> {code:java}
> @Override
> public Object newRecord(Object old, Schema schema) {
> try {
> Class c = SpecificData.get().getClass(schema);
> if (c == null)
> return newRecord(old, schema); // punt to generic <-- endless recursion
> if (c.isInstance(old))
> return old; // reuse instance
> return c.getMethod("newBuilder").invoke(null);
> } catch (Exception e) {
> throw new RuntimeException(e);
> }
> }
> {code}
> Further experiment shows that if the map payload is not set, the decoding
> would be successful.
>
> protobuf-java:3.8.0 is used.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
