riferrei opened a new issue #6669: Using Map's with Avro Generated Types Generates Weird Errors URL: https://github.com/apache/pulsar/issues/6669 **Description of the bug** I have a project that uses Avro generated types (POJO's generated by the Avro compiler) as the type to be serialized with the producer. Everything works fine excepts if one of the fields is a map. When the field is a map and a value is set into this map, the following error is generated: ``` org.apache.pulsar.shade.org.apache.avro.UnresolvedUnionException: Not in union ["null",{"type":"array","items":{"type":"record","name":"PairCharSequenceDouble", "namespace":"org.apache.pulsar.shade.org.apache.avro.reflect", "fields":[{"name":"key","type":"string"},{"name":"value","type":"double"}]}, "java-class":"java.util.Map"}] ``` Anecdotally, if the field containing a map is not set, the message is properly serialized, proving that the problem is indeed the map field. Also, the same type if implemented as a POJO class by hand (without using a Avro compiler) works fine. Which means that something happens during the message serialization when the type being serialized contains the code generated by the compiler. **To Reproduce** Steps to reproduce the behavior: 1. Define an new type like the one below: ``` { "namespace": "com.acme.something", "type": "record", "name": "AvroGenComplexType", "fields": [ { "name": "stringField", "type": "string" }, { "name": "booleanField", "type": "boolean" }, { "name": "bytesField", "type": "bytes" }, { "name": "intField", "type": "int" }, { "name": "longField", "type": "long" }, { "name": "floatField", "type": "float" }, { "name": "doubleField", "type": "double" }, { "name": "mapField", "type": [ "null", { "type": "map", "values": "double" } ], "default": null }, { "name": "innerField", "type": [ "null", { "type": "record", "name": "AvroGenInnerType", "fields": [ { "name": "doubleField", "type": "double" }, { "name": "arrayField", "type": { "type": "array", "items": "string" } }, { "name": "enumField", "type": { "type": "enum", "name": "AvroGenMultipleOptions", "symbols": [ "FirstOption", "SecondOption", "ThirdOption", "FourthOption" ], "default": "FirstOption" } } ] } ], "default": null } ] } ``` 2. Using a Avro compiler, generate the types for Java. 3. Write a Pulsar producer with the following code: ``` protected AvroGenComplexType createAvroGenComplexType() { AvroGenComplexType.Builder complexType = AvroGenComplexType.newBuilder(); complexType.setStringField(String.valueOf(RANDOM.nextInt())); complexType.setBooleanField(RANDOM.nextBoolean()); byte[] bytes = new byte[1024]; RANDOM.nextBytes(bytes); ByteBuffer byteBuffer = ByteBuffer.wrap(bytes); complexType.setBytesField(byteBuffer); complexType.setIntField(RANDOM.nextInt()); complexType.setLongField(RANDOM.nextLong()); complexType.setFloatField(RANDOM.nextFloat()); complexType.setDoubleField(RANDOM.nextDouble()); // ******** This part will generate the error ******** Map<CharSequence, Double> mapField = new HashMap<>(1); mapField.put(String.valueOf(RANDOM.nextInt()), RANDOM.nextDouble()); complexType.setMapField(mapField); // ***************************************************** complexType.setInnerFieldBuilder(AvroGenInnerType.newBuilder() .setDoubleField(RANDOM.nextDouble()) .setArrayField(Arrays.asList(String.valueOf(RANDOM.nextInt()))) .setEnumField(AvroGenMultipleOptions.ThirdOption)); return complexType.build(); } public void produceAvroGenBasedMessages(String topic, int numMessages) throws PulsarClientException { Producer<AvroGenComplexType> producer = null; try { producer = pulsarClient.newProducer( Schema.AVRO(AvroGenComplexType.class)) .topic(topic) .create(); for (int i = 0; i < numMessages; i++) { producer.send(createAvroGenComplexType()); } } finally { if (producer != null) { producer.closeAsync(); } } } ``` 4. Execute the code and observe the error being thrown. **Expected behavior** The code should execute completely without complaining about any errors related to the type being serialized. This is true specifically because the type being used was generated by the Avro compiler, which double-checks any compliance with the spec. **Screenshots** N/A **Desktop (please complete the following information):** - OS: Tested on Linux (Fedora 31)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
