Fwd: Re: AVRO Union type support in Flink

[Timo Walther]

We have similar checks in our KafkaAvroTableSource, but I could not find 
such a check in AvroTypeInfo. The field should have a generic type, so 
you can work with it. If you want to use it as key, you might have to 
use a mapper before and convert it into a valid key type.

Timo


-------- Weitergeleitete Nachricht --------
Betreff:        Re: AVRO Union type support in Flink
Datum:  Wed, 19 Jul 2017 10:26:24 -0400
Von:    Vishnu Viswanath <vishnu.viswanat...@gmail.com>
An:     Timo Walther <twal...@apache.org>



Hi Timo,

Thanks for checking that. I did not try yet. My current application uses 
Cascading and it has the limitation that Union cannot contain two 
concrete types - link 
<https://github.com/ScaleUnlimited/cascading.avro/blob/master/scheme/src/main/java/cascading/avro/AvroToCascading.java#L137>, 
so was wondering if I can use Flink. Will give it a try.

Hi Martin,
The documentation is here 
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/connectors.html#avro-support-in-flink>
I use it to create AVRO files from source files in S3 and write to Kafka.

Thanks,
Vishnu


On Wed, Jul 19, 2017 at 5:55 AM, Timo Walther <twal...@apache.org 
<mailto:twal...@apache.org>> wrote:

   Hi Vishnu,

   I took a look into the code. Actually, we should support it.
   However, those types might be mapped to Java Objects that will be
   serialized with our generic Kryo serializer. Have you tested it?

   Regards,
   Timo


   Am 19.07.17 um 06:30 schrieb Martin Eden:
    Hey Vishnu,

    For those of us on the list that are not very familiar with Flink
    and Avro can you give a pointed to the docs you are referring to
    and how you intend to use it? Just so we gain understanding as well.

    Thanks,
    Martin

    On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath
    <vishnu.viswanat...@gmail.com
    <mailto:vishnu.viswanat...@gmail.com>> wrote:

        Hi All,

        Does Flink support AVRO union types - Documentation says it
        supports nullable types: {"name": "type_double_test", "type":
        ["null", "double"]}

        But my schema has something like : {"name": "union_field",
        "type": ["string", "double"]}

        Thanks
        Vishnu