Thanks Sean. But the real issue is on the surface, my UDF should have
no knowledge of the Avro schema at all! Here are the high-level steps
of what happened:
1. read the JSON to create a Avro Schema object -- OK
2. Convert the Avro schema to a StructType using the Spark's
SchemaConverter
Your function is somehow capturing the actual Avro schema object, which
won't seiralize. Try rewriting it to ensure that it isn't used in the
function.
On Tue, Feb 2, 2021 at 2:32 PM Artemis User wrote:
> We tried to standardize the SQL data source management using the Avro
> schema, but encount
We tried to standardize the SQL data source management using the Avro
schema, but encountered some serialization exceptions when trying to use
the data. The interesting part is that we didn't have any problems in
reading the Avro schema JSON file and converting the Avro schema into a
SQL Struc
