Oh yes, you're right, you face with the limitation of field names <https://avro.apache.org/docs/1.8.0/spec.html#names>. Apart from solving this via a map, you might consider using Avro aliases <https://avro.apache.org/docs/1.8.2/spec.html#Aliases>, since looks like aliases don't have this limitation, can you use them?
Nandor On Wed, Oct 25, 2017 at 1:40 PM, Mike Thomsen <[email protected]> wrote: > Hi Nandor, > > It's not the numeric portion that is the problem for me, but the $date > field name. Mongo apparently requires the structure I provided in the > example, and whenever I use $date as the field name the Java Avro API > throws an exception about an invalid character in the field definition. > > The logical type thing is good to know for future reference. > > I admit that this is likely a really uncommon edge case for Avro. The work > around I found for defining a schema that is at least compatible with the > Mongo Extended JSON requirements was to do this (one field example): > > { > "namespace": "test", > "name": "PutTestRecord", > "type": "record", > "fields": [{ > "name": "timestampField", > "type": { > "type": "map", > "values": "long" > } > }] > } > > It doesn't give you the full validation that would be ideal if we could > define a field with the name "$date," but it's an 80% solution that works > with NiFi and other tools that have to generate Extended JSON for Mongo. > > Thanks, > > Mike > > On Wed, Oct 25, 2017 at 4:48 AM, Nandor Kollar <[email protected]> > wrote: > >> Hi Mike, >> >> This JSON doesn't seems like a valid Avro schema >> <https://avro.apache.org/docs/1.8.1/spec.html#schemas>. If you'd like to >> use timestamps in your schema, you should use Timestamp logical types, >> <https://avro.apache.org/docs/1.8.1/spec.html#Timestamp+%28millisecond+precision%29> >> which annotate Avro longs. In this case the schema of this field should >> look like this: >> >> { >> "name":"timestamp", >> "type":"long", >> "logicalType":"timestamp-millis" >> } >> >> If you'd like to create Avro files with this schema, there's on Avro wiki >> you can find a brief tutorial >> <https://avro.apache.org/docs/1.8.1/gettingstartedjava.html#Compiling+the+schema> >> how to create and write Avro files with this schema in Java. >> >> Regards, >> Nandor >> >> On Tue, Oct 24, 2017 at 8:18 PM, Mike Thomsen <[email protected]> >> wrote: >> >>> I am trying to build an avro schema for a NiFi flow that is going to >>> insert data into Mongo, and Mongo extended JSON requires the use of $ >>> characters in cases like this (to represent a date): >>> >>> { >>> "timestamp": { >>> "$date": TIMESTAMP_LONG_HERE >>> } >>> } >>> >>> I tried building a schema with that, and it failed saying there was an >>> invalid character in the schema. just wanted to check and see if there was >>> a work around for this or if I'll have to choose another option. >>> >>> Thanks, >>> >>> Mike >>> >> >> >
