Hi Pieter, The first field has a logical type of `date` which is in the documentation, but I just noticed that I had it marked as a `long`, I've switched it to an `int` and that works.
Regards, Elliot On 30 March 2017 at 13:13, Dekinder, Pieter (Nokia - BE/Antwerp) < [email protected]> wrote: > Hi Eliot, > > > > I don’t see these logical types in the documentation. So I guess they are > not predefined. (maybe Redshift implements them?) > > Indeed the official documentation should be a bit better, but I think > there is already a Jira for that. > > Can you check with one of the documented logical types and see if that > does work? > > > > Regards, > > Pieter > > > > *From:* Elliot Crosby-McCullough [mailto:elliot.crosby- > [email protected]] > *Sent:* donderdag 30 maart 2017 13:18 > *To:* [email protected] > *Subject:* Re: Parsed JSON schemas not incorporating logical types > > > > Hi Pieter, > > > > Thanks for the `getProp` pointer, I've switched our code over. > > > > That ticket is not from me so that's not my schema, however I've reviewed > our schemas and we were doing the same thing. Perhaps we should update the > docs here to indicate that it needs to be nested within a `type` field: > https://avro.apache.org/docs/1.8.1/spec.html#Logical+Types > > > > I can see that the spec says "Language implementations must ignore unknown > logical types when reading, and should use the underlying Avro type" which > explains your suggestion that `getLogicalType` should the logical type for > built-in types, but it doesn't: > > > > Schema snippet: > > { > "name": "date_field", > "type": { > "type": "long", > "logicalType": "date" > } > }, > { > "name": "iso8601_date_field", > "type": { > "type": "string", > "logicalType": "iso8601_date" > } > }, > { > "name": "iso8601_datetime_field", > "type": { > "type": "string", > "logicalType": "iso8601_datetime" > } > } > > Output: > > > > From getProp: date > > From getLogicalType: null > > > > From getProp: iso8601_date > > From getLogicalType: null > > > > From getProp: iso8601_datetime > > From getLogicalType: null > > > > I would be happy to use the built-in logical types but it seems that while > Redshift supports ingesting Avro files, it doesn't seem to know how to > convert Avro dates and times into its own date and time fields. > > > > Maybe that was caused by the above misconfiguration, I'll double check and > provide an update. > > > > On 30 March 2017 at 11:14, Dekinder, Pieter (Nokia - BE/Antwerp) < > [email protected]> wrote: > > Hi Elliot, > > > > I also came across this inconvenience. I’m using `field.schema.getProp( > LogicalType.LOGICAL_TYPE_PROP)` for that. > > This one is not deprecated. > > > > Are you using predefined logical types or custom ones? > > I would expect the predefined ones to be retrievable with `field.schema(). > getLogicalType().getName()`. > > > > In the referenced post the JSON was also not really correctly formatted I > think. > > I think It should have been the below… wrapping the logic type related > things in yet another set of curly braces. > > > > { > > "namespace": "example.avro", > > "type": "record", > > "name": "User", > > "fields": [ > > { > > "name": "name", "type": "string" > > }, > > { > > "name": "account_balance", > > * "type": * > > * {* > > "type": "bytes", > > "logicalType": "decimal", > > "precision": 4, > > "scale": 2 > > * }* > > } > > ] > > } > > > > Regards, > > Pieter > > > > *From:* Elliot Crosby-McCullough [mailto:elliot.crosby- > [email protected]] > *Sent:* woensdag 29 maart 2017 16:37 > *To:* [email protected] > *Subject:* Parsed JSON schemas not incorporating logical types > > > > Hi, > > > > As noted in this thread from last year[1], Avro seems not to incorporate > the logical types of fields when parsing JSON files. > > > > This means that instead of using `field.schema().getLogicalType().getName()` > I have to use the deprecated `field.props().get("logicalType")` (where > `field` is a `Schema.Field` retrieved via the record's `schema.getFields()` > method). > > > > Any idea why this might be the case? I'm using Avro 1.8.1 for this. > > > > Regards, > > Elliot > > > > [1] http://apache-avro.679487.n3.nabble.com/Decimal-types-in- > IDL-and-generated-Java-not-carrying-logical-types-to- > schemas-td4034919.html > > >
