Hi Eliot, I don’t see these logical types in the documentation. So I guess they are not predefined. (maybe Redshift implements them?) Indeed the official documentation should be a bit better, but I think there is already a Jira for that. Can you check with one of the documented logical types and see if that does work?
Regards, Pieter From: Elliot Crosby-McCullough [mailto:[email protected]] Sent: donderdag 30 maart 2017 13:18 To: [email protected] Subject: Re: Parsed JSON schemas not incorporating logical types Hi Pieter, Thanks for the `getProp` pointer, I've switched our code over. That ticket is not from me so that's not my schema, however I've reviewed our schemas and we were doing the same thing. Perhaps we should update the docs here to indicate that it needs to be nested within a `type` field: https://avro.apache.org/docs/1.8.1/spec.html#Logical+Types I can see that the spec says "Language implementations must ignore unknown logical types when reading, and should use the underlying Avro type" which explains your suggestion that `getLogicalType` should the logical type for built-in types, but it doesn't: Schema snippet: { "name": "date_field", "type": { "type": "long", "logicalType": "date" } }, { "name": "iso8601_date_field", "type": { "type": "string", "logicalType": "iso8601_date" } }, { "name": "iso8601_datetime_field", "type": { "type": "string", "logicalType": "iso8601_datetime" } } Output: From getProp: date From getLogicalType: null From getProp: iso8601_date From getLogicalType: null From getProp: iso8601_datetime From getLogicalType: null I would be happy to use the built-in logical types but it seems that while Redshift supports ingesting Avro files, it doesn't seem to know how to convert Avro dates and times into its own date and time fields. Maybe that was caused by the above misconfiguration, I'll double check and provide an update. On 30 March 2017 at 11:14, Dekinder, Pieter (Nokia - BE/Antwerp) <[email protected]<mailto:[email protected]>> wrote: Hi Elliot, I also came across this inconvenience. I’m using `field.schema.getProp(LogicalType.LOGICAL_TYPE_PROP)` for that. This one is not deprecated. Are you using predefined logical types or custom ones? I would expect the predefined ones to be retrievable with `field.schema().getLogicalType().getName()`. In the referenced post the JSON was also not really correctly formatted I think. I think It should have been the below… wrapping the logic type related things in yet another set of curly braces. { "namespace": "example.avro", "type": "record", "name": "User", "fields": [ { "name": "name", "type": "string" }, { "name": "account_balance", "type": { "type": "bytes", "logicalType": "decimal", "precision": 4, "scale": 2 } } ] } Regards, Pieter From: Elliot Crosby-McCullough [mailto:[email protected]<mailto:[email protected]>] Sent: woensdag 29 maart 2017 16:37 To: [email protected]<mailto:[email protected]> Subject: Parsed JSON schemas not incorporating logical types Hi, As noted in this thread from last year[1], Avro seems not to incorporate the logical types of fields when parsing JSON files. This means that instead of using `field.schema().getLogicalType().getName()` I have to use the deprecated `field.props().get("logicalType")` (where `field` is a `Schema.Field` retrieved via the record's `schema.getFields()` method). Any idea why this might be the case? I'm using Avro 1.8.1 for this. Regards, Elliot [1] http://apache-avro.679487.n3.nabble.com/Decimal-types-in-IDL-and-generated-Java-not-carrying-logical-types-to-schemas-td4034919.html
