RyanSkraba commented on a change in pull request #14858:
URL: https://github.com/apache/beam/pull/14858#discussion_r644606333
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
##########
@@ -906,6 +906,23 @@ private void readObject(ObjectInputStream in) throws
IOException, ClassNotFoundE
.map(x -> getFieldSchema(x.getType(), x.getName(),
namespace))
.collect(Collectors.toList()));
break;
+
+ case "NVARCHAR":
+ case "VARCHAR":
+ case "LONGNVARCHAR":
+ case "LONGVARCHAR":
+ baseType = org.apache.avro.Schema.create(Type.STRING);
Review comment:
Super interesting conversation from 2017! This could go either way --
the `"char"` and `"varchar"` logical types that Hive adds are not part of the
Avro specification and should be ignored by implementations that don't
understand them. A developer could implement their own "custom" logical types
that understand them and do any necessary truncation, but AFAIK, nobody does.
I have a small preference for aligning with the de facto behaviour of Spark
and Hive, since this information might be useful as data is sent downstream,
and it's easy to ignore. But I'd be OK either way, it could be added in a
later PR as a new feature.
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
##########
@@ -1277,4 +1308,17 @@ private static void checkTypeName(Schema.TypeName got,
Schema.TypeName expected,
checkArgument(
got.equals(expected), "Can't convert '%s' to %s, expected: %s", label,
got, expected);
}
+
+ /** Helper factory to build JDBC Logical types for AVRO Schema. */
+ private static org.apache.avro.Schema makeJdbcLogicalStringAvroType(
+ Schema.LogicalType<?, ?> logicalType) {
+ JDBCType jdbcType = JDBCType.valueOf(logicalType.getIdentifier());
+ Integer size = logicalType.getArgument();
+
+ String schemaJson =
Review comment:
Hey, optionally you can create Avro schemas in a more readable way:
```
return org.apache.avro.SchemaBuilder.builder()
.stringBuilder().prop("logicalType", jdbcType.name()).prop("maxLength",
size).endString();
```
This is for info -- they should (must!) be equivalent to using the Parser.
This snippet is small but builders are a good practice for larger schemas!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
