igor-suhorukov commented on issue #14039: URL: https://github.com/apache/arrow/issues/14039#issuecomment-1238111546
Hello @lidavidm Thank you for your time and knowledge sharing! >For Map: it'd be reasonable to add support directory Sounds perfect! Something similar [with following](https://github.com/apache/arrow/blob/3e40cd3648a5b4f6ee7203a6c408f08c0abe4696/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java#L274) snippet for schema: ` if(arrowType.getTypeID() == ArrowType.ArrowTypeID.Map) { FieldType mapType = new FieldType(false, ArrowType.Struct.INSTANCE, null, null); FieldType keyType1 = new FieldType(false, new ArrowType.Utf8(), null, null); children = new ArrayList<>(); children.add(new Field("child", mapType, Arrays.asList(new Field(MapVector.KEY_NAME, keyType1, null), new Field(MapVector.VALUE_NAME, keyType1, null)))); }` >For geometry type: the question would be, what type do you expect this to be mapped to? binary representation is OK for me for current task >And would it be useful to extend the JDBC adapter with full control over the Arrow type support, so that you can add custom types at runtime? It could be good solution for custom type mapping in complex cases. First step is [add column index](https://github.com/apache/arrow/blob/3e40cd3648a5b4f6ee7203a6c408f08c0abe4696/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java#L260) in `final ArrowType arrowType = config.getJdbcToArrowTypeConverter().apply(columnIndex, columnFieldInfo); ` It is allow write custom type mapping by end users (Leveraging jdbc driver specific features and types. The same story with arrays elements types - much more easy to get element type by column index then iterate all columns to collect data for jdbcToArrowConfig.setArraySubTypeByColumnIndexMap() I also have proposal for column/schema level metadata - propagate "comment" for table/columns [here](https://github.com/apache/arrow/blob/3e40cd3648a5b4f6ee7203a6c408f08c0abe4696/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java#L249) by using java.sql.DatabaseMetaData#getColumns / java.sql.DatabaseMetaData#getTables from connection.getMetaData() or just allow user to provide it from custom comment handler in JdbcToArrowConfig . It will be very useful metadata in real life (medium to large scale project) for documentation and maintenance topics. Apache Spark code use "comment" key for such metadata, so this looks like reasonable default name for metadata in Arrow schema too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
