ixtza opened a new pull request, #92: URL: https://github.com/apache/doris-kafka-connector/pull/92
The logic implementation is heavily inspired by Debezium JDBC Sink Connector as it uses `RecordDescriptor` to build the DDL table creation. `RecordDescriptor` class now have a `keyFieldNames` member of a `Set<>`, this helps DDL creation process to identify which column are used as a `UNIQUE KEY`. This implementation also provide new `createTable` method in the `RestService` class that calls the FE's `/api/query/` endpoint. **Feel free to raise some concern regarding the [implementation](https://github.com/ixtza/doris-kafka-connector/blob/00dd5280421033f8ff82b46d54a70a0d7fc7b420/src/main/java/org/apache/doris/kafka/connector/service/RestService.java#L334) if there's any issue**. Currently some adjustment are made on `Type` classes in order to prevent data loss and tackle some issue. - `String` mapped `key` fields are automatically mapped to `VARCHAR(65533)` ([ref.](https://github.com/ixtza/doris-kafka-connector/blob/00dd5280421033f8ff82b46d54a70a0d7fc7b420/src/main/java/org/apache/doris/kafka/connector/converter/schema/SchemaCreationManager.java#L113)). - `AbstractTimestampType` and `AbstractTimeType` precision now default to `6` to support `PostgreSQL`'s `Money` datatype. ([ref.AbstractTimestampType](https://github.com/ixtza/doris-kafka-connector/blob/00dd5280421033f8ff82b46d54a70a0d7fc7b420/src/main/java/org/apache/doris/kafka/connector/converter/type/AbstractTimestampType.java#L38) & [ref.AbstractTimeType](https://github.com/ixtza/doris-kafka-connector/blob/00dd5280421033f8ff82b46d54a70a0d7fc7b420/src/main/java/org/apache/doris/kafka/connector/converter/type/AbstractTimeType.java#L44)) - `io.debezium.time.MicroTimestamp` is mapped to `VARCHAR(255)` ([ref.](https://github.com/ixtza/doris-kafka-connector/blob/00dd5280421033f8ff82b46d54a70a0d7fc7b420/src/main/java/org/apache/doris/kafka/connector/converter/schema/SchemaCreationManager.java#L111)). There are some works need to be done, like: - [ ] Support specific table types besides `UNIQUE` such as `DUPLICATE` and `AGGREGATE`. - [ ] Utilize `DorisOptions` to further specify table's properties during DDL creation. This does mean that the table created is sub-optimal as it heavily depends on the struct's record definitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
