BentsiLeviav opened a new issue, #37613:
URL: https://github.com/apache/beam/issues/37613
### What would you like to happen?
### Description
The current ClickHouseIO implementation manually fetches table schemas by
executing `DESCRIBE TABLE` queries and parsing the results. However, the
ClickHouse Java Client v2 already provides this functionality through
`Client.getTableSchema(String table, String database)`.
### Current Implementation
```java
public static TableSchema getTableSchema(
String clickHouseUrl, String database, String table, Properties
properties) {
// ... setup client
String query = "DESCRIBE TABLE " + quoteIdentifier(table);
try (Records records = client.queryRecords(query).get()) {
for (GenericRecord record : records) {
// Manually parse name, type, default_type, default_expression
// Build TableSchema.Column objects
}
}
// Return TableSchema
}
```
### Java Client v2 Provides This
The Java Client v2 already has:
```java
Client client = new Client.Builder()
.addEndpoint(clickHouseUrl)
.setUsername(user)
.setPassword(password)
.setDefaultDatabase(database)
.build();
// Built-in method returns com.clickhouse.client.api.metadata.TableSchema
TableSchema schema = client.getTableSchema(table, database);
```
### Proposed Solution
Leverage the client's built-in `getTableSchema()` method and convert the
result to Beam's `org.apache.beam.sdk.io.clickhouse.TableSchema`:
```java
public static TableSchema getTableSchema(
String clickHouseUrl, String database, String table, Properties
properties) {
try (Client client = createClient(clickHouseUrl, database, properties)) {
com.clickhouse.client.api.metadata.TableSchema chSchema =
client.getTableSchema(table, database);
return convertToBeamTableSchema(chSchema);
}
}
```
### Benefits
1. **Less code**: Remove manual DESCRIBE query execution and parsing logic
2. **Better compatibility**: Client handles version-specific differences
(e.g., ClickHouse 25.x backticks in tuple types)
3. **Reduced maintenance**: Schema parsing logic maintained by ClickHouse
team
4. **Consistency**: Use the same schema representation as other Java Client
v2 operations
### Considerations
- Need to map between `com.clickhouse.client.api.metadata.TableSchema` and
Beam's `org.apache.beam.sdk.io.clickhouse.TableSchema`
- The tuple preprocessing logic may no longer be needed if the client
handles it
### Issue Priority
Priority: 3 (nice-to-have improvement)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
