morningman commented on code in PR #31765:
URL: https://github.com/apache/doris/pull/31765#discussion_r1519134731
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +81,128 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void init(List<Column> insertCols, List<Long> partitionIds) throws
AnalysisException {
+ THiveTableSink tSink = new THiveTableSink();
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTableName(targetTable.getName());
+ List<FieldSchema> partitionKeys =
targetTable.getRemoteTable().getPartitionKeys();
Review Comment:
```suggestion
List<FieldSchema> partitionKeys =
targetTable.getPartitionColumnNames()
```
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +81,128 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void init(List<Column> insertCols, List<Long> partitionIds) throws
AnalysisException {
+ THiveTableSink tSink = new THiveTableSink();
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTableName(targetTable.getName());
+ List<FieldSchema> partitionKeys =
targetTable.getRemoteTable().getPartitionKeys();
+
+ Set<String> nameToPartitions = new HashSet<>();
Review Comment:
```suggestion
Set<String> partitionNames = new HashSet<>();
```
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +81,128 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void init(List<Column> insertCols, List<Long> partitionIds) throws
AnalysisException {
+ THiveTableSink tSink = new THiveTableSink();
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTableName(targetTable.getName());
+ List<FieldSchema> partitionKeys =
targetTable.getRemoteTable().getPartitionKeys();
+
+ Set<String> nameToPartitions = new HashSet<>();
+ for (FieldSchema partitionKey : partitionKeys) {
+ nameToPartitions.add(partitionKey.getName());
+ }
+ List<FieldSchema> hmsColumns =
targetTable.getRemoteTable().getSd().getCols();
Review Comment:
Use schema in targetTable directly, no need to visit remote table
##########
gensrc/thrift/DataSinks.thrift:
##########
@@ -299,10 +300,25 @@ struct THiveBucket {
}
enum THiveCompressionType {
- SNAPPY = 3,
- LZ4 = 4,
- ZLIB = 6,
- ZSTD = 7,
+ NO_COMPRESSION = 0,
Review Comment:
Reuse the existing definition: TFileCompressType
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +81,128 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void init(List<Column> insertCols, List<Long> partitionIds) throws
AnalysisException {
+ THiveTableSink tSink = new THiveTableSink();
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTableName(targetTable.getName());
+ List<FieldSchema> partitionKeys =
targetTable.getRemoteTable().getPartitionKeys();
+
+ Set<String> nameToPartitions = new HashSet<>();
+ for (FieldSchema partitionKey : partitionKeys) {
+ nameToPartitions.add(partitionKey.getName());
+ }
+ List<FieldSchema> hmsColumns =
targetTable.getRemoteTable().getSd().getCols();
+ Set<String> nameToColumns = new HashSet<>();
Review Comment:
```suggestion
Set<String> colNames = new HashSet<>();
```
##########
gensrc/thrift/DataSinks.thrift:
##########
@@ -314,13 +330,12 @@ struct THivePartition {
struct THiveTableSink {
1: optional string db_name
2: optional string table_name
- 3: optional list<string> data_column_names
- 4: optional list<string> partition_column_names
- 5: optional list<THivePartition> partitions
- 6: optional list<THiveBucket> buckets
- 7: optional PlanNodes.TFileFormatType file_format
- 8: optional THiveCompressionType compression_type
- 9: optional THiveLocationParams location
+ 3: optional list<THiveColumn> columns
+ 4: optional list<THivePartition> partitions
+ 5: optional THiveBucket bucket_info
+ 6: optional PlanNodes.TFileFormatType file_format
+ 7: optional THiveCompressionType compression_type
+ 8: optional THiveLocationParams location
Review Comment:
Why here is a `THiveLocationParams`?
There is already a `THiveLocationParams` in `THivePartition`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
