Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
danny0405 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1580753077
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##
@@ -518,7 +499,7 @@ private MergeOnReadInputFormat mergeOnReadInputFormat(
tableAvroSchema.toString(),
AvroSchemaConverter.convertToSchema(requiredRowType).toString(),
inputSplits,
-conf.getString(FlinkOptions.RECORD_KEY_FIELD).split(","));
+OptionsResolver.getRecordKeyField(conf));
Review Comment:
> Case problem. The columns created based on calsite in the upstream are all
lowercase. If there are uppercase in the downstream, such as "eventTime", the
columns will not be found.
->Uniformly converted to lowercase
This is not expected to be handled by Hudi, I think. At least, from the
catalog layer, we should make the case-sensitivity agnostic to specific engines.
> The table created by the upstream write (recorded in the existing
metadata) do not match the columns configured by the do
In `HoodieTableFactory#createDynamicTableSource`, add a sanity check for the
catalog table resolved schema and the existing Hudi table schema, that should
be enough I guess. Similiar with the primary key definition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1580704716
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##
@@ -518,7 +499,7 @@ private MergeOnReadInputFormat mergeOnReadInputFormat(
tableAvroSchema.toString(),
AvroSchemaConverter.convertToSchema(requiredRowType).toString(),
inputSplits,
-conf.getString(FlinkOptions.RECORD_KEY_FIELD).split(","));
+OptionsResolver.getRecordKeyField(conf));
Review Comment:
Or I split this PR？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1580704716
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##
@@ -518,7 +499,7 @@ private MergeOnReadInputFormat mergeOnReadInputFormat(
tableAvroSchema.toString(),
AvroSchemaConverter.convertToSchema(requiredRowType).toString(),
inputSplits,
-conf.getString(FlinkOptions.RECORD_KEY_FIELD).split(","));
+OptionsResolver.getRecordKeyField(conf));
Review Comment:
Or I split this PR？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1580701002
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##
@@ -518,7 +499,7 @@ private MergeOnReadInputFormat mergeOnReadInputFormat(
tableAvroSchema.toString(),
AvroSchemaConverter.convertToSchema(requiredRowType).toString(),
inputSplits,
-conf.getString(FlinkOptions.RECORD_KEY_FIELD).split(","));
+OptionsResolver.getRecordKeyField(conf));
Review Comment:
1. The table created by the upstream write (recorded in the existing
metadata) do not match the columns configured by the downstream stream read.
For example, some columns do not exist, resulting in the columns not be found.
-> Verification failed, throwing exception
2. The recordkey configuration does not exist
-> Verification failed, throwing exception
3. Case problem. The columns created based on calsite in the upstream are
all lowercase. If there are uppercase in the downstream, such as "eventTime",
the columns will not be found.
->Uniformly converted to lowercase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
danny0405 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1580678775
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##
@@ -518,7 +499,7 @@ private MergeOnReadInputFormat mergeOnReadInputFormat(
tableAvroSchema.toString(),
AvroSchemaConverter.convertToSchema(requiredRowType).toString(),
inputSplits,
-conf.getString(FlinkOptions.RECORD_KEY_FIELD).split(","));
+OptionsResolver.getRecordKeyField(conf));
Review Comment:
The change is way too complicated, can you re-illustrate the issue again?
What is the use case from user and what the correct behavior is expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2078829966 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * 20af652a75eb2017463f548321a18c83f14121e0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23511) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2078818694 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * e667c4be8a4327e5725c5dee59ce3b1c5a1c47cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23510) * 20af652a75eb2017463f548321a18c83f14121e0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2078807695 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * e667c4be8a4327e5725c5dee59ce3b1c5a1c47cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23510) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1580565594
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##
@@ -86,12 +84,14 @@ public DynamicTableSource createDynamicTableSource(Context
context) {
setupTableOptions(conf.getString(FlinkOptions.PATH), conf);
ResolvedSchema schema = context.getCatalogTable().getResolvedSchema();
setupConfOptions(conf, context.getObjectIdentifier(),
context.getCatalogTable(), schema);
-return new HoodieTableSource(
+HoodieTableSource source = new HoodieTableSource(
schema,
path,
context.getCatalogTable().getPartitionKeys(),
conf.getString(FlinkOptions.PARTITION_DEFAULT_NAME),
Review Comment:
@danny0405 cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1580556201
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##
@@ -86,12 +84,14 @@ public DynamicTableSource createDynamicTableSource(Context
context) {
setupTableOptions(conf.getString(FlinkOptions.PATH), conf);
ResolvedSchema schema = context.getCatalogTable().getResolvedSchema();
setupConfOptions(conf, context.getObjectIdentifier(),
context.getCatalogTable(), schema);
-return new HoodieTableSource(
+HoodieTableSource source = new HoodieTableSource(
schema,
path,
context.getCatalogTable().getPartitionKeys(),
conf.getString(FlinkOptions.PARTITION_DEFAULT_NAME),
Review Comment:
@danny0405 cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2078741091 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * d591f2e04ab802687fa15b46c84071c7c49a8bd1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23509) * e667c4be8a4327e5725c5dee59ce3b1c5a1c47cb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1580556201
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##
@@ -86,12 +84,14 @@ public DynamicTableSource createDynamicTableSource(Context
context) {
setupTableOptions(conf.getString(FlinkOptions.PATH), conf);
ResolvedSchema schema = context.getCatalogTable().getResolvedSchema();
setupConfOptions(conf, context.getObjectIdentifier(),
context.getCatalogTable(), schema);
-return new HoodieTableSource(
+HoodieTableSource source = new HoodieTableSource(
schema,
path,
context.getCatalogTable().getPartitionKeys(),
conf.getString(FlinkOptions.PARTITION_DEFAULT_NAME),
Review Comment:
@danny0405 cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2078731973 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * 0b3c631226ab99ed3ecb20c3a870529d3e8aa0ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23399) * d591f2e04ab802687fa15b46c84071c7c49a8bd1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
danny0405 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1578662024
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##
@@ -86,12 +84,14 @@ public DynamicTableSource createDynamicTableSource(Context
context) {
setupTableOptions(conf.getString(FlinkOptions.PATH), conf);
ResolvedSchema schema = context.getCatalogTable().getResolvedSchema();
setupConfOptions(conf, context.getObjectIdentifier(),
context.getCatalogTable(), schema);
-return new HoodieTableSource(
+HoodieTableSource source = new HoodieTableSource(
schema,
path,
context.getCatalogTable().getPartitionKeys(),
conf.getString(FlinkOptions.PARTITION_DEFAULT_NAME),
Review Comment:
> Metadataclient needs to be initialized in the factory
That's okay, we already do that for the sink sanity check of table config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1575580400
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##
@@ -86,12 +84,14 @@ public DynamicTableSource createDynamicTableSource(Context
context) {
setupTableOptions(conf.getString(FlinkOptions.PATH), conf);
ResolvedSchema schema = context.getCatalogTable().getResolvedSchema();
setupConfOptions(conf, context.getObjectIdentifier(),
context.getCatalogTable(), schema);
-return new HoodieTableSource(
+HoodieTableSource source = new HoodieTableSource(
schema,
path,
context.getCatalogTable().getPartitionKeys(),
conf.getString(FlinkOptions.PARTITION_DEFAULT_NAME),
Review Comment:
If according to what you said, Metadataclient needs to be initialized in
the factory(Hoodie source sanity check need). It seems more reasonable to
initialize in source?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
danny0405 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1575532144
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##
@@ -86,12 +84,14 @@ public DynamicTableSource createDynamicTableSource(Context
context) {
setupTableOptions(conf.getString(FlinkOptions.PATH), conf);
ResolvedSchema schema = context.getCatalogTable().getResolvedSchema();
setupConfOptions(conf, context.getObjectIdentifier(),
context.getCatalogTable(), schema);
-return new HoodieTableSource(
+HoodieTableSource source = new HoodieTableSource(
schema,
path,
context.getCatalogTable().getPartitionKeys(),
conf.getString(FlinkOptions.PARTITION_DEFAULT_NAME),
Review Comment:
Let's keep the sanity check in the `HoodieTableFactory`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2070158131 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * 0b3c631226ab99ed3ecb20c3a870529d3e8aa0ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23399) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2070129843 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * a487acf5ea90bbb7733eda56e65d7dbda7d95f52 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23397) * 0b3c631226ab99ed3ecb20c3a870529d3e8aa0ce UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2069692649 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * a487acf5ea90bbb7733eda56e65d7dbda7d95f52 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23397) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2069662022 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * a7270a970988e18ba0997ee59b74678c2dc720ed Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23336) * a487acf5ea90bbb7733eda56e65d7dbda7d95f52 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2062919442 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * a7270a970988e18ba0997ee59b74678c2dc720ed Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23336) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2062867237 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * 5ffea9f5122b0302cc254c88edb4fbe5feb05d39 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23321) * a7270a970988e18ba0997ee59b74678c2dc720ed Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23336) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2062861117 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * 5ffea9f5122b0302cc254c88edb4fbe5feb05d39 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23321) * a7270a970988e18ba0997ee59b74678c2dc720ed UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2062848895 @danny0405 cc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2061795890 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * 5ffea9f5122b0302cc254c88edb4fbe5feb05d39 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23321) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2061682075 ## CI report: * e5c2f3c7cdb84ef5219cc02ba3dc11dd39de581c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23307) * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * 5ffea9f5122b0302cc254c88edb4fbe5feb05d39 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23321) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2061658606 ## CI report: * e5c2f3c7cdb84ef5219cc02ba3dc11dd39de581c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23307) * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * 5ffea9f5122b0302cc254c88edb4fbe5feb05d39 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2061641672 ## CI report: * e5c2f3c7cdb84ef5219cc02ba3dc11dd39de581c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23307) * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
danny0405 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1568440953
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadTableState.java:
##
@@ -83,11 +96,20 @@ public int getOperationPos() {
}
public int[] getRequiredPositions() {
-final List fieldNames = rowType.getFieldNames();
-return requiredRowType.getFieldNames().stream()
-.map(fieldNames::indexOf)
-.mapToInt(i -> i)
-.toArray();
+List fieldNames = rowType.getFieldNames();
+List expColumns = new ArrayList<>();
+int[] pos = requiredRowType.getFieldNames().stream()
+.mapToInt(fn -> {
+ if (!fieldNames.contains(fn.toLowerCase())) {
+expColumns.add(fn);
+ }
+ return fieldNames.indexOf(fn.toLowerCase());
+})
+.toArray();
+if (!expColumns.isEmpty()) {
+ throw new HoodieException("Column(s) " + String.join(", ", expColumns) +
" does not exists in the hudi table " + this.tableName + ".");
+}
Review Comment:
Should we move the check into `HoodieTableFactory#createDynamicTableSource` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
danny0405 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1568440953
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadTableState.java:
##
@@ -83,11 +96,20 @@ public int getOperationPos() {
}
public int[] getRequiredPositions() {
-final List fieldNames = rowType.getFieldNames();
-return requiredRowType.getFieldNames().stream()
-.map(fieldNames::indexOf)
-.mapToInt(i -> i)
-.toArray();
+List fieldNames = rowType.getFieldNames();
+List expColumns = new ArrayList<>();
+int[] pos = requiredRowType.getFieldNames().stream()
+.mapToInt(fn -> {
+ if (!fieldNames.contains(fn.toLowerCase())) {
+expColumns.add(fn);
+ }
+ return fieldNames.indexOf(fn.toLowerCase());
+})
+.toArray();
+if (!expColumns.isEmpty()) {
+ throw new HoodieException("Column(s) " + String.join(", ", expColumns) +
" does not exists in the hudi table " + this.tableName + ".");
+}
Review Comment:
Should we move the check into `HoodieTableFactory#createDynamicTableSource` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2060539425 ## CI report: * e5c2f3c7cdb84ef5219cc02ba3dc11dd39de581c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23307) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1568296222
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadTableState.java:
##
@@ -83,11 +96,20 @@ public int getOperationPos() {
}
public int[] getRequiredPositions() {
-final List fieldNames = rowType.getFieldNames();
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2060453357 ## CI report: * 4e167a80603c4bb8c6bfcd5e69bf4d7f1065b36f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23302) * e5c2f3c7cdb84ef5219cc02ba3dc11dd39de581c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23307) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2060444207 ## CI report: * 4e167a80603c4bb8c6bfcd5e69bf4d7f1065b36f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23302) * e5c2f3c7cdb84ef5219cc02ba3dc11dd39de581c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2060435950 ## CI report: * 4e167a80603c4bb8c6bfcd5e69bf4d7f1065b36f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23302) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2060310726 ## CI report: * 821e081933a557b4e064677f03f807711c3ffdd5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23284) * 4e167a80603c4bb8c6bfcd5e69bf4d7f1065b36f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23302) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2060304865 ## CI report: * 821e081933a557b4e064677f03f807711c3ffdd5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23284) * 4e167a80603c4bb8c6bfcd5e69bf4d7f1065b36f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
danny0405 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1568173216
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadTableState.java:
##
@@ -83,11 +96,20 @@ public int getOperationPos() {
}
public int[] getRequiredPositions() {
-final List fieldNames = rowType.getFieldNames();
Review Comment:
Just fetch it with `OptionsResolver` and param `conf`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1568167071
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadTableState.java:
##
@@ -83,11 +96,20 @@ public int getOperationPos() {
}
public int[] getRequiredPositions() {
-final List fieldNames = rowType.getFieldNames();
Review Comment:
The method getRequiredPositions is called by objects CdcInputFormat and
MergeOnReadInputFormat，neither of these two objects stores table names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
zhuanshenbsj1 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1568164751
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadTableState.java:
##
@@ -83,11 +96,20 @@ public int getOperationPos() {
}
public int[] getRequiredPositions() {
-final List fieldNames = rowType.getFieldNames();
-return requiredRowType.getFieldNames().stream()
-.map(fieldNames::indexOf)
-.mapToInt(i -> i)
-.toArray();
+List fieldNames = rowType.getFieldNames();
+List expColumns = new ArrayList<>();
+int[] pos = requiredRowType.getFieldNames().stream()
+.mapToInt(fn -> {
+ if (!fieldNames.contains(fn.toLowerCase())) {
+expColumns.add(fn);
+ }
+ return fieldNames.indexOf(fn.toLowerCase());
+})
+.toArray();
+if (!expColumns.isEmpty()) {
+ throw new HoodieException("Column(s) " + String.join(", ", expColumns) +
" does not exists in the hudi table " + this.tableName + ".");
+}
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
danny0405 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1568078282
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadTableState.java:
##
@@ -83,11 +96,20 @@ public int getOperationPos() {
}
public int[] getRequiredPositions() {
-final List fieldNames = rowType.getFieldNames();
Review Comment:
Can we add the table name param for this method `getRequiredPositions`
instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
danny0405 commented on code in PR #11031:
URL: https://github.com/apache/hudi/pull/11031#discussion_r1568078034
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadTableState.java:
##
@@ -83,11 +96,20 @@ public int getOperationPos() {
}
public int[] getRequiredPositions() {
-final List fieldNames = rowType.getFieldNames();
-return requiredRowType.getFieldNames().stream()
-.map(fieldNames::indexOf)
-.mapToInt(i -> i)
-.toArray();
+List fieldNames = rowType.getFieldNames();
+List expColumns = new ArrayList<>();
+int[] pos = requiredRowType.getFieldNames().stream()
+.mapToInt(fn -> {
+ if (!fieldNames.contains(fn.toLowerCase())) {
+expColumns.add(fn);
+ }
+ return fieldNames.indexOf(fn.toLowerCase());
+})
+.toArray();
+if (!expColumns.isEmpty()) {
+ throw new HoodieException("Column(s) " + String.join(", ", expColumns) +
" does not exists in the hudi table " + this.tableName + ".");
+}
Review Comment:
Column(s) [`$col_a`, `$col_b`, `$col_c` ...] does not exist in the table
`$tableName`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2059532263 ## CI report: * 821e081933a557b4e064677f03f807711c3ffdd5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23284) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2059328484 ## CI report: * 821e081933a557b4e064677f03f807711c3ffdd5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23284) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Optimization function MergeOnReadTableState#getRequiredPositions [hudi]
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2059311723 ## CI report: * 821e081933a557b4e064677f03f807711c3ffdd5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
