This is an automated email from the ASF dual-hosted git repository.
xushiyan pushed a change to branch release-feature-rfc46
in repository https://gitbox.apache.org/repos/asf/hudi.git
omit 6e9342d1ec [HUDI-3378][HUDI-3379][HUDI-3381] Migrate usage of
HoodieRecordPayload and raw Avro payload to HoodieRecord (#5522)
add 0d069b5e57 [HUDI-4174] Add hive conf dir option for flink sink (#5725)
add dfcd6d9a86 [HUDI-4011] Add hudi-aws-bundle (#5674)
add 7276d0eaa6 [HUDI-3670] free temp views in sql transformers (#5080)
add 7f8630cc57 [HUDI-4167] Remove the timeline refresh with initializing
hoodie table (#5716)
add 51602a34f7 [HUDI-4179] Cluster with sort cloumns invalid (#5739)
add 3759a38b99 [HUDI-4183] Fix using HoodieCatalog to create non-hudi
tables (#5743)
add 80783c27f5 [HUDI-4187] Fix partition order in aws glue sync (#5731)
add bd26d633d7 [HUDI-4168] Add Call Procedure for marker deletion (#5738)
add 5d18b80343 [HUDI-4190] Include hbase-protocol for shading in the
bundles (#5750)
add 73b0be3c96 [HUDI-4192] HoodieHFileReader scan top cells after bottom
cells throw NullPointerException (#5755)
add 22c45a7704 [HUDI-4188] Fix flaky
ITTestDataSTreamWrite.testWriteCopyOnWrite (#5749)
add 21ab0ff8be [HUDI-4195] Bulk insert should use right keygen for
non-partitioned table (#5759)
add 132c0aa8c7 [HUDI-4101] When BucketIndexPartitioner take partition path
for dispersion may cause the fileID of the task to not be loaded correctly
(#5763)
add 4f7ea8c79a [HUDI-4176] Fixing `TableSchemaResolver` to avoid repeated
`HoodieCommitMetadata` parsing (#5733)
add 4f6fc726d0 [HUDI-4140] Fixing hive style partitioning and default
partition with bulk insert row writer with SimpleKeyGen and virtual keys (#5664)
add 21b903fddb [HUDI-4197] Fix Async indexer to support building FILES
partition (#5766)
add 7da97c8096 [HUDI-4171] Fixing Non partitioned with virtual keys in
read path (#5747)
add e5710a8e7c [MINOR] Mark AWSGlueCatalogSyncClient experimental (#5775)
add 4f5cad8029 [MINOR][RFC-53] Fix typos (#5764)
add f85cd9b16d [HUDI-4200] Fixing sorting of keys fetched from metadata
table (#5773)
add 1349b596a1 [HUDI-4198] Fix hive config for AWSGlueClientFactory (#5768)
add 35afdb4316 [HUDI-4178] Addressing performance regressions in Spark
DataSourceV2 Integration (#5737)
add f5ab921300 [MINOR][DOCS] Update the README.md file in hudi-examples
(#5803)
add 8ff17b0470 [MINOR] FlinkStateBackendConverter add more exception
message (#5809)
add c608dbd6c2 [HUDI-4213] Infer keygen clazz for Spark SQL (#5815)
add ba47904fa2 [HUDI-4139]improvement for flink write operator name to
identify tables easily (#5744)
add 2b3a85528a [HUDI-3889] Do not validate table config if save mode is
set to Overwrite (#5619)
add 08fe281091 [HUDI-4221] Fixing getAllPartitionPaths perf hit w/
FileSystemBackedMetadata (#5829)
add 97ccf5dd18 [HUDI-4223] Fix NullPointerException from
getLogRecordScanner when reading metadata table (#5840)
add fd8f7c5f6c [HUDI-4205] Fix NullPointerException in HFile reader
creation (#5841)
add 5aaac21d1d [HUDI-4224] Fix CI issues (#5842)
add c82e3462e3 [MINOR] fix AvroSchemaConverter duplicate branch in
'switch' (#5813)
add 14d8735a1c Strip extra spaces when creating new configuration (#5849)
add e89f5627e4 [HUDI-3682] testReaderFilterRowKeys fails in
TestHoodieOrcReaderWriter (#5790)
add 0d859fe58b [HUDI-3863] Add UT for drop partition column in
deltastreamer testsuite (#5727)
add 4774c4248f [HUDI-4006] failOnDataLoss on delta-streamer kafka sources
(#5718)
add 264b15df87 [HUDI-4207] HoodieFlinkWriteClient.getOrCreateWriteHandle
throws an e… (#5788)
add f16b1e8982 [MINOR] Fix typo of DisruptorExecutor in RFC 53 (#5860)
add 25bbff64cf [minor] Following HUDI-4207, remote the new wrapper #init
method (#5865)
new dd3d003303 [HUDI-3378][HUDI-3379][HUDI-3381] Migrate usage of
HoodieRecordPayload and raw Avro payload to HoodieRecord (#5522)
new 65caefe4b4 [MINOR] Fix type casting in TestHoodieHFileReaderWriter
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6e9342d1ec)
\
N -- N -- N refs/heads/release-feature-rfc46 (65caefe4b4)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
azure-pipelines.yml | 116 +++--
hudi-aws/pom.xml | 30 ++
.../hudi/aws/sync/AWSGlueCatalogSyncClient.java | 13 +-
hudi-cli/pom.xml | 66 ++-
.../org/apache/hudi/cli/commands/SparkMain.java | 4 +-
.../SparkUtilTest.java => TestSparkUtil.java} | 4 +-
.../hudi/cli/integ/ITTestBootstrapCommand.java | 4 +-
.../hudi/cli/integ/ITTestClusteringCommand.java | 9 +-
.../hudi/cli/integ/ITTestCommitsCommand.java | 32 +-
.../hudi/cli/integ/ITTestCompactionCommand.java | 7 +-
.../cli/integ/ITTestHDFSParquetImportCommand.java | 4 +-
.../hudi/cli/integ/ITTestMarkersCommand.java | 4 +-
.../hudi/cli/integ/ITTestRepairsCommand.java | 4 +-
.../hudi/cli/integ/ITTestSavepointsCommand.java | 10 +-
...Test.java => HoodieCLIIntegrationTestBase.java} | 2 +-
...t.java => HoodieCLIIntegrationTestHarness.java} | 2 +-
.../apache/hudi/client/BaseHoodieWriteClient.java | 28 +-
.../org/apache/hudi/config/HoodieWriteConfig.java | 24 +-
.../java/org/apache/hudi/io/HoodieMergeHandle.java | 11 +-
.../table/action/index/RunIndexActionExecutor.java | 78 ++--
.../action/index/ScheduleIndexActionExecutor.java | 5 +-
.../io/storage/TestHoodieHFileReaderWriter.java | 41 ++
.../hudi/io/storage/TestHoodieOrcReaderWriter.java | 10 +-
.../io/storage/TestHoodieReaderWriterBase.java | 18 +-
.../apache/hudi/client/HoodieFlinkWriteClient.java | 3 +-
.../org/apache/hudi/table/HoodieFlinkTable.java | 14 +-
.../index/bloom/TestFlinkHoodieBloomIndex.java | 2 +-
.../apache/hudi/client/HoodieJavaWriteClient.java | 4 +-
.../apache/hudi/client/SparkRDDWriteClient.java | 12 +-
.../RDDCustomColumnsSortPartitioner.java | 2 +-
.../org/apache/hudi/table/HoodieSparkTable.java | 21 +-
.../scala/org/apache/hudi/util/JFunction.scala | 17 +-
.../org/apache/spark/sql/hudi/SparkAdapter.scala | 18 +-
.../hudi/client/TestTableSchemaEvolution.java | 8 +-
.../functional/TestHoodieBackedTableMetadata.java | 2 +-
.../TestHoodieClientOnCopyOnWriteStorage.java | 4 +-
.../hudi/table/TestHoodieMergeOnReadTable.java | 2 +-
.../hudi/testutils/HoodieClientTestHarness.java | 36 +-
.../hudi/testutils/HoodieClientTestUtils.java | 2 +-
.../java/org/apache/hudi/avro/AvroSchemaUtils.java | 11 +
.../java/org/apache/hudi/avro/HoodieAvroUtils.java | 12 -
.../hudi/common/config/HoodieCommonConfig.java | 5 +
.../hudi/common/model/HoodieCommitMetadata.java | 2 +-
.../hudi/common/table/TableSchemaResolver.java | 467 ++++++++++-----------
.../table/timeline/HoodieActiveTimeline.java | 74 ++--
.../table/view/RocksDbBasedFileSystemView.java | 1 +
.../hudi/io/storage/HoodieAvroHFileReader.java | 4 +
.../metadata/FileSystemBackedTableMetadata.java | 39 +-
.../hudi/metadata/HoodieBackedTableMetadata.java | 40 +-
.../src/main/java/org/apache/hudi/util/Lazy.java | 2 +
.../hudi/common/table/TestTableSchemaResolver.java | 16 +-
.../hudi/common/testutils/HoodieTestUtils.java | 11 +-
hudi-examples/README.md | 2 +-
hudi-examples/hudi-examples-common/pom.xml | 28 ++
hudi-examples/hudi-examples-java/pom.xml | 28 ++
hudi-flink-datasource/hudi-flink/pom.xml | 20 +
.../apache/hudi/configuration/FlinkOptions.java | 10 +-
.../hudi/configuration/HadoopConfigurations.java | 14 +
.../hudi/sink/StreamWriteOperatorCoordinator.java | 10 +-
.../sink/bucket/BucketStreamWriteFunction.java | 25 +-
.../apache/hudi/sink/utils/HiveSyncContext.java | 4 +-
.../java/org/apache/hudi/sink/utils/Pipelines.java | 14 +-
.../org/apache/hudi/util/AvroSchemaConverter.java | 5 +-
.../hudi/util/FlinkStateBackendConverter.java | 2 +-
.../apache/hudi/sink/ITTestDataStreamWrite.java | 2 +-
hudi-flink-datasource/hudi-flink1.13.x/pom.xml | 46 +-
hudi-flink-datasource/hudi-flink1.14.x/pom.xml | 46 +-
hudi-gcp/pom.xml | 36 ++
.../hadoop/HoodieCopyOnWriteTableInputFormat.java | 7 +-
.../hudi/hadoop/realtime/HoodieVirtualKeyInfo.java | 16 +-
.../apache/hudi/hadoop/realtime/RealtimeSplit.java | 12 +-
.../utils/HoodieRealtimeInputFormatUtils.java | 7 +-
.../hudi/hadoop/TestHoodieParquetInputFormat.java | 27 ++
.../hudi/hadoop/testutils/InputFormatTestUtil.java | 29 +-
hudi-integ-test/pom.xml | 47 ++-
.../TestDFSHoodieTestSuiteWriterAdapter.java | 2 +-
.../integ/testsuite/TestFileDeltaInputWriter.java | 2 +-
.../testsuite/job/TestHoodieTestSuiteJob.java | 2 +-
.../reader/TestDFSAvroDeltaInputReader.java | 2 +-
.../reader/TestDFSHoodieDatasetInputReader.java | 2 +-
hudi-spark-datasource/hudi-spark-common/pom.xml | 37 ++
.../apache/hudi/HoodieDatasetBulkInsertHelper.java | 4 +-
.../BulkInsertDataInternalWriterHelper.java | 7 +-
.../scala/org/apache/hudi/DataSourceOptions.scala | 37 +-
.../main/scala/org/apache/hudi/DefaultSource.scala | 27 +-
.../scala/org/apache/hudi/HoodieBaseRelation.scala | 70 +--
.../org/apache/hudi/HoodieMergeOnReadRDD.scala | 6 +-
.../org/apache/hudi/HoodieSparkSqlWriter.scala | 16 +-
.../scala/org/apache/hudi/HoodieWriterUtils.scala | 70 +--
.../org/apache/hudi/IncrementalRelation.scala | 10 +-
.../spark/sql/hudi/HoodieSqlCommonUtils.scala | 7 +-
.../spark/sql/hudi/ProvidesHoodieConfig.scala | 10 +-
.../hudi/command/CreateHoodieTableCommand.scala | 4 +-
.../spark/sql/hudi/command/SqlKeyGenerator.scala | 9 +-
.../HoodieBulkInsertInternalWriterTestBase.java | 16 +-
.../test/resources/log4j-surefire-quiet.properties | 0
.../src/test/resources/log4j-surefire.properties | 0
.../sql/hudi/HoodieSparkSessionExtension.scala | 10 +-
.../spark/sql/hudi/analysis/HoodieAnalysis.scala | 66 ++-
.../hudi/command/MergeIntoHoodieTableCommand.scala | 2 +-
...Procedure.scala => DeleteMarkerProcedure.scala} | 53 ++-
.../hudi/command/procedures/HoodieProcedures.scala | 1 +
.../org/apache/hudi/TestHoodieSparkSqlWriter.scala | 23 +
.../hudi/TestTableSchemaResolverWithSparkSQL.scala | 2 +-
.../apache/hudi/functional/TestCOWDataSource.scala | 11 +
.../hudi/functional/TestColumnStatsIndex.scala | 2 +-
.../TestMetadataTableWithSparkDataSource.scala | 4 +-
.../spark/sql/hudi/HoodieSparkSqlTestBase.scala | 8 +-
.../apache/spark/sql/hudi/TestCreateTable.scala | 31 ++
.../apache/spark/sql/hudi/TestInsertTable.scala | 2 +-
.../sql/hudi/procedure/TestCallProcedure.scala | 40 ++
hudi-spark-datasource/hudi-spark2-common/pom.xml | 43 ++
hudi-spark-datasource/hudi-spark2/pom.xml | 60 +++
.../apache/spark/sql/adapter/Spark2Adapter.scala | 6 -
.../TestHoodieBulkInsertDataInternalWriter.java | 47 ++-
.../TestHoodieDataSourceInternalWriter.java | 1 -
.../spark/sql/adapter/BaseSpark3Adapter.scala | 4 +-
.../apache/spark/sql/adapter/Spark3_1Adapter.scala | 13 -
.../sql/hudi/ResolveHudiAlterTableCommand312.scala | 7 +-
.../org/apache/hudi/Spark3DefaultSource.scala | 17 +-
.../apache/spark/sql/adapter/Spark3_2Adapter.scala | 13 -
.../hudi/ResolveHudiAlterTableCommandSpark32.scala | 59 +--
.../sql/hudi/analysis/HoodieSpark3Analysis.scala | 103 +++--
.../spark/sql/hudi/catalog/HoodieCatalog.scala | 118 +++---
.../sql/hudi/catalog/HoodieInternalV2Table.scala | 4 +-
.../TestHoodieBulkInsertDataInternalWriter.java | 2 +-
hudi-sync/hudi-adb-sync/pom.xml | 36 ++
hudi-sync/hudi-datahub-sync/pom.xml | 18 +
.../java/org/apache/hudi/hive/HiveSyncTool.java | 2 +
.../org/apache/hudi/hive/TestHiveSyncTool.java | 3 +-
.../hudi/timeline/service/TimelineService.java | 3 +-
.../org/apache/hudi/utilities/HoodieIndexer.java | 11 +-
.../utilities/sources/helpers/KafkaOffsetGen.java | 19 +-
.../utilities/transform/FlatteningTransformer.java | 6 +-
.../transform/SqlFileBasedTransformer.java | 4 +-
.../transform/SqlQueryBasedTransformer.java | 6 +-
.../apache/hudi/utilities/TestHoodieIndexer.java | 99 ++++-
.../functional/HoodieDeltaStreamerTestBase.java | 2 +-
.../functional/TestHoodieDeltaStreamer.java | 22 +
.../utilities/sources/TestHoodieIncrSource.java | 38 +-
.../utilities/sources/TestJsonKafkaSource.java | 39 ++
.../hudi/utilities/sources/TestSqlSource.java | 2 +-
.../debezium/TestAbstractDebeziumSource.java | 2 +-
.../utilities/testutils/UtilitiesTestBase.java | 20 +-
.../AbstractCloudObjectsSourceTestBase.java | 2 +-
.../sources/AbstractDFSSourceTestBase.java | 2 +-
.../transform/TestSqlFileBasedTransformer.java | 2 +-
.../pom.xml | 286 ++++---------
.../main/java/org/apache/hudi/aws/bundle/Main.java | 10 +-
packaging/hudi-datahub-sync-bundle/pom.xml | 2 +
packaging/hudi-flink-bundle/pom.xml | 2 +
packaging/hudi-gcp-bundle/pom.xml | 2 +
packaging/hudi-hadoop-mr-bundle/pom.xml | 2 +
packaging/hudi-hive-sync-bundle/pom.xml | 2 +
packaging/hudi-integ-test-bundle/pom.xml | 3 +-
packaging/hudi-kafka-connect-bundle/pom.xml | 2 +
packaging/hudi-presto-bundle/pom.xml | 2 +
packaging/hudi-spark-bundle/pom.xml | 3 +-
packaging/hudi-timeline-server-bundle/pom.xml | 2 +
packaging/hudi-trino-bundle/pom.xml | 2 +
packaging/hudi-utilities-bundle/pom.xml | 3 +-
packaging/hudi-utilities-slim-bundle/pom.xml | 3 +-
pom.xml | 46 +-
rfc/rfc-53/rfc-53.md | 4 +-
164 files changed, 2341 insertions(+), 1259 deletions(-)
rename
hudi-cli/src/test/java/org/apache/hudi/cli/{testutils/SparkUtilTest.java =>
TestSparkUtil.java} (95%)
rename
hudi-cli/src/test/java/org/apache/hudi/cli/testutils/{AbstractShellIntegrationTest.java
=> HoodieCLIIntegrationTestBase.java} (93%)
rename
hudi-cli/src/test/java/org/apache/hudi/cli/testutils/{AbstractShellBaseIntegrationTest.java
=> HoodieCLIIntegrationTestHarness.java} (96%)
copy
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/trees/HoodieLeafLike.scala
=>
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/util/JFunction.scala
(67%)
copy {hudi-common =>
hudi-spark-datasource/hudi-spark-common}/src/test/resources/log4j-surefire-quiet.properties
(100%)
copy {hudi-client/hudi-client-common =>
hudi-spark-datasource/hudi-spark-common}/src/test/resources/log4j-surefire.properties
(100%)
copy
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/{ShowSavepointsProcedure.scala
=> DeleteMarkerProcedure.scala} (52%)
copy packaging/{hudi-kafka-connect-bundle => hudi-aws-bundle}/pom.xml (54%)
copy
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/TableUtils.java
=>
packaging/hudi-aws-bundle/src/main/java/org/apache/hudi/aws/bundle/Main.java
(77%)
