This is an automated email from the ASF dual-hosted git repository.
codope pushed a change to branch release-0.12.0
in repository https://gitbox.apache.org/repos/asf/hudi.git
from 3383b2388b Create release branch for version 0.12.0.
new 7097eec4cd [HUDI-4504] Disable metadata table by default for flink
(#6241)
new c647af536e [HUDI-4505] Returns instead of throws if lock file exists
for FileSystemBasedLockProvider (#6242)
new cdf2657420 [HUDI-4507] Improve file name extraction logic in metadata
utils (#6250)
new a701201fac [MINOR] Fix convertPathWithScheme tests (#6251)
new 9fe7c2f32b [MINOR] Add license header (#6247)
new 7a7e55d8b2 [HUDI-4025] Add Presto and Trino query node to validate
queries (#5578)
new 85b2612bbe [HUDI-4518] Free lock if allocated but not acquired (#6272)
new 6875fe43d0 [HUDI-4510] Repair config "hive_sync.metastore.uris" in
flink sql hive schema sync is not effective (#6257)
new 5125fba450 [HUDI-3848] Fixing minor bug in listing based rollback
request generation (#6244)
new 46200a1411 [HUDI-4512][HUDI-4513] Fix bundle name for spark3 profile
(#6261)
new 9f284a0125 [HUDI-4501] Throwing exception when restore is attempted
with hoodie.arhive.beyond.savepoint is enabled (#6239)
new 92843dc409 [HUDI-4516] fix Task not serializable error when run
HoodieCleaner after one failure (#6265)
new 8ce363a169 remove test resources (#6147)
new 6234064d14 [HUDI-4477] Adjust partition number of flink sink task
(#6218)
new 78a56ba082 [HUDI-4298] Mor table reading for base and log files lost
sequence of events (#6286)
new 79ef1a784b [HUDI-4525] Fixing Spark 3.3 `AvroSerializer`
implementation (#6279)
new ea90f1ecd7 [HUDI-4447] fix no partitioned path extractor error when
sync meta (#6263)
new bf56e6fe24 [HUDI-4520] Support qualified table 'db.table' in call
procedures (#6274)
new e49c2826e4 [HUDI-4531] Wrong partition path for flink hive catalog
when the partition fields are not in the last (#6292)
new d3b0ee7ee6 [HUDI-4487] support to create ro/rt table by spark sql
(#6262)
new d013cf5cec [HUDI-4533] Fix RunCleanProcedure's
ArrayIndexOutOfBoundsException (#6293)
new 9d5d4b9219 [HUDI-4536] ClusteringOperator causes the
NullPointerException when writing with BulkInsertWriterHelper in clustering
(#6298)
new 0b24419376 [HUDI-4385] Support online compaction in the flink batch
mode write (#6093)
new 9091202a5f [HUDI-4530] fix default payloadclass in mor is different
with cow (#6288)
new 4d6aac90d9 [HUDI-4545] Do not modify the current record directly for
OverwriteNonDefaultsWithLatestAvroPayload (#6306)
new 76cc3c9c4d [HUDI-4547] Fix SortOperatorGen sort indices (#6309)
new cd14c26ea5 [HUDI-4470] Remove spark dataPrefetch disabled prop in
DefaultSource
new ccfaf01612 [HUDI-4540] Cover different table types in functional tests
of Spark structured streaming (#6317)
new afd5eabbc0 [HUDI-4474] Fix inferring props for meta sync (#6310)
new bbf6c880e4 [HUDI-4550] Fallback to listing based rollback for
completed instant (#6313)
new 0cbe468b6c [HUDI-4303] Adding 4 to 5 upgrade handler to check for old
deprecated "default" partition value (#6248)
new b124f5e487 [HUDI-4534] Fixing upgrade to reload Metaclient for
deltastreamer writes (#6296)
new 4dd0a2f46e [HUDI-4517] If no marker type file, fallback to timeline
based marker (#6266)
new 6cf5c2447a [HUDI-3669] Add a remote request retry mechanism for
'Remotehoodietablefiles… (#5884)
new ce52f63f79 [HUDI-4464] Clear warnings in Azure CI (#6210)
new c272d8e133 [MINOR] Update PR description template (#6323)
new dbc1697431 [HUDI-4508] Repair the exception when reading optimized
query for mor in hive and presto/trino (#6254)
new 1b32114507 [HUDI-4548] Unpack the column max/min to string instead of
Utf8 for Mor table (#6311)
new 7777c78e5c [HUDI-4514] optimize CTAS to adapt to saveAsTable api in
different modes (#6295)
new 741d3023fb [HUDI-4447] fix SQL metasync when perform delete table
operation (#6180)
new 58ca196b59 [MINOR] improve flink dummySink's parallelism (#6325)
new 449ba61d44 [HUDI-4568] Shade dropwizard metrics-core in
hudi-aws-bundle (#6327)
new 528aaf7bc6 [HUDI-4572] Fix 'Not a valid schema field: ts' error in
HoodieFlinkCompactor if precombine field is not ts (#6331)
new 4970ab0417 [HUDI-4570] Fix hive sync path error due to reuse of
storage descriptors. (#6329)
new 79ffeabba4 [HUDI-4571] Fix partition extractor infer function when
partition field mismatch (#6333)
new 0c1e65581d [HUDI-4570] Add test for updating multiple partitions in
hive sync (#6340)
new 354b03fadb [MINOR] Fix wrong key to determine sync sql cascade (#6339)
new d99bb22944 Bumping release candidate number 2
The 48 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md | 48 ++----
.github/workflows/bot.yml | 2 +-
azure-pipelines.yml | 26 ++--
docker/demo/config/hoodie-incr.properties | 5 +-
.../test-suite/deltastreamer-hive-sync-presto.yaml | 36 +++--
docker/hoodie/hadoop/base/pom.xml | 2 +-
docker/hoodie/hadoop/base_java11/pom.xml | 2 +-
docker/hoodie/hadoop/datanode/pom.xml | 2 +-
docker/hoodie/hadoop/historyserver/pom.xml | 2 +-
docker/hoodie/hadoop/hive_base/pom.xml | 2 +-
docker/hoodie/hadoop/namenode/pom.xml | 2 +-
docker/hoodie/hadoop/pom.xml | 2 +-
docker/hoodie/hadoop/prestobase/pom.xml | 2 +-
docker/hoodie/hadoop/spark_base/pom.xml | 2 +-
docker/hoodie/hadoop/sparkadhoc/pom.xml | 2 +-
docker/hoodie/hadoop/sparkmaster/pom.xml | 2 +-
docker/hoodie/hadoop/sparkworker/pom.xml | 2 +-
docker/hoodie/hadoop/trinobase/pom.xml | 2 +-
docker/hoodie/hadoop/trinocoordinator/pom.xml | 2 +-
docker/hoodie/hadoop/trinoworker/pom.xml | 2 +-
hudi-aws/pom.xml | 4 +-
hudi-cli/pom.xml | 2 +-
hudi-client/hudi-client-common/pom.xml | 4 +-
.../apache/hudi/client/BaseHoodieWriteClient.java | 12 +-
.../client/embedded/EmbeddedTimelineService.java | 5 +
.../lock/FileSystemBasedLockProvider.java | 15 +-
.../org/apache/hudi/config/HoodieWriteConfig.java | 17 +++
.../org/apache/hudi/io/HoodieAppendHandle.java | 6 +-
.../restore/CopyOnWriteRestoreActionExecutor.java | 1 -
.../restore/MergeOnReadRestoreActionExecutor.java | 1 -
.../rollback/BaseRollbackActionExecutor.java | 10 +-
.../rollback/BaseRollbackPlanActionExecutor.java | 2 +-
.../CopyOnWriteRollbackActionExecutor.java | 3 +-
.../rollback/ListingBasedRollbackStrategy.java | 6 +-
.../MergeOnReadRollbackActionExecutor.java | 3 +-
.../table/marker/MarkerBasedRollbackUtils.java | 40 +++--
...andler.java => FiveToFourDowngradeHandler.java} | 15 +-
.../table/upgrade/FourToFiveUpgradeHandler.java | 74 ++++++++++
.../hudi/table/upgrade/UpgradeDowngrade.java | 4 +
hudi-client/hudi-flink-client/pom.xml | 4 +-
hudi-client/hudi-java-client/pom.xml | 4 +-
hudi-client/hudi-spark-client/pom.xml | 4 +-
.../scala/org/apache/hudi/HoodieSparkUtils.scala | 1 +
.../org/apache/hudi/client/TestClientRollback.java | 55 +++++++
.../functional/TestHoodieBackedMetadata.java | 5 +-
.../TestHoodieClientOnCopyOnWriteStorage.java | 81 +++++++---
.../TestCopyOnWriteRollbackActionExecutor.java | 67 +++++++++
.../TestMergeOnReadRollbackActionExecutor.java | 19 +--
.../TestMarkerBasedRollbackStrategy.java | 22 +++
.../hudi/table/upgrade/TestUpgradeDowngrade.java | 92 +++++++++++-
hudi-client/pom.xml | 2 +-
hudi-common/pom.xml | 2 +-
.../java/org/apache/hudi/avro/HoodieAvroUtils.java | 35 +++--
.../java/org/apache/hudi/common/fs/FSUtils.java | 18 +++
.../OverwriteNonDefaultsWithLatestAvroPayload.java | 8 +-
.../hudi/common/table/HoodieTableVersion.java | 6 +-
.../common/table/view/FileSystemViewManager.java | 3 +-
.../table/view/FileSystemViewStorageConfig.java | 76 ++++++++++
.../view/RemoteHoodieTableFileSystemView.java | 67 +++++++--
.../hudi/common/util/DefaultSizeEstimator.java | 4 +-
.../common/util/HoodieRecordSizeEstimator.java | 4 +-
.../org/apache/hudi/common/util/MarkerUtils.java | 3 +-
.../hudi/common/util/PartitionPathEncodeUtils.java | 1 +
.../org/apache/hudi/common/util/RetryHelper.java | 46 +++---
.../util/collection/ExternalSpillableMap.java | 2 +-
.../common/util/collection/RocksDBBasedMap.java | 4 +-
.../hudi/metadata/HoodieTableMetadataUtil.java | 23 +--
.../org/apache/hudi/avro/TestHoodieAvroUtils.java | 8 +
.../org/apache/hudi/common/fs/TestFSUtils.java | 12 +-
.../apache/hudi/common/fs/TestStorageSchemes.java | 7 +-
...tOverwriteNonDefaultsWithLatestAvroPayload.java | 11 +-
hudi-examples/hudi-examples-common/pom.xml | 2 +-
hudi-examples/hudi-examples-flink/pom.xml | 2 +-
hudi-examples/hudi-examples-java/pom.xml | 2 +-
hudi-examples/hudi-examples-spark/pom.xml | 2 +-
hudi-examples/pom.xml | 2 +-
hudi-flink-datasource/hudi-flink/pom.xml | 7 +-
.../apache/hudi/configuration/FlinkOptions.java | 4 +-
.../hudi/sink/StreamWriteOperatorCoordinator.java | 5 +
.../hudi/sink/bulk/sort/SortOperatorGen.java | 5 +-
.../hudi/sink/clustering/ClusteringOperator.java | 1 +
.../hudi/sink/compact/CompactionPlanOperator.java | 9 +-
.../hudi/sink/compact/HoodieFlinkCompactor.java | 2 +
.../apache/hudi/sink/utils/HiveSyncContext.java | 3 +
.../java/org/apache/hudi/sink/utils/Pipelines.java | 9 +-
.../org/apache/hudi/table/HoodieTableSink.java | 4 +
.../apache/hudi/table/catalog/HiveSchemaUtils.java | 31 +++-
.../hudi/table/catalog/HoodieCatalogUtil.java | 20 +++
.../hudi/table/catalog/HoodieHiveCatalog.java | 32 ++--
.../hudi/table/catalog/TableOptionProperties.java | 8 +-
.../table/format/mor/MergeOnReadInputFormat.java | 11 +-
.../java/org/apache/hudi/util/CompactionUtil.java | 14 ++
.../java/org/apache/hudi/util/StreamerUtil.java | 24 ++-
.../sink/compact/ITTestHoodieFlinkCompactor.java | 18 +++
.../apache/hudi/table/ITTestHoodieDataSource.java | 6 +-
.../hudi/table/catalog/TestHoodieHiveCatalog.java | 28 ++--
.../apache/hudi/table/format/TestInputFormat.java | 26 ++++
hudi-flink-datasource/hudi-flink1.13.x/pom.xml | 4 +-
hudi-flink-datasource/hudi-flink1.14.x/pom.xml | 4 +-
hudi-flink-datasource/hudi-flink1.15.x/pom.xml | 4 +-
hudi-flink-datasource/pom.xml | 4 +-
hudi-gcp/pom.xml | 2 +-
hudi-hadoop-mr/pom.xml | 2 +-
.../hadoop/HoodieCopyOnWriteTableInputFormat.java | 16 ++
.../HoodieMergeOnReadTableInputFormat.java | 12 ++
hudi-integ-test/pom.xml | 12 +-
.../hudi/integ/testsuite/HoodieTestSuiteJob.java | 23 +++
.../integ/testsuite/configuration/DeltaConfig.java | 54 ++++++-
.../apache/hudi/integ/testsuite/dag/DagUtils.java | 114 +++++++++-----
.../integ/testsuite/dag/nodes/BaseQueryNode.java | 62 ++++++++
.../integ/testsuite/dag/nodes/HiveQueryNode.java | 46 ++----
.../integ/testsuite/dag/nodes/PrestoQueryNode.java | 60 ++++++++
.../integ/testsuite/dag/nodes/TrinoQueryNode.java | 60 ++++++++
.../org/apache/hudi/integ/ITTestHoodieDemo.java | 4 +-
.../src/test/resources/unit-test-cow-dag.yaml | 22 ++-
hudi-kafka-connect/pom.xml | 4 +-
hudi-spark-datasource/hudi-spark-common/pom.xml | 4 +-
.../main/scala/org/apache/hudi/DefaultSource.scala | 3 +-
.../scala/org/apache/hudi/HoodieCLIUtils.scala | 15 ++
.../sql/catalyst/catalog/HoodieCatalogTable.scala | 21 ++-
.../apache/spark/sql/hudi/HoodieOptionConfig.scala | 3 +-
.../spark/sql/hudi/ProvidesHoodieConfig.scala | 5 +
.../hudi/command/CreateHoodieTableCommand.scala | 33 ++++-
hudi-spark-datasource/hudi-spark/pom.xml | 4 +-
.../command/CreateHoodieTableAsSelectCommand.scala | 18 ++-
.../hudi/command/procedures/BaseProcedure.scala | 4 +-
.../procedures/CommitsCompareProcedure.scala | 3 +-
.../procedures/ExportInstantsProcedure.scala | 3 +-
.../RollbackToInstantTimeProcedure.scala | 3 +-
.../command/procedures/RunCleanProcedure.scala | 5 +-
.../procedures/ShowArchivedCommitsProcedure.scala | 3 +-
.../procedures/ShowCommitFilesProcedure.scala | 3 +-
.../procedures/ShowCommitPartitionsProcedure.scala | 3 +-
.../procedures/ShowCommitWriteStatsProcedure.scala | 3 +-
.../command/procedures/ShowCommitsProcedure.scala | 3 +-
.../TestConvertFilterToCatalystExpression.scala | 4 +-
.../org/apache/hudi/TestHoodieSparkSqlWriter.scala | 34 ++++-
.../apache/hudi/functional/TestCOWDataSource.scala | 70 ++++++++-
.../hudi/functional/TestStructuredStreaming.scala | 164 +++++++++++----------
.../apache/spark/sql/hudi/TestCreateTable.scala | 134 ++++++++++++++++-
.../spark/sql/hudi/TestHoodieOptionConfig.scala | 4 +-
.../apache/spark/sql/hudi/TestMergeIntoTable.scala | 3 +-
.../hudi/TestNestedSchemaPruningOptimization.scala | 1 -
.../procedure/HoodieSparkProcedureTestBase.scala} | 13 +-
.../hudi/procedure/TestBootstrapProcedure.scala | 3 +-
.../sql/hudi/procedure/TestCallProcedure.scala | 3 +-
.../sql/hudi/procedure/TestCleanProcedure.scala | 8 +-
.../hudi/procedure/TestClusteringProcedure.scala | 3 +-
.../sql/hudi/procedure/TestCommitsProcedure.scala | 4 +-
.../hudi/procedure/TestCompactionProcedure.scala | 3 +-
.../hudi/procedure/TestCopyToTableProcedure.scala | 3 +-
.../procedure/TestExportInstantsProcedure.scala | 4 +-
.../sql/hudi/procedure/TestFsViewProcedure.scala | 4 +-
.../procedure/TestHdfsParquetImportProcedure.scala | 3 +-
.../procedure/TestHoodieLogFileProcedure.scala | 4 +-
.../sql/hudi/procedure/TestMetadataProcedure.scala | 4 +-
.../sql/hudi/procedure/TestRepairsProcedure.scala | 3 +-
.../hudi/procedure/TestSavepointsProcedure.scala | 4 +-
.../procedure/TestShowFsPathDetailProcedure.scala | 4 +-
.../sql/hudi/procedure/TestStatsProcedure.scala | 4 +-
.../TestUpgradeOrDowngradeProcedure.scala | 7 +-
hudi-spark-datasource/hudi-spark2-common/pom.xml | 2 +-
hudi-spark-datasource/hudi-spark2/pom.xml | 4 +-
hudi-spark-datasource/hudi-spark3-common/pom.xml | 2 +-
hudi-spark-datasource/hudi-spark3.1.x/pom.xml | 4 +-
hudi-spark-datasource/hudi-spark3.2.x/pom.xml | 4 +-
.../org/apache/spark/sql/avro/AvroSerializer.scala | 27 +++-
hudi-spark-datasource/hudi-spark3.3.x/pom.xml | 4 +-
.../apache/spark/sql/avro/AvroDeserializer.scala | 35 ++---
.../org/apache/spark/sql/avro/AvroSerializer.scala | 121 +++++++++++----
hudi-spark-datasource/pom.xml | 2 +-
hudi-sync/hudi-adb-sync/pom.xml | 2 +-
hudi-sync/hudi-datahub-sync/pom.xml | 2 +-
hudi-sync/hudi-hive-sync/pom.xml | 2 +-
.../java/org/apache/hudi/hive/HiveSyncTool.java | 3 +-
.../org/apache/hudi/hive/ddl/HMSDDLExecutor.java | 5 +-
.../hudi/hive/ddl/QueryBasedDDLExecutor.java | 2 +-
.../lock/HiveMetastoreBasedLockProvider.java | 2 +
.../org/apache/hudi/hive/TestHiveSyncTool.java | 22 ++-
.../replication/TestHiveSyncGlobalCommitTool.java | 3 +
.../apache/hudi/hive/testutils/HiveTestUtil.java | 3 +
hudi-sync/hudi-sync-common/pom.xml | 2 +-
.../apache/hudi/sync/common/HoodieSyncConfig.java | 45 ++++--
.../hudi/sync/common/TestHoodieSyncConfig.java | 62 ++++++--
.../hudi/sync/common/util/TestConfigUtils.java | 18 +++
hudi-sync/pom.xml | 2 +-
hudi-timeline-service/pom.xml | 2 +-
.../service/handlers/marker/MarkerDirState.java | 2 +-
.../TestRemoteHoodieTableFileSystemView.java | 29 ++++
hudi-utilities/pom.xml | 2 +-
.../hudi/utilities/deltastreamer/DeltaSync.java | 1 +
packaging/hudi-aws-bundle/pom.xml | 7 +-
packaging/hudi-datahub-sync-bundle/pom.xml | 2 +-
packaging/hudi-flink-bundle/pom.xml | 3 +-
packaging/hudi-gcp-bundle/pom.xml | 2 +-
packaging/hudi-hadoop-mr-bundle/pom.xml | 2 +-
packaging/hudi-hive-sync-bundle/pom.xml | 2 +-
packaging/hudi-integ-test-bundle/pom.xml | 4 +-
packaging/hudi-kafka-connect-bundle/pom.xml | 3 +-
packaging/hudi-presto-bundle/pom.xml | 2 +-
packaging/hudi-spark-bundle/pom.xml | 2 +-
packaging/hudi-timeline-server-bundle/pom.xml | 2 +-
packaging/hudi-trino-bundle/pom.xml | 2 +-
packaging/hudi-utilities-bundle/pom.xml | 3 +-
packaging/hudi-utilities-slim-bundle/pom.xml | 3 +-
pom.xml | 18 ++-
scripts/release/deploy_staging_jars.sh | 1 +
scripts/release/validate_staged_bundles.sh | 1 +
208 files changed, 2241 insertions(+), 721 deletions(-)
copy hudi-integ-test/src/test/resources/unit-test-cow-dag.yaml =>
docker/demo/config/test-suite/deltastreamer-hive-sync-presto.yaml (72%)
copy
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/{FourToThreeDowngradeHandler.java
=> FiveToFourDowngradeHandler.java} (66%)
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FourToFiveUpgradeHandler.java
create mode 100644
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/BaseQueryNode.java
create mode 100644
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/PrestoQueryNode.java
create mode 100644
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/TrinoQueryNode.java
copy
hudi-spark-datasource/hudi-spark/src/{main/scala/org/apache/spark/sql/hudi/command/procedures/ProcedureArgs.scala
=>
test/scala/org/apache/spark/sql/hudi/procedure/HoodieSparkProcedureTestBase.scala}
(75%)
