This is an automated email from the ASF dual-hosted git repository.
danny0405 pushed a change to branch release-0.10.0
in repository https://gitbox.apache.org/repos/asf/hudi.git.
from 92dbb41 Bumping release candidate number 2
new 69d1435 [MINOR] Fix typo,rename 'getUrlEncodePartitoning' to
'getUrlEncodePartitioning' (#4130)
new 63a7df7 [HUDI-2642] Add support ignoring case in update sql operation
(#3882)
new c38ebbc [HUDI-2891] Fix write configs for Java engine in Kafka
Connect Sink (#4161)
new df70e32 Revert "[HUDI-2855] Change the default value of
'PAYLOAD_CLASS_NAME' to 'DefaultHoodieRecordPayload' (#4115)" (#4169)
new 7df81a6 Revert "[HUDI-2856] Bit cask disk map delete modified
(#4116)" (#4171)
new 9d7f681 [HUDI-2880] Fixing loading of props from default dir (#4167)
new 7d2c248 [HUDI-2881] Compact the file group with larger log files to
reduce write amplification (#4152)
new 1ee35c4 Fixed partitions produced by layout optimization in case
order-by key is composed of a single column (#4183)
new 3e2116e [MINOR] Fix the wrong usage of timestamp length variable bug
(#4179)
new 318818b [HUDI-2904] Fix metadata table archival overstepping between
regular writers and table services (#4186)
new 6c79a68 [HUDI-2914] Fix remote timeline server config for flink
(#4191)
new fe2aae4 [minor] Refactor write profile to always generate fs view
(#4198)
new 2a4e718 [HUDI-2924] Refresh the fs view on successful checkpoints for
write profile (#4199)
new 1bf13d1 [MINOR] use catalog schema if can not find table schema
(#4182)
new 7de5d17 [HUDI-2902] Fixing populate meta fields with Hfile writers
and Disabling virtual keys by default for metadata table (#4194)
new 008df11 [HUDI-2911] Removing default value for
`PARTITIONPATH_FIELD_NAME` resulting in incorrect `KeyGenerator` configuration
(#4195)
new 342acbe Revert "[HUDI-2495] Resolve inconsistent key generation for
timestamp types by GenericRecord and Row (#3944)" (#4201)
new a49e5ac [HUDI-2894][HUDI-2905] Metadata table - avoiding key lookup
failures on base files over S3 (#4185)
new 474af05 Revert "[HUDI-2489]Tuning HoodieROTablePathFilter by caching
hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests"
new 31947a0 [MINOR] Mitigate CI jobs timeout issues (#4173)
new 7034d1a [HUDI-2933] DISABLE Metadata table by default (#4213)
new b88d4be [HUDI-2890] Kafka Connect: Fix failed writes and avoid table
service concurrent operations (#4211)
new e131507 [HUDI-2923] Fixing metadata table reader when metadata
compaction is inflight (#4206)
new 148e069 [HUDI-2934] Optimize RequestHandler code style
new 40879bc [HUDI-2935] Remove special casing of clustering in
deltastreamer checkpoint retrival (#4216)
new e9e6939 Bumping release candidate number 3
The 26 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
azure-pipelines.yml | 14 +-
docker/hoodie/hadoop/base/pom.xml | 2 +-
docker/hoodie/hadoop/datanode/pom.xml | 2 +-
docker/hoodie/hadoop/historyserver/pom.xml | 2 +-
docker/hoodie/hadoop/hive_base/pom.xml | 2 +-
docker/hoodie/hadoop/namenode/pom.xml | 2 +-
docker/hoodie/hadoop/pom.xml | 2 +-
docker/hoodie/hadoop/prestobase/pom.xml | 2 +-
docker/hoodie/hadoop/spark_base/pom.xml | 2 +-
docker/hoodie/hadoop/sparkadhoc/pom.xml | 2 +-
docker/hoodie/hadoop/sparkmaster/pom.xml | 2 +-
docker/hoodie/hadoop/sparkworker/pom.xml | 2 +-
hudi-aws/pom.xml | 4 +-
hudi-cli/pom.xml | 2 +-
hudi-client/hudi-client-common/pom.xml | 4 +-
.../hudi/client/AbstractHoodieWriteClient.java | 40 ++-
.../apache/hudi/config/HoodieCompactionConfig.java | 23 ++
.../org/apache/hudi/config/HoodieWriteConfig.java | 12 +-
.../apache/hudi/io/HoodieSortedMergeHandle.java | 3 +-
.../hudi/io/storage/HoodieFileWriterFactory.java | 2 +-
.../apache/hudi/io/storage/HoodieHFileWriter.java | 14 +-
.../metadata/HoodieBackedTableMetadataWriter.java | 10 +-
.../LogFileSizeBasedCompactionStrategy.java | 9 +-
.../io/storage/TestHoodieHFileReaderWriter.java | 55 +++-
...chema.avsc => exampleSchemaWithMetaFields.avsc} | 36 ++-
hudi-client/hudi-flink-client/pom.xml | 4 +-
.../apache/hudi/client/HoodieFlinkWriteClient.java | 11 +-
.../FlinkHoodieBackedTableMetadataWriter.java | 3 +-
hudi-client/hudi-java-client/pom.xml | 4 +-
hudi-client/hudi-spark-client/pom.xml | 4 +-
.../SparkHoodieBackedTableMetadataWriter.java | 3 +-
.../java/org/apache/spark/OrderingIndexHelper.java | 2 +-
.../hudi/client/functional/TestHBaseIndex.java | 6 +-
.../functional/TestHoodieBackedMetadata.java | 281 +++++++++++++++++++++
.../client/functional/TestHoodieMetadataBase.java | 2 +-
.../strategy/TestHoodieCompactionStrategy.java | 9 +-
.../TestHoodieSparkMergeOnReadTableRollback.java | 2 +-
hudi-client/pom.xml | 2 +-
hudi-common/pom.xml | 2 +-
.../java/org/apache/hudi/avro/HoodieAvroUtils.java | 3 -
.../bootstrap/index/HFileBootstrapIndex.java | 8 +-
.../common/config/DFSPropertiesConfiguration.java | 31 ++-
.../hudi/common/config/HoodieMetadataConfig.java | 14 +-
.../hudi/common/table/HoodieTableConfig.java | 2 +-
.../common/util/collection/BitCaskDiskMap.java | 6 +-
...ruptedDataException.java => ExceptionUtil.java} | 22 +-
.../apache/hudi/io/storage/HoodieHFileReader.java | 2 +-
.../hudi/keygen/constant/KeyGeneratorOptions.java | 2 +-
.../hudi/metadata/HoodieBackedTableMetadata.java | 2 +-
.../HoodieMetadataMergedLogRecordReader.java | 4 +-
.../hudi/metadata/HoodieTableMetadataUtil.java | 8 +-
.../hudi/common/testutils/FileCreateUtils.java | 10 +
.../util/TestDFSPropertiesConfiguration.java | 5 +-
hudi-examples/pom.xml | 2 +-
hudi-flink/pom.xml | 2 +-
.../java/org/apache/hudi/sink/CleanFunction.java | 4 +-
.../sink/partitioner/BucketAssignFunction.java | 8 +-
.../hudi/sink/partitioner/BucketAssigner.java | 6 +-
.../partitioner/profile/DeltaWriteProfile.java | 12 +-
.../sink/partitioner/profile/WriteProfile.java | 51 ++--
.../java/org/apache/hudi/util/StreamerUtil.java | 21 +-
.../hudi/sink/partitioner/TestBucketAssigner.java | 2 +-
.../org/apache/hudi/utils/TestStreamerUtil.java | 11 +
hudi-hadoop-mr/pom.xml | 2 +-
.../hudi/hadoop/HoodieROTablePathFilter.java | 21 +-
hudi-integ-test/pom.xml | 2 +-
hudi-kafka-connect/README.md | 71 +++---
hudi-kafka-connect/demo/config-sink-hive.json | 1 -
hudi-kafka-connect/demo/config-sink.json | 1 -
hudi-kafka-connect/demo/setupKafka.sh | 2 +-
hudi-kafka-connect/pom.xml | 4 +-
.../transaction/ConnectTransactionCoordinator.java | 55 +++-
.../writers/ConnectTransactionServices.java | 2 +-
.../hudi/connect/writers/KafkaConnectConfigs.java | 14 +
.../writers/KafkaConnectTransactionServices.java | 30 ++-
.../writers/KafkaConnectWriterProvider.java | 14 +-
.../connect/TestConnectTransactionCoordinator.java | 117 ++++++---
.../helper/MockConnectTransactionServices.java | 3 +-
.../hudi/writers/TestBufferedConnectWriter.java | 2 +
.../test/resources/log4j-surefire-quiet.properties | 0
.../src/test/resources/log4j-surefire.properties | 0
hudi-spark-datasource/hudi-spark-common/pom.xml | 4 +-
.../scala/org/apache/hudi/DataSourceOptions.scala | 2 +-
hudi-spark-datasource/hudi-spark/pom.xml | 4 +-
.../scala/org/apache/hudi/HoodieWriterUtils.scala | 1 -
.../sql/catalyst/catalog/HoodieCatalogTable.scala | 2 +-
.../org/apache/spark/sql/hudi/HoodieSqlUtils.scala | 2 +-
.../hudi/command/DeleteHoodieTableCommand.scala | 2 +-
.../command/InsertIntoHoodieTableCommand.scala | 2 +-
.../hudi/command/MergeIntoHoodieTableCommand.scala | 2 +-
.../spark/sql/hudi/command/SqlKeyGenerator.scala | 3 +-
.../hudi/command/UpdateHoodieTableCommand.scala | 7 +-
.../hudi/TestGenericRecordAndRowConsistency.scala | 104 --------
.../org/apache/hudi/TestHoodieSparkSqlWriter.scala | 33 ++-
.../apache/spark/sql/hudi/TestUpdateTable.scala | 45 +++-
hudi-spark-datasource/hudi-spark2/pom.xml | 4 +-
hudi-spark-datasource/hudi-spark3/pom.xml | 4 +-
hudi-spark-datasource/pom.xml | 2 +-
hudi-sync/hudi-dla-sync/pom.xml | 2 +-
hudi-sync/hudi-hive-sync/pom.xml | 2 +-
.../org/apache/hudi/hive/SchemaDifference.java | 1 -
.../hudi/hive/ddl/QueryBasedDDLExecutor.java | 5 +-
hudi-sync/hudi-sync-common/pom.xml | 2 +-
hudi-sync/pom.xml | 2 +-
hudi-timeline-service/pom.xml | 2 +-
.../hudi/timeline/service/RequestHandler.java | 14 +-
hudi-utilities/pom.xml | 2 +-
.../hudi/utilities/deltastreamer/DeltaSync.java | 3 -
...inuousModeWithMultipleWriters.COPY_ON_WRITE.zip | Bin 2592485 -> 2592484
bytes
...inuousModeWithMultipleWriters.MERGE_ON_READ.zip | Bin 3015940 -> 3015939
bytes
packaging/hudi-flink-bundle/pom.xml | 2 +-
packaging/hudi-hadoop-mr-bundle/pom.xml | 2 +-
packaging/hudi-hive-sync-bundle/pom.xml | 2 +-
packaging/hudi-integ-test-bundle/pom.xml | 2 +-
packaging/hudi-kafka-connect-bundle/pom.xml | 2 +-
packaging/hudi-presto-bundle/pom.xml | 2 +-
packaging/hudi-spark-bundle/pom.xml | 2 +-
packaging/hudi-timeline-server-bundle/pom.xml | 2 +-
packaging/hudi-utilities-bundle/pom.xml | 2 +-
pom.xml | 2 +-
120 files changed, 987 insertions(+), 455 deletions(-)
copy
hudi-client/hudi-client-common/src/test/resources/{exampleEvolvedSchema.avsc =>
exampleSchemaWithMetaFields.avsc} (58%)
copy
hudi-common/src/main/java/org/apache/hudi/exception/{HoodieCorruptedDataException.java
=> ExceptionUtil.java} (66%)
copy {hudi-utilities =>
hudi-kafka-connect}/src/test/resources/log4j-surefire-quiet.properties (100%)
copy {hudi-spark-datasource/hudi-spark3 =>
hudi-kafka-connect}/src/test/resources/log4j-surefire.properties (100%)
delete mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestGenericRecordAndRowConsistency.scala
