This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a change to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git
discard 6f681078cf1a fix(ci): Flink version corresponds to used in Docker
image, resolution for Everit JSON schema (#18905)
discard 10dadf593f3f fix(spark): align CTAS partition fields by table
partition order (#18899)
discard 48ea522bf598 refactor: Add Lombok Builders (#17781)
discard 2bb92e242bb9 refactor: Add Lombok annotations to hudi-utilities (Part
1) (#17823)
discard d6ee4e285763 fix(flink): Trigger a failover after pending instants
recommitted for both global and partitioned RLI (#18793)
discard cb52f102eda2 fix: Fix NPE due to race condition while handling rocksdb
handles (#18834)
new d1b9e3f7dc4f feat(utilities): migrate
HoodieStreamerWriteStatusValidator into pre-commit validator framework (#18765)
new 6a65c1efd2c2 fix: Fix NPE due to race condition while handling rocksdb
handles (#18834)
new 6572fe07dd10 fix(flink): Trigger a failover after pending instants
recommitted for both global and partitioned RLI (#18793)
new 0471c31172c4 refactor: Add Lombok annotations to hudi-utilities (Part
1) (#17823)
new 7bc38af1be0b refactor: Add Lombok Builders (#17781)
new 5cdfefe4118a fix(spark): align CTAS partition fields by table
partition order (#18899)
new 7c8020b6aa31 fix(ci): Flink version corresponds to used in Docker
image, resolution for Everit JSON schema (#18905)
new 920d63ccb55c refactor: Add Lombok annotations to hudi-utilities (Part
3) (#17877)
new 7b457efbae56 refactor: Add Lombok annotations to hudi-common module
(part 3) (#17825)
new 32c564aea5ef refactor: Add Lombok Builder annotation to
HoodieLogFormat (#17785)
new 8d0358e760df chore(docker): silence Dockerfile lint warnings across
image set (#18664)
new d36f87f74ae3 refactor(core): Unify record key/index key splitting and
extraction (#18842)
new da2e4efa17d2 fix(streamer): Use checkpoint V1 for non-incremental
streamer sources (#18896)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6f681078cf1a)
\
N -- N -- N refs/heads/release-1.2.1 (da2e4efa17d2)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 13 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docker/hoodie/hadoop/base/Dockerfile | 10 +-
docker/hoodie/hadoop/base_java11/Dockerfile | 10 +-
docker/hoodie/hadoop/base_java17/Dockerfile | 2 +-
docker/hoodie/hadoop/datanode/Dockerfile | 3 +-
docker/hoodie/hadoop/historyserver/Dockerfile | 3 +-
docker/hoodie/hadoop/hive_base/Dockerfile | 14 +-
docker/hoodie/hadoop/namenode/Dockerfile | 3 +-
docker/hoodie/hadoop/prestobase/Dockerfile | 20 +-
docker/hoodie/hadoop/spark_base/Dockerfile | 26 +-
docker/hoodie/hadoop/sparkadhoc/Dockerfile | 10 +-
docker/hoodie/hadoop/sparkmaster/Dockerfile | 6 +-
docker/hoodie/hadoop/sparkworker/Dockerfile | 6 +-
docker/hoodie/hadoop/trinobase/Dockerfile | 8 +-
.../cli/commands/TestHoodieLogFileCommand.java | 34 +-
.../timeline/versioning/v1/TimelineArchiverV1.java | 11 +-
.../org/apache/hudi/config/HoodieIndexConfig.java | 11 +-
.../config/HoodiePreCommitValidatorConfig.java | 4 +-
.../apache/hudi/index/bucket/BucketIdentifier.java | 2 +-
.../hudi/index/bucket/HoodieBucketIndex.java | 4 +-
.../org/apache/hudi/io/HoodieAppendHandle.java | 4 +-
.../java/org/apache/hudi/io/HoodieWriteHandle.java | 13 +-
.../apache/hudi/keygen/CustomAvroKeyGenerator.java | 6 +-
.../java/org/apache/hudi/keygen/KeyGenUtils.java | 26 +-
.../metadata/HoodieBackedTableMetadataWriter.java | 7 +-
.../table/BucketIndexBulkInsertPartitioner.java | 4 +-
.../table/action/clean/CleanActionExecutor.java | 20 +-
.../table/action/rollback/RollbackHelperV1.java | 7 +-
.../hudi/client/TestBaseHoodieWriteClient.java | 5 +-
.../utils/TestLegacyArchivedMetaEntryReader.java | 12 +-
.../apache/hudi/config/TestHoodieWriteConfig.java | 20 +
.../org/apache/hudi/keygen/TestKeyGenUtils.java | 21 ++
.../hudi/testutils/HoodieWriteableTestTable.java | 12 +-
.../hudi/utils/TestMetadataConversionUtils.java | 14 +-
.../testutils/HoodieFlinkWriteableTestTable.java | 14 +-
...arkConsistentBucketDuplicateUpdateStrategy.java | 4 +-
.../org/apache/hudi/keygen/CustomKeyGenerator.java | 8 +-
.../functional/TestExternalPathHandling.java | 10 +-
.../apache/hudi/io/TestHoodieTimelineArchiver.java | 15 +-
.../java/org/apache/hudi/table/TestCleaner.java | 24 +-
.../hudi/testutils/HoodieCleanerTestBase.java | 31 +-
.../testutils/HoodieSparkClientTestHarness.java | 14 +-
hudi-common/pom.xml | 1 +
.../org/apache/hudi/common/HoodieCleanStat.java | 210 ++---------
.../hudi/common/HoodiePendingRollbackInfo.java | 18 +-
.../org/apache/hudi/common/HoodieRollbackStat.java | 34 +-
.../common/bloom/InternalDynamicBloomFilter.java | 9 +-
.../apache/hudi/common/bloom/InternalFilter.java | 7 +-
.../java/org/apache/hudi/common/bloom/Key.java | 8 +-
.../bootstrap/index/hfile/HFileBootstrapIndex.java | 15 +-
.../index/hfile/HFileBootstrapIndexReader.java | 22 +-
.../index/hfile/HFileBootstrapIndexWriter.java | 17 +-
.../apache/hudi/common/config/ConfigGroups.java | 20 +-
.../apache/hudi/common/config/ConfigProperty.java | 66 +---
.../apache/hudi/common/config/HoodieConfig.java | 12 +-
.../DirectMarkerBasedDetectionStrategy.java | 21 +-
.../hudi/common/data/HoodieBaseListData.java | 16 +-
.../hudi/common/engine/HoodieEngineContext.java | 18 +-
.../hudi/common/engine/HoodieReaderContext.java | 128 ++-----
.../apache/hudi/common/engine/RecordContext.java | 12 +-
.../java/org/apache/hudi/common/fs/FSUtils.java | 9 +-
.../hudi/common/fs/FailSafeConsistencyGuard.java | 16 +-
.../hudi/common/fs/OptimisticConsistencyGuard.java | 10 +-
.../common/heartbeat/HoodieHeartbeatUtils.java | 7 +-
.../apache/hudi/common/schema/HoodieSchema.java | 38 +-
.../common/schema/HoodieSchemaCompatibility.java | 8 +-
.../hudi/common/schema/HoodieSchemaField.java | 14 +-
.../common/table/checkpoint/CheckpointUtils.java | 32 +-
.../hudi/common/table/log/HoodieLogFormat.java | 323 +++++++----------
.../table/checkpoint/TestCheckpointUtils.java | 66 +---
.../hudi/examples/common/RandomJsonSource.java | 4 +-
.../hudi/examples/common/TestRandomJsonSource.java | 76 ++++
.../apache/hudi/configuration/OptionsResolver.java | 13 +-
.../bucket/ConsistentBucketAssignFunction.java | 2 +-
.../ConsistentBucketStreamWriteFunction.java | 4 +-
.../org/apache/hudi/table/HoodieTableSource.java | 2 +-
.../hudi/configuration/TestOptionsResolver.java | 4 +-
.../common/table/log/HoodieLogFormatWriter.java | 114 +++---
.../common/functional/TestHoodieLogFormat.java | 402 ++++++++++++++-------
.../TestHoodieLogFormatAppendFailure.java | 22 +-
.../hudi/common/table/TestTableSchemaResolver.java | 10 +-
....java => TestHoodieLogFormatWriterBuilder.java} | 13 +-
.../table/timeline/TestArchivedTimelineV1.java | 20 +-
.../table/view/TestIncrementalFSViewSync.java | 13 +-
.../common/testutils/HoodieCommonTestHarness.java | 14 +-
.../hudi/common/testutils/HoodieTestTable.java | 24 +-
.../testutils/reader/HoodieFileSliceTestUtils.java | 7 +-
.../hudi/hadoop/testutils/InputFormatTestUtil.java | 25 +-
.../helpers/DFSTestSuitePathSelector.java | 7 +-
.../scala/org/apache/hudi/BucketIndexSupport.scala | 4 +-
.../HoodieFileGroupReaderBasedFileFormat.scala | 2 +-
.../sql/hudi/streaming/HoodieStreamSourceV2.scala | 2 +-
...dieSparkMergeOnReadTableInsertUpdateDelete.java | 7 +-
.../procedure/TestShowTimelineTableProcedure.scala | 8 +-
.../hudi/command/DeleteHoodieTableCommand.scala | 5 +-
.../hudi/command/DeleteHoodieTableCommand.scala | 5 +-
.../apache/hudi/hive/testutils/HiveTestUtil.java | 18 +-
.../hudi/utilities/sources/GcsEventsSource.java | 12 +-
.../hudi/utilities/sources/HiveIncrPullSource.java | 11 +-
.../hudi/utilities/sources/HoodieIncrSource.java | 4 +-
.../apache/hudi/utilities/sources/InputBatch.java | 9 -
.../apache/hudi/utilities/sources/JdbcSource.java | 8 +-
.../apache/hudi/utilities/sources/KafkaSource.java | 7 +-
.../hudi/utilities/sources/KinesisSource.java | 5 +-
.../hudi/utilities/sources/PulsarSource.java | 4 +-
.../org/apache/hudi/utilities/sources/Source.java | 53 +--
.../hudi/utilities/sources/SqlFileBasedSource.java | 4 +-
.../utilities/sources/debezium/DebeziumSource.java | 5 +-
.../utilities/sources/helpers/DFSPathSelector.java | 6 +-
.../sources/helpers/DatePartitionPathSelector.java | 6 +-
.../sources/helpers/S3EventsMetaSelector.java | 9 +-
.../hudi/utilities/streamer/BootstrapExecutor.java | 16 +-
.../utilities/streamer/DefaultStreamContext.java | 20 +-
.../apache/hudi/utilities/streamer/ErrorEvent.java | 37 +-
.../utilities/streamer/ErrorTableCommitter.java | 86 +++++
.../streamer/HoodieMultiTableStreamer.java | 34 +-
.../hudi/utilities/streamer/HoodieStreamer.java | 62 ++--
.../utilities/streamer/HoodieStreamerUtils.java | 8 +-
.../streamer/NoNewDataTerminationStrategy.java | 8 +-
.../utilities/streamer/SchedulerConfGenerator.java | 12 +-
.../utilities/streamer/SourceFormatAdapter.java | 23 +-
.../utilities/streamer/SparkSampleWritesUtils.java | 24 +-
.../apache/hudi/utilities/streamer/StreamSync.java | 345 +++++++-----------
.../streamer/StreamerCheckpointUtils.java | 35 +-
.../streamer/SuccessfulRecordCounter.java | 106 ++++++
.../utilities/streamer/TableExecutionContext.java | 58 +--
.../utilities/streamer/WriteErrorReporter.java | 69 ++++
.../validator/SparkWriteErrorValidator.java | 136 +++++++
.../TestHoodieMetadataTableValidator.java | 31 +-
.../hudi/utilities/TestHoodieRepairTool.java | 8 +-
.../deltastreamer/HoodieDeltaStreamerTestBase.java | 53 ++-
.../MockConfigurationHotUpdateStrategy.java | 7 +-
.../deltastreamer/TestHoodieDeltaStreamer.java | 104 +++---
.../TestHoodieDeltaStreamerWithMultiWriter.java | 18 +-
.../TestHoodieMultiTableDeltaStreamer.java | 12 +-
.../deltastreamer/TestSourceFormatAdapter.java | 6 +-
.../functional/TestHiveSchemaProvider.java | 10 +-
.../functional/TestJdbcbasedSchemaProvider.java | 7 +-
.../TestHoodieMultiTableServicesMain.java | 14 +-
.../utilities/sources/BaseTestKafkaSource.java | 22 +-
.../sources/S3EventsHoodieIncrSourceHarness.java | 23 +-
.../hudi/utilities/sources/TestDataSource.java | 15 +-
.../sources/TestGcsEventsHoodieIncrSource.java | 6 +-
.../utilities/sources/TestGcsEventsSource.java | 3 +-
.../utilities/sources/TestHiveIncrPullSource.java | 99 +++++
.../utilities/sources/TestHoodieIncrSource.java | 39 +-
.../hudi/utilities/sources/TestInputBatch.java | 6 +-
.../hudi/utilities/sources/TestJdbcSource.java | 5 +-
.../TestStreamerSourceCheckpointVersion.java | 351 ++++++++++++++++++
.../helpers/TestDFSPathSelectorCommonMethods.java | 15 +
.../streamer/TestErrorTableCommitter.java | 135 +++++++
.../streamer/TestHoodieIncrSourceE2E.java | 9 +-
.../TestParquetDfsCheckpointFormatOnV6.java | 128 +++++++
.../hudi/utilities/streamer/TestStreamSync.java | 31 +-
.../streamer/TestStreamerCheckpointUtils.java | 52 +++
.../streamer/TestSuccessfulRecordCounter.java | 200 ++++++++++
.../utilities/streamer/TestWriteErrorReporter.java | 99 +++++
.../validator/TestSparkWriteErrorValidator.java | 198 ++++++++++
.../hudi/utilities/testutils/JdbcTestUtils.java | 18 +-
.../utilities/testutils/UtilitiesTestBase.java | 7 +-
.../testutils/sources/AbstractBaseTestSource.java | 18 +-
.../sources/DistributedTestDataSource.java | 16 +-
.../checkpoint-v6/parquet-dfs-v1-fixture.zip | Bin 0 -> 103450 bytes
162 files changed, 3354 insertions(+), 2095 deletions(-)
create mode 100644
hudi-examples/hudi-examples-spark/src/test/java/org/apache/hudi/examples/common/TestRandomJsonSource.java
rename
hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/log/block/{TestHoodieLogWriterBuilder.java
=> TestHoodieLogFormatWriterBuilder.java} (87%)
rename {hudi-common =>
hudi-hadoop-common}/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java
(98%)
create mode 100644
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/ErrorTableCommitter.java
create mode 100644
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/SuccessfulRecordCounter.java
create mode 100644
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/WriteErrorReporter.java
create mode 100644
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/validator/SparkWriteErrorValidator.java
create mode 100644
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHiveIncrPullSource.java
create mode 100644
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestStreamerSourceCheckpointVersion.java
create mode 100644
hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/TestErrorTableCommitter.java
create mode 100644
hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/TestParquetDfsCheckpointFormatOnV6.java
create mode 100644
hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/TestSuccessfulRecordCounter.java
create mode 100644
hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/TestWriteErrorReporter.java
create mode 100644
hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/validator/TestSparkWriteErrorValidator.java
create mode 100644
hudi-utilities/src/test/resources/checkpoint-v6/parquet-dfs-v1-fixture.zip
