This is an automated email from the ASF dual-hosted git repository.
forwardxu pushed a change to branch release-0.12.1
in repository https://gitbox.apache.org/repos/asf/hudi.git
from a5978cd230 [MINOR] Update release version to reflect published version
0.12.1
new ac0d1d81a4 [MINOR] Adapt to tianqiong spark
new 23412b2bee [MINOR] Add Zhiyan metrics reporter
new d785b41f01 fix cherry pick err
new ab5b4fa780 fix the bug, log file will not roll over to a new file
new 553bb9eab4 [HUDI-4475] fix create table with not exists hoodie
properties file
new c1ceb628e5 [MINOR] fix Invalid value for YearOfEra
new 1d029e668b add 'backup_invalid_parquet' procedure
new 6dbe53e623 fix zhiyan reporter for metadata
new f80900d91c [MINOR] Adapt to tianqiong spark
new 9c94e388fe adapt tspark changes: backport 3.3 VectorizedParquetReader
related code to 3.1
new e45564102b fix file not exists for getFileSize
new 895f260983 opt procedure backup_invalid_parquet
new 4e66857849 fix RowDataProjection with project and projectAsValues's NPE
new 8d692f38c1 [HUDI-5041] Fix lock metric register confict error (#6968)
new ee07cc6a3b Remove proxy
new 8ba01dc70a [HUDI-2624] Implement Non Index type for HUDI
new 97ce2b7f7b temp_view_support (#6990)
new 90c09053da [HUDI-5105] Add Call show_commit_extra_metadata for spark
sql (#7091)
new ecd39e3ad7 add log to print scanInternal's logFilePath
new 5f6d6ae42d remove hudi-kafka-connect module
new 3c364bdf72 [MINOR] add integrity check of merged parquet file for
HoodieMergeHandle.
new 738e2cce8f [HUDI-4898] presto/hive respect payload during merge
parquet file and logfile when reading mor table (#6741)
new 7d6654c1d0 [HUDI-5178] Add Call show_table_properties for spark sql
(#7161)
new e95a9f56ae [HUDI-4526] Improve spillableMapBasePath when disk
directory is full (#6284)
new 9fbf3b920d [HUDI-5095] Flink: Stores a special watermark(flag) to
identify the current progress of writing data
new f02fef936b fix none index partition format
new 005e913403 [HUDI-5095] Flink: Stores a special watermark(flag) to
identify the current progress of writing data
new 619b7504ca Reduce the scope and duration of holding checkpoint lock in
stream read
new f7fe437faf Fix tauth issue (merge request !102)
new 00c3443cb4 optimize schema settings
new ab5deef087 Merge branch 'optimize_schema_settings' into
'release-0.12.1' (merge request !108)
new c070e0963a [HUDI-5095] Flink: Stores a special watermark(flag) to
identify the current progress of writing data
new b1f204fa55 Merge branch 'release-0.12.1' of
https://git.woa.com/data-lake-technology/hudi into release-0.12.1
new f2256ec94c exclude hudi-kafka-connect & add some api to support
FLIP-27 source
new 0cf7d3dac8 fix database default error
new bddf061a79 [HUDI-5223] Partial failover for flink (#7208)
new adc8aa6ebd remove ZhiyanReporter's report print
new 0d71705ec2 [MINOR] add integrity check of parquet file for
HoodieRowDataParquetWriter.
new e0faa0bbe1 [HUDI-5350] Fix oom cause compaction event lost problem
(#7408)
new 5488c951e4 [HUDI-5314] add call help procedure (#7361)
new 4fe2aec44a fix read log not exist
new 79abc24265 improve checkstyle
new d0b3b36e96 check parquet file does not exist
new 4f005ea5d4 improve DropHoodieTableCommand
new 78fe5c73a4 [HUDI-3572] support DAY_ROLLING strategy in
ClusteringPlanPartitionFilterMode (#4966)
The 45 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
dev/settings.xml | 247 +++++++++++++
dev/tencent-install.sh | 158 ++++++++
dev/tencent-release.sh | 155 ++++++++
hudi-cli/pom.xml | 4 +-
hudi-client/hudi-client-common/pom.xml | 7 +
.../apache/hudi/async/AsyncPostEventService.java | 93 +++++
.../apache/hudi/client/BaseHoodieWriteClient.java | 28 +-
.../lock/metrics/HoodieLockMetrics.java | 19 +-
.../org/apache/hudi/config/HoodieIndexConfig.java | 43 +++
.../org/apache/hudi/config/HoodieMemoryConfig.java | 9 +-
.../org/apache/hudi/config/HoodieWriteConfig.java | 108 +++++-
.../hudi/config/metrics/HoodieMetricsConfig.java | 23 +-
.../config/metrics/HoodieMetricsZhiyanConfig.java | 143 +++++++
.../java/org/apache/hudi/index/HoodieIndex.java | 2 +-
.../java/org/apache/hudi/io/HoodieMergeHandle.java | 23 +-
.../src/main/java/org/apache/hudi/io/IOUtils.java | 14 +
.../apache/hudi/keygen/EmptyAvroKeyGenerator.java | 70 ++++
.../keygen/TimestampBasedAvroKeyGenerator.java | 4 +-
.../metadata/HoodieBackedTableMetadataWriter.java | 11 +
.../org/apache/hudi/metrics/HoodieMetrics.java | 120 +++++-
.../main/java/org/apache/hudi/metrics/Metrics.java | 1 +
.../hudi/metrics/MetricsReporterFactory.java | 4 +
.../apache/hudi/metrics/MetricsReporterType.java | 2 +-
.../hudi/metrics/zhiyan/ZhiyanHttpClient.java | 129 +++++++
.../hudi/metrics/zhiyan/ZhiyanMetricsReporter.java | 60 +++
.../apache/hudi/metrics/zhiyan/ZhiyanReporter.java | 169 +++++++++
.../cluster/ClusteringPlanPartitionFilter.java | 23 ++
.../cluster/ClusteringPlanPartitionFilterMode.java | 3 +-
.../hudi/table/action/commit/BucketInfo.java | 4 +
.../hudi/table/action/commit/BucketType.java | 2 +-
.../java/org/apache/hudi/tdbank/TDBankClient.java | 103 ++++++
.../java/org/apache/hudi/tdbank/TdbankConfig.java | 82 +++++
.../hudi/tdbank/TdbankHoodieMetricsEvent.java | 110 ++++++
.../apache/hudi/client/HoodieFlinkWriteClient.java | 6 +
.../apache/hudi/index/FlinkHoodieIndexFactory.java | 2 +
.../apache/hudi/index/FlinkHoodieNonIndex.java} | 47 ++-
.../java/org/apache/hudi/io/FlinkMergeHandle.java | 8 +-
.../io/storage/row/HoodieRowDataParquetWriter.java | 4 +
hudi-client/hudi-spark-client/pom.xml | 4 +-
.../apache/hudi/client/SparkRDDWriteClient.java | 22 ++
.../apache/hudi/index/SparkHoodieIndexFactory.java | 3 +
.../hudi/index/nonindex/SparkHoodieNonIndex.java} | 55 ++-
.../hudi/io/storage/row/HoodieRowCreateHandle.java | 5 +-
...eteKeyGenerator.java => EmptyKeyGenerator.java} | 53 ++-
.../commit/BaseSparkCommitActionExecutor.java | 17 +
.../table/action/commit/UpsertPartitioner.java | 35 +-
.../TestSparkClusteringPlanPartitionFilter.java | 29 ++
.../java/org/apache/hudi/avro/HoodieAvroUtils.java | 28 ++
.../java/org/apache/hudi/common/fs/FSUtils.java | 16 +-
.../org/apache/hudi/common/model/FileSlice.java | 13 +
.../org/apache/hudi/common/model/HoodieKey.java | 2 +
.../hudi/common/table/HoodieTableConfig.java | 6 +-
.../table/log/AbstractHoodieLogRecordReader.java | 41 ++-
.../common/table/log/HoodieLogFormatReader.java | 8 +-
.../common/table/log/HoodieLogFormatWriter.java | 9 +-
.../table/log/HoodieMergedLogRecordScanner.java | 12 +-
.../table/timeline/HoodieActiveTimeline.java | 18 +
.../hudi/common/table/timeline/TimelineUtils.java | 2 +-
.../org/apache/hudi/common/util/DateTimeUtils.java | 8 +
.../org/apache/hudi/common/util/FileIOUtils.java | 36 ++
.../metadata/FileSystemBackedTableMetadata.java | 2 +
hudi-examples/hudi-examples-spark/pom.xml | 4 +-
.../apache/hudi/configuration/FlinkOptions.java | 22 +-
.../apache/hudi/configuration/OptionsResolver.java | 4 +
.../org/apache/hudi/sink/StreamWriteFunction.java | 38 +-
.../hudi/sink/StreamWriteOperatorCoordinator.java | 39 ++
.../hudi/sink/append/AppendWriteFunction.java | 2 +-
.../hudi/sink/bulk/BulkInsertWriteFunction.java | 10 +-
.../sink/common/AbstractStreamWriteFunction.java | 23 +-
.../hudi/sink/common/AbstractWriteFunction.java | 103 ++++++
.../hudi/sink/common/AbstractWriteOperator.java | 9 +
.../apache/hudi/sink/event/WriteMetadataEvent.java | 31 +-
.../sink/nonindex/NonIndexStreamWriteFunction.java | 265 +++++++++++++
.../NonIndexStreamWriteOperator.java} | 12 +-
.../apache/hudi/sink/utils/NonThrownExecutor.java | 8 +-
.../java/org/apache/hudi/sink/utils/Pipelines.java | 7 +
.../hudi/source/StreamReadMonitoringFunction.java | 33 +-
.../apache/hudi/streamer/FlinkStreamerConfig.java | 4 +
.../apache/hudi/streamer/HoodieFlinkStreamer.java | 4 +-
.../org/apache/hudi/table/HoodieTableFactory.java | 21 ++
.../org/apache/hudi/table/HoodieTableSource.java | 42 ++-
.../table/format/mor/MergeOnReadInputFormat.java | 29 ++
.../table/format/mor/MergeOnReadInputSplit.java | 8 +-
.../java/org/apache/hudi/util/DataTypeUtils.java | 141 +++++++
.../java/org/apache/hudi/util/HoodiePipeline.java | 14 +
.../org/apache/hudi/util/RowDataProjection.java | 21 +-
.../apache/hudi/sink/ITTestDataStreamWrite.java | 35 ++
.../sink/TestWriteFunctionEventTimeExtract.java | 232 ++++++++++++
.../org/apache/hudi/sink/TestWriteMergeOnRead.java | 54 +++
.../hudi/sink/utils/InsertFunctionWrapper.java | 6 +
.../sink/utils/StreamWriteFunctionWrapper.java | 24 +-
.../apache/hudi/sink/utils/TestDataTypeUtils.java | 39 +-
.../hudi/sink/utils/TestFunctionWrapper.java | 6 +
.../org/apache/hudi/sink/utils/TestWriteBase.java | 48 +++
.../test/java/org/apache/hudi/utils/TestData.java | 34 ++
.../hudi/utils/source/ContinuousFileSource.java | 5 +
.../realtime/AbstractRealtimeRecordReader.java | 72 +++-
.../realtime/HoodieHFileRealtimeInputFormat.java | 2 +-
.../realtime/HoodieParquetRealtimeInputFormat.java | 14 +-
.../realtime/RealtimeCompactedRecordReader.java | 25 +-
.../hudi/hadoop/utils/HiveAvroSerializer.java | 409 +++++++++++++++++++++
.../utils/HoodieRealtimeInputFormatUtils.java | 19 +-
.../utils/HoodieRealtimeRecordReaderUtils.java | 5 +
.../hudi/hadoop/utils/TestHiveAvroSerializer.java | 148 ++++++++
hudi-integ-test/pom.xml | 4 +-
hudi-spark-datasource/hudi-spark-common/pom.xml | 12 +-
.../main/java/org/apache/hudi/DataSourceUtils.java | 15 +-
.../scala/org/apache/hudi/HoodieCLIUtils.scala | 2 +-
.../org/apache/hudi/HoodieSparkSqlWriter.scala | 12 +-
.../spark/sql/hudi/HoodieSqlCommonUtils.scala | 6 +-
.../AlterHoodieTableAddColumnsCommand.scala | 1 +
.../sql/hudi/command/DropHoodieTableCommand.scala | 13 +-
hudi-spark-datasource/hudi-spark/pom.xml | 12 +-
.../hudi/spark/sql/parser/HoodieSqlCommon.g4 | 6 +-
.../hudi/command/MergeIntoHoodieTableCommand.scala | 3 +-
...e.scala => BackupInvalidParquetProcedure.scala} | 37 +-
.../hudi/command/procedures/BaseProcedure.scala | 4 +-
...ToTableProcedure.scala => CopyToTempView.scala} | 60 ++-
.../hudi/command/procedures/HelpProcedure.scala | 125 +++++++
.../hudi/command/procedures/HoodieProcedures.scala | 9 +
...cala => ShowCommitExtraMetadataProcedure.scala} | 64 ++--
...re.scala => ShowTablePropertiesProcedure.scala} | 40 +-
.../sql/parser/HoodieSqlCommonAstBuilder.scala | 21 +-
.../java/org/apache/hudi/TestDataSourceUtils.java | 2 +-
.../org/apache/hudi/TestHoodieSparkSqlWriter.scala | 5 +-
.../test/scala/org/apache/hudi/TestNonIndex.scala | 110 ++++++
...ala => TestBackupInvalidParquetProcedure.scala} | 24 +-
.../sql/hudi/procedure/TestCommitsProcedure.scala | 54 ++-
.../procedure/TestCopyToTempViewProcedure.scala | 168 +++++++++
.../sql/hudi/procedure/TestHelpProcedure.scala | 84 +++++
...cala => TestShowTablePropertiesProcedure.scala} | 15 +-
hudi-spark-datasource/hudi-spark2/pom.xml | 12 +-
.../org/apache/hudi/internal/DefaultSource.java | 1 +
hudi-spark-datasource/hudi-spark3-common/pom.xml | 2 +-
.../apache/hudi/spark3/internal/DefaultSource.java | 4 +-
hudi-spark-datasource/hudi-spark3.1.x/pom.xml | 2 +-
.../datasources/Spark31NestedSchemaPruning.scala | 24 +-
.../parquet/Spark31HoodieParquetFileFormat.scala | 20 +-
.../hudi/command/Spark31AlterTableCommand.scala | 2 +-
hudi-spark-datasource/hudi-spark3.2.x/pom.xml | 6 +-
hudi-spark-datasource/hudi-spark3.3.x/pom.xml | 6 +-
hudi-sync/hudi-hive-sync/pom.xml | 4 +-
hudi-utilities/pom.xml | 10 +-
.../org/apache/hudi/utilities/UtilHelpers.java | 38 +-
packaging/hudi-integ-test-bundle/pom.xml | 8 +-
pom.xml | 130 ++++---
146 files changed, 5170 insertions(+), 542 deletions(-)
create mode 100644 dev/settings.xml
create mode 100644 dev/tencent-install.sh
create mode 100644 dev/tencent-release.sh
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncPostEventService.java
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/metrics/HoodieMetricsZhiyanConfig.java
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/EmptyAvroKeyGenerator.java
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/zhiyan/ZhiyanHttpClient.java
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/zhiyan/ZhiyanMetricsReporter.java
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/zhiyan/ZhiyanReporter.java
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/tdbank/TDBankClient.java
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/tdbank/TdbankConfig.java
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/tdbank/TdbankHoodieMetricsEvent.java
copy
hudi-client/{hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/JavaUpsertCommitActionExecutor.java
=>
hudi-flink-client/src/main/java/org/apache/hudi/index/FlinkHoodieNonIndex.java}
(52%)
copy
hudi-client/{hudi-flink-client/src/main/java/org/apache/hudi/index/state/FlinkInMemoryStateIndex.java
=>
hudi-spark-client/src/main/java/org/apache/hudi/index/nonindex/SparkHoodieNonIndex.java}
(51%)
copy
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/{GlobalDeleteKeyGenerator.java
=> EmptyKeyGenerator.java} (61%)
create mode 100644
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/nonindex/NonIndexStreamWriteFunction.java
copy
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/{bucket/BucketStreamWriteOperator.java
=> nonindex/NonIndexStreamWriteOperator.java} (74%)
create mode 100644
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteFunctionEventTimeExtract.java
copy hudi-cli/src/test/java/org/apache/hudi/cli/TestSparkUtil.java =>
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/TestDataTypeUtils.java
(50%)
create mode 100644
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java
create mode 100644
hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/utils/TestHiveAvroSerializer.java
copy
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/{ShowInvalidParquetProcedure.scala
=> BackupInvalidParquetProcedure.scala} (66%)
copy
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/{CopyToTableProcedure.scala
=> CopyToTempView.scala} (74%)
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/HelpProcedure.scala
copy
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/{ShowCommitWriteStatsProcedure.scala
=> ShowCommitExtraMetadataProcedure.scala} (69%)
copy
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/{ShowSavepointsProcedure.scala
=> ShowTablePropertiesProcedure.scala} (58%)
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestNonIndex.scala
copy
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/{TestShowInvalidParquetProcedure.scala
=> TestBackupInvalidParquetProcedure.scala} (78%)
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestCopyToTempViewProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestHelpProcedure.scala
copy
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/{TestShowFsPathDetailProcedure.scala
=> TestShowTablePropertiesProcedure.scala} (72%)
