This is an automated email from the ASF dual-hosted git repository.
xushiyan pushed a change to branch release-feature-rfc46
in repository https://gitbox.apache.org/repos/asf/hudi.git
omit 2100a52154 [HUDI-4292][RFC-46] Update doc to align with the Record
Merge API changes (#5927)
omit bf4ef73a6b [MINOR] Fix type casting in TestHoodieHFileReaderWriter
omit 23faed86c1 [HUDI-3378][HUDI-3379][HUDI-3381] Migrate usage of
HoodieRecordPayload and raw Avro payload to HoodieRecord (#5522)
add 7b946cf351 [HUDI-3499] Add Call Procedure for show rollbacks (#5848)
add c291b05699 [HUDI-4218] [HUDI-4218] Expose the real exception
information when an exception occurs in the tableExists method (#5827)
add 2bf0a1906d [HUDI-4217] improve repeat init object in ExpressionPayload
(#5825)
add 0ff34b6974 [HUDI-4214] improve repeat init write schema in
ExpressionPayload (#5820)
add 7689e62cd9 [HUDI-4265] Deprecate useless targetTableName parameter in
HoodieMultiTableDeltaStreamer (#5883)
add fec49dc12b [HUDI-4165] Support Create/Drop/Show/Refresh Index Syntax
for Spark SQL (#5761)
add c5c4cfec91 [HUDI-3507] Support export command based on Call Produce
Command (#5901)
add d4f0326b4b [HUDI-4275] Refactor rollback inflight instant for
clustering/compaction to reuse some code (#5894)
add efafb79eeb [MINOR] Add "spillable_map_path" in FlinkCompactionConfig.
To avoid the disk space of "/tmp" full when compacting offline. (#5905)
add 7481eacf23 [HUDI-4277] supoort flink table source with computed column
(#5897)
add d7facb8cb8 fix remove redundant Variable (#5806)
add 7c6bedff25 [HUDI-4259] Flink create avro schema not conformance to
standards (#5878)
add f1103281d2 [HUDI-4258] Fix when HoodieTable removes data file before
the end of Flink job (#5876)
add 7601e9e4c7 [MINOR] Update DOAP with 0.11.1 Release (#5908)
add 17ac5a4573 [HUDI-4173] Fix wrong results if the user read no base
files hudi table by glob paths (#5723)
add ba4d5bd847 [HUDI-4251] Fix the problem that the command 'commits sync'
description does not match. (#5881)
add 5c204f1416 [HUDI-4177] Fix hudi-cli rollback with rollbackUsingMarkers
method call (#5734)
add 7c4aaa9715 [HUDI-4270] Bootstrap op data loading missing (#5888)
add 4640a3bbb8 [HUDI-3475] Initialize hudi table management module.
add 092e35c1e3 udate
add c7e430bb46 Revert master (#5925)
add c9590790f8 [HUDI-4279] Strength the remote fs view lagging check when
latest commit refresh is enabled (#5917)
add 1dbd9d407a [minor] following 4270, add unit tests for the keys lost
case (#5918)
add 1bb017d396 [HUDI-3508] Add call procedure for FileSystemViewCommand
(#5929)
add 38ff18a199 [HUDI-4299] Fix problem about hudi-example-java run failed
on idea. (#5936)
add eeb78f23e6 [HUDI-4290] Fix fetchLatestBaseFiles to filter replaced
filegroups (#5941)
add af9f09047d [HUDI-3509] Add call procedure for HoodieLogFileCommand
(#5949)
add 6456bd3a51 [HUDI-4273] Support inline schedule clustering for Flink
stream (#5890)
add 30ebdc708b [HUDI-3735] TestHoodieSparkMergeOnReadTableRollback is
flaky (#5874)
add 59978ef4a9 [HUDI-4260] Change KEYGEN_CLASS_NAME without default value
(#5877)
add eeafaeacd2 [HUDI-3512] Add call procedure for StatsCommand (#5955)
add e8fbd4daf4 [TEST][DO_NOT_MERGE]fix random failed for ci (#5948)
add 360df576a9 Revert "[TEST][DO_NOT_MERGE]fix random failed for ci
(#5948)" (#5971)
add c86edfc28e [HUDI-4319] Fixed Parquet's `PLAIN_DICTIONARY` encoding not
being applied when bulk-inserting (#5966)
add 142adf4ccb [HUDI-4296] Fix the bug that
TestHoodieSparkSqlWriter.testSchemaEvolutionForTableType is flaky (#5973)
add 1c43c590ac [HUDI-3502] Support hdfs parquet import command based on
Call Produce Command (#5956)
add 559b26fb7c [MINOR] Remove -T option from CI build (#5972)
add 0a9e568ff5 [HUDI-5246] Bumping mysql connector version due to security
vulnerability (#5851)
add 7a6eb0f6e1 [HUDI-4309] Spark3.2 custom parser should not throw
exception (#5947)
add 72fa19bcc9 [HUDI-4316] Support for spillable diskmap configuration
when constructing HoodieMergedLogRecordScanner (#5959)
add 8f4e2a189e [HUDI-4315] Do not throw exception in
BaseSpark3Adapter#toTableIdentifier (#5957)
add 26c967bac6 [HUDI-3504] Support bootstrap command based on Call Produce
Command (#5977)
add 3a1fd22841 [HUDI-4311] Fix Flink lose data on some rollback scene
(#5950)
add 8846849a03 [HUDI-4291] Fix flaky
TestCleanPlanExecutor#testKeepLatestFileVersions (#5930)
add b14ed47f21 [HUDI-3506] Add call procedure for CommitsCommand (#5974)
add 09dc001430 [HUDI-4325] fix spark sql procedure cause ParseException
with semicolon (#5982)
add 08eba914ed [HUDI-4333] fix HoodieFileIndex's listFiles method log
print skipping percent NaN (#5990)
add efb9719018 [HUDI-4332] The current instant may be wrong under some
extreme conditions in AppendWriteFunction. (#5988)
new a0969fc451 [HUDI-3378][HUDI-3379][HUDI-3381] Migrate usage of
HoodieRecordPayload and raw Avro payload to HoodieRecord (#5522)
new 28306ff797 [MINOR] Fix type casting in TestHoodieHFileReaderWriter
new 22976d05a4 [HUDI-4292][RFC-46] Update doc to align with the Record
Merge API changes (#5927)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (2100a52154)
\
N -- N -- N refs/heads/release-feature-rfc46 (22976d05a4)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
azure-pipelines.yml | 2 +-
doap_HUDI.rdf | 5 +
.../apache/hudi/cli/commands/CommitsCommand.java | 4 +-
.../apache/hudi/cli/commands/ExportCommand.java | 34 +-
.../org/apache/hudi/cli/commands/SparkMain.java | 2 +-
.../scala/org/apache/hudi/cli/SparkHelpers.scala | 4 +-
.../hudi/cli/integ/ITTestCommitsCommand.java | 17 +
.../apache/hudi/client/BaseHoodieWriteClient.java | 11 +-
.../apache/hudi/client/CompactionAdminClient.java | 7 +-
.../hudi/io/storage/HoodieAvroParquetWriter.java | 4 +-
.../hudi/io/storage/HoodieBaseParquetWriter.java | 2 +-
.../hudi/io/storage/HoodieFileWriterFactory.java | 2 +-
.../java/org/apache/hudi/table/HoodieTable.java | 41 ++-
.../cluster/strategy/ClusteringPlanStrategy.java | 13 +-
.../compact/ScheduleCompactionActionExecutor.java | 2 +-
.../hudi/testutils/HoodieWriteableTestTable.java | 19 +-
.../FlinkRecentDaysClusteringPlanStrategy.java | 65 ----
...nkSelectedPartitionsClusteringPlanStrategy.java | 67 ----
.../row/HoodieRowDataFileWriterFactory.java | 3 +-
.../io/storage/row/HoodieRowDataParquetConfig.java | 36 ---
.../io/storage/row/HoodieRowDataParquetWriter.java | 3 +-
.../run/strategy/JavaExecutionStrategy.java | 2 +
.../apache/hudi/client/SparkRDDWriteClient.java | 2 +-
.../MultipleSparkJobExecutionStrategy.java | 2 +
.../row/HoodieInternalRowFileWriterFactory.java | 14 +-
.../row/HoodieInternalRowParquetWriter.java | 3 +-
.../io/storage/row/HoodieRowParquetConfig.java | 36 ---
.../org/apache/spark/sql/hudi/SparkAdapter.scala | 7 +-
.../row/TestHoodieInternalRowParquetWriter.java | 5 +-
.../table/functional/TestCleanPlanExecutor.java | 172 ++++++----
.../TestHoodieSparkMergeOnReadTableRollback.java | 14 +-
.../org/apache/hudi/common/index/HoodieIndex.java | 119 +++++++
.../apache/hudi/common/index/HoodieIndexType.java | 54 ++++
.../hudi/common/table/log/HoodieLogFileReader.java | 2 +-
.../table/log/block/HoodieParquetDataBlock.java | 22 +-
.../table/timeline/HoodieActiveTimeline.java | 36 +--
.../table/view/AbstractTableFileSystemView.java | 20 +-
.../view/RemoteHoodieTableFileSystemView.java | 2 +
.../hudi/io/storage/HoodieAvroParquetConfig.java | 42 ---
...ParquetConfig.java => HoodieParquetConfig.java} | 10 +-
.../hudi/io/storage/HoodieParquetStreamWriter.java | 2 +-
.../table/timeline/TestHoodieActiveTimeline.java | 4 +-
.../view/TestHoodieTableFSViewWithClustering.java | 181 +++++++++++
.../table/view/TestHoodieTableFileSystemView.java | 6 +-
hudi-examples/hudi-examples-java/pom.xml | 18 +-
.../apache/hudi/configuration/FlinkOptions.java | 30 +-
.../apache/hudi/configuration/OptionsResolver.java | 51 ++-
.../hudi/sink/StreamWriteOperatorCoordinator.java | 15 +-
.../hudi/sink/append/AppendWriteFunction.java | 15 +-
.../hudi/sink/bootstrap/BootstrapOperator.java | 2 +
.../org/apache/hudi/sink/bulk/RowDataKeyGen.java | 2 +-
.../sink/clustering/ClusteringCommitEvent.java | 10 +-
.../hudi/sink/clustering/ClusteringCommitSink.java | 28 +-
.../hudi/sink/clustering/ClusteringOperator.java | 167 ++++++----
.../sink/clustering/ClusteringPlanOperator.java | 139 +++++++++
.../clustering/ClusteringPlanSourceFunction.java | 13 +-
.../sink/clustering/FlinkClusteringConfig.java | 30 +-
.../sink/clustering/HoodieFlinkClusteringJob.java | 346 +++++++++++++++------
.../apache/hudi/sink/compact/CompactFunction.java | 3 +-
.../hudi/sink/compact/CompactionCommitSink.java | 2 +-
.../hudi/sink/compact/FlinkCompactionConfig.java | 6 +
.../apache/hudi/sink/event/WriteMetadataEvent.java | 12 +
.../org/apache/hudi/sink/meta/CkpMetadata.java | 2 -
.../apache/hudi/sink/utils/NonThrownExecutor.java | 68 ++--
.../java/org/apache/hudi/sink/utils/Pipelines.java | 63 +++-
.../apache/hudi/streamer/HoodieFlinkStreamer.java | 3 +-
.../org/apache/hudi/table/HoodieTableSink.java | 10 +-
.../org/apache/hudi/table/HoodieTableSource.java | 2 +-
.../org/apache/hudi/table/format/FormatUtils.java | 1 +
.../org/apache/hudi/util/AvroSchemaConverter.java | 4 +-
.../java/org/apache/hudi/util/ClusteringUtil.java | 80 +++++
.../java/org/apache/hudi/util/StreamerUtil.java | 23 +-
.../apache/hudi/sink/ITTestDataStreamWrite.java | 74 +++++
.../sink/TestStreamWriteOperatorCoordinator.java | 38 +++
.../org/apache/hudi/sink/TestWriteCopyOnWrite.java | 29 +-
.../org/apache/hudi/sink/TestWriteMergeOnRead.java | 25 ++
.../sink/cluster/ITTestHoodieFlinkClustering.java | 45 ++-
.../hudi/sink/utils/ClusteringFunctionWrapper.java | 121 +++++++
.../hudi/sink/utils/InsertFunctionWrapper.java | 34 +-
.../sink/utils/StreamWriteFunctionWrapper.java | 5 +-
.../org/apache/hudi/sink/utils/TestWriteBase.java | 7 +-
.../apache/hudi/table/TestHoodieTableFactory.java | 8 +-
.../test/java/org/apache/hudi/utils/TestData.java | 19 ++
.../test/java/org/apache/hudi/utils/TestUtils.java | 10 +
.../org/apache/hudi/BaseFileOnlyRelation.scala | 7 +-
.../scala/org/apache/hudi/HoodieFileIndex.scala | 8 +-
.../org/apache/hudi/HoodieMergeOnReadRDD.scala | 9 +-
.../apache/hudi/MergeOnReadSnapshotRelation.scala | 30 +-
.../datasources/HoodieInMemoryFileIndex.scala | 75 ++++-
.../sql/hudi/command/payload/SqlTypedRecord.scala | 44 ++-
.../hudi/spark/sql/parser/HoodieSqlCommon.g4 | 78 ++++-
.../apache/hudi/cli/BootstrapExecutorUtils.java | 132 +++++---
.../apache/hudi/cli/HDFSParquetImporterUtils.java | 325 +++++++++++++++++++
.../java/org/apache/hudi/cli}/SchemaProvider.java | 5 +-
.../spark/sql/catalyst/plans/logical/Index.scala | 111 +++++++
.../spark/sql/hudi/analysis/HoodieAnalysis.scala | 22 ++
.../spark/sql/hudi/command/IndexCommands.scala | 101 ++++++
.../hudi/command/payload/ExpressionPayload.scala | 58 ++--
.../procedures/CommitsCompareProcedure.scala | 81 +++++
.../procedures/ExportInstantsProcedure.scala | 239 ++++++++++++++
.../procedures/HdfsParquetImportProcedure.scala | 85 +++++
.../hudi/command/procedures/HoodieProcedures.scala | 19 ++
.../command/procedures/RunBootstrapProcedure.scala | 144 +++++++++
...re.scala => ShowArchivedCommitsProcedure.scala} | 58 ++--
.../procedures/ShowBootstrapMappingProcedure.scala | 117 +++++++
...cala => ShowBootstrapPartitionsProcedure.scala} | 36 ++-
.../procedures/ShowCommitFilesProcedure.scala | 123 ++++++++
.../procedures/ShowCommitPartitionsProcedure.scala | 140 +++++++++
.../procedures/ShowCommitWriteStatsProcedure.scala | 115 +++++++
.../procedures/ShowFileSystemViewProcedure.scala | 258 +++++++++++++++
.../ShowHoodieLogFileMetadataProcedure.scala | 139 +++++++++
.../ShowHoodieLogFileRecordsProcedure.scala | 125 ++++++++
.../procedures/ShowRollbacksProcedure.scala | 147 +++++++++
.../procedures/StatsFileSizeProcedure.scala | 108 +++++++
.../StatsWriteAmplificationProcedure.scala | 85 +++++
.../sql/parser/HoodieSqlCommonAstBuilder.scala | 143 ++++++++-
.../apache/hudi/functional/TestMORDataSource.scala | 2 +
.../apache/spark/sql/hudi/TestInsertTable.scala | 44 +++
.../spark/sql/hudi/TestTimeTravelTable.scala | 12 +
.../sql/hudi/command/index/TestIndexSyntax.scala | 85 +++++
.../hudi/procedure/TestBootstrapProcedure.scala | 89 ++++++
.../sql/hudi/procedure/TestCallCommandParser.scala | 20 ++
.../sql/hudi/procedure/TestCallProcedure.scala | 105 +++++++
.../sql/hudi/procedure/TestCommitsProcedure.scala | 293 +++++++++++++++++
.../procedure/TestExportInstantsProcedure.scala | 52 ++++
.../sql/hudi/procedure/TestFsViewProcedure.scala | 95 ++++++
.../procedure/TestHdfsParquetImportProcedure.scala | 202 ++++++++++++
.../procedure/TestHoodieLogFileProcedure.scala | 103 ++++++
.../sql/hudi/procedure/TestStatsProcedure.scala | 99 ++++++
.../apache/spark/sql/adapter/Spark2Adapter.scala | 6 +-
.../spark/sql/adapter/BaseSpark3Adapter.scala | 23 +-
.../parser/HoodieSpark3_2ExtendedSqlParser.scala | 13 +-
hudi-sync/hudi-adb-sync/pom.xml | 2 +-
.../hudi/timeline/service/RequestHandler.java | 22 +-
.../org/apache/hudi/utilities/UtilHelpers.java | 2 +-
.../HoodieMultiTableDeltaStreamer.java | 17 +-
136 files changed, 6255 insertions(+), 858 deletions(-)
delete mode 100644
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/clustering/plan/strategy/FlinkRecentDaysClusteringPlanStrategy.java
delete mode 100644
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/clustering/plan/strategy/FlinkSelectedPartitionsClusteringPlanStrategy.java
delete mode 100644
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataParquetConfig.java
delete mode 100644
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetConfig.java
create mode 100644
hudi-common/src/main/java/org/apache/hudi/common/index/HoodieIndex.java
create mode 100644
hudi-common/src/main/java/org/apache/hudi/common/index/HoodieIndexType.java
delete mode 100644
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieAvroParquetConfig.java
rename
hudi-common/src/main/java/org/apache/hudi/io/storage/{HoodieBaseParquetConfig.java
=> HoodieParquetConfig.java} (82%)
create mode 100644
hudi-common/src/test/java/org/apache/hudi/common/table/view/TestHoodieTableFSViewWithClustering.java
create mode 100644
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringPlanOperator.java
create mode 100644
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/ClusteringUtil.java
create mode 100644
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/ClusteringFunctionWrapper.java
copy
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/BootstrapExecutor.java
=>
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java
(64%)
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/HDFSParquetImporterUtils.java
copy {hudi-utilities/src/main/java/org/apache/hudi/utilities/schema =>
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli}/SchemaProvider.java
(97%)
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Index.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/IndexCommands.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/CommitsCompareProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ExportInstantsProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/HdfsParquetImportProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/RunBootstrapProcedure.scala
copy
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/{ShowCommitsProcedure.scala
=> ShowArchivedCommitsProcedure.scala} (79%)
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowBootstrapMappingProcedure.scala
copy
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/{ShowSavepointsProcedure.scala
=> ShowBootstrapPartitionsProcedure.scala} (64%)
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowCommitFilesProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowCommitPartitionsProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowCommitWriteStatsProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowFileSystemViewProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowHoodieLogFileMetadataProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowHoodieLogFileRecordsProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowRollbacksProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/StatsFileSizeProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/StatsWriteAmplificationProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestIndexSyntax.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestBootstrapProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestCommitsProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestExportInstantsProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestFsViewProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestHdfsParquetImportProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestHoodieLogFileProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestStatsProcedure.scala
