This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch hudi_test_suite_refactor in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git.
omit 09c34a0 [HUDI-442] Fix TestComplexKeyGenerator#testSingleValueKeyGenerator and testMultipleValueKeyGenerator NPE omit 66463ff [MINOR] Fix compile error about the deletion of HoodieActiveTimeline#createNewCommitTime omit 1d2ecbc [HUDI-391] Rename module name from hudi-bench to hudi-test-suite and fix some checkstyle issues (#1102) omit 9b55d37 [HUDI-394] Provide a basic implementation of test suite add 8172197 Fix Error: java.lang.IllegalArgumentException: Can not create a Path from an empty string in HoodieCopyOnWrite#deleteFilesFunc (#1126) add 4b1b3fc [MINOR] Set info servity for ImportOrder temporarily (#1127) add 41f3677 [MINOR] fix typo add dd06660 [MINOR] fix typo add 94aec96 [minor] Fix few typos in the java docs (#1132) add 9c4217a [HUDI-389] Fixing Index look up to return right partitions for a given key along with fileId with Global Bloom (#1091) add 8affdf8 [HUDI-416] Improve hint information for cli (#1110) add 3c811ec [MINOR] fix typos add def18a5 [MINOR] optimize hudi timeline service (#1137) add 842eabb [HUDI-470] Fix NPE when print result via hudi-cli (#1138) add f20a130 [MINOR] typo fix (#1142) add 01c25d6 [MINOR] Update the java doc of HoodieTableType (#1148) add 58c5bed [HUDI-453] Fix throw failed to archive commits error when writing data to MOR/COW table add 179837e Fix checkstyle add 2f25416 Skip setting commit metadata add 8440482 Fix empty content clean plan add e4ea7a2 Update comment add 2a823f3 [MINOR]: alter some wrong params which bring fatal exception add ab6ae5c [HUDI-482] Fix missing @Override annotation on methods (#1156) add e637d9e [HUDI-455] Redo hudi-client log statements using SLF4J (#1145) add bb90ded [MINOR] Fix out of limits for results add 36c0e6b [MINOR] Fix out of limits for results add 74b00d1 trigger rebuild add 619f501 Clean up code add add4b1e Merge pull request #1143 from BigDataArtisans/outoflimit add 47c1f74 [HUDI-343]: Create a DOAP file for Hudi add 98c0d8c Merge pull request #1160 from smarthi/HUDI-343 add dde21e7 [HUDI-402]: code clean up in test cases add e1e5fe3 [MINOR] Fix error usage of String.format (#1169) add ff1113f [HUDI-492]Fix show env all in hudi-cli add 290278f [HUDI-118]: Options provided for passing properties to Cleaner, compactor and importer commands add a733f4e [MINOR] Optimize hudi-cli module (#1136) add 726ae47 [MINOR]Optimize hudi-client module (#1139) add 7031445 [HUDI-377] Adding Delete() support to DeltaStreamer (#1073) add 28ccf8c [HUDI-484] Fix NPE when reading IncrementalPull.sqltemplate in HiveIncrementalPuller (#1167) add b9fab0b Revert "[HUDI-455] Redo hudi-client log statements using SLF4J (#1145)" (#1181) add 2d5b79d [HUDI-438] Merge duplicated code fragment in HoodieSparkSqlWriter (#1114) add 8f935e7 [HUDI-406]: added default partition path in TimestampBasedKeyGenerator add c78092d [HUDI-501] Execute docker/setup_demo.sh in any directory add 75c3f63 [HUDI-405] Remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource add b5df672 [HUDI-464] Use Hive Exec Core for tests (#1125) add 8306f74 [HUDI-417] Refactor HoodieWriteClient so that commit logic can be shareable by both bootstrap and normal write operations (#1166) add 9706f65 [HUDI-508] Standardizing on "Table" instead of "Dataset" across code (#1197) add 9884972 [MINOR] Remove old jekyll config file (#1198) add aba8387 Update deprecated HBase API add 480fc78 [HUDI-319] Add a new maven profile to generate unified Javadoc for all Java and Scala classes (#1195) add d09eacd [HUDI-25] Optimize HoodieInputformat.listStatus() for faster Hive incremental queries on Hoodie add 5af3dc6 [HUDI-331]Fix java docs for all public apis in HoodieWriteClient (#1111) add 3c90d25 [HUDI-114]: added option to overwrite payload implementation in hoodie.properties file add 04afac9 [HUDI-248] CLI doesn't allow rolling back a Delta commit add b95367d [HUDI-469] Fix: HoodieCommitMetadata only show first commit insert rows. add e103165 [CLEAN] replace utf-8 constant with StandardCharsets.UTF_8 add 017ee8e [MINOR] Fix partition typo (#1209) add d9675c4 [HUDI-522] Use the same version jcommander uniformly (#1214) add ad50008 [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types add 971c7d4 [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits for MOR tables add a44c61b [HUDI-502] provide a custom time zone definition for TimestampBasedKeyGenerator (#1188) add c1f8aca [HUDI-526] fix the HoodieAppendHandle add fd8f1c7 [MINOR] Reuse random object (#1222) add d5549f5 [HUDI-394] Provide a basic implementation of test suite add 53e73e5 [HUDI-391] Rename module name from hudi-bench to hudi-test-suite and fix some checkstyle issues (#1102) add f8d25b1 [MINOR] Fix compile error about the deletion of HoodieActiveTimeline#createNewCommitTime add 3dc85eb [HUDI-442] Fix TestComplexKeyGenerator#testSingleValueKeyGenerator and testMultipleValueKeyGenerator NPE This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (09c34a0) \ N -- N -- N refs/heads/hudi_test_suite_refactor (3dc85eb) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: LICENSE | 22 +- README.md | 6 + _config.yml | 16 - doap_HUDI.rdf | 59 +++ docker/setup_demo.sh | 9 +- docker/stop_demo.sh | 5 +- .../main/java/org/apache/hudi/cli/HoodieCLI.java | 17 +- .../hudi/cli/HoodieHistoryFileNameProvider.java | 1 + .../org/apache/hudi/cli/HoodiePrintHelper.java | 2 +- .../java/org/apache/hudi/cli/HoodiePrompt.java | 2 +- .../org/apache/hudi/cli/HoodieSplashScreen.java | 3 + .../src/main/java/org/apache/hudi/cli/Table.java | 38 +- .../hudi/cli/commands/ArchivedCommitsCommand.java | 63 +-- .../apache/hudi/cli/commands/CleansCommand.java | 72 +-- .../apache/hudi/cli/commands/CommitsCommand.java | 80 +--- .../hudi/cli/commands/CompactionCommand.java | 353 ++++++++------- .../hudi/cli/commands/FileSystemViewCommand.java | 24 +- .../cli/commands/HDFSParquetImportCommand.java | 28 +- .../hudi/cli/commands/HoodieLogFileCommand.java | 59 ++- .../hudi/cli/commands/HoodieSyncCommand.java | 57 ++- .../apache/hudi/cli/commands/RepairsCommand.java | 27 +- .../apache/hudi/cli/commands/RollbacksCommand.java | 30 +- .../hudi/cli/commands/SavepointsCommand.java | 45 +- .../apache/hudi/cli/commands/SparkEnvCommand.java | 2 +- .../org/apache/hudi/cli/commands/SparkMain.java | 88 +++- .../org/apache/hudi/cli/commands/StatsCommand.java | 18 +- .../{DatasetsCommand.java => TableCommand.java} | 49 +- .../java/org/apache/hudi/cli/utils/SparkUtil.java | 12 +- hudi-client/pom.xml | 8 +- .../java/org/apache/hudi/AbstractHoodieClient.java | 1 + .../org/apache/hudi/AbstractHoodieWriteClient.java | 421 ++++++++++++++++++ .../org/apache/hudi/CompactionAdminClient.java | 7 +- .../java/org/apache/hudi/HoodieReadClient.java | 6 +- .../java/org/apache/hudi/HoodieWriteClient.java | 494 +++++---------------- .../apache/hudi/config/HoodieCompactionConfig.java | 6 +- .../apache/hudi/config/HoodieHBaseIndexConfig.java | 19 +- .../org/apache/hudi/config/HoodieIndexConfig.java | 5 +- .../org/apache/hudi/config/HoodieMemoryConfig.java | 9 +- .../apache/hudi/config/HoodieMetricsConfig.java | 5 +- .../apache/hudi/config/HoodieStorageConfig.java | 5 +- .../org/apache/hudi/config/HoodieWriteConfig.java | 21 +- .../hudi/func/SparkBoundedInMemoryExecutor.java | 1 + .../hudi/index/bloom/BloomIndexFileInfo.java | 3 + .../apache/hudi/index/bloom/HoodieBloomIndex.java | 14 +- .../index/bloom/HoodieBloomIndexCheckFunction.java | 6 +- .../hudi/index/bloom/HoodieGlobalBloomIndex.java | 44 +- .../apache/hudi/index/bloom/IndexFileFilter.java | 11 +- .../IntervalTreeBasedGlobalIndexFileFilter.java | 26 +- .../bloom/IntervalTreeBasedIndexFileFilter.java | 12 +- .../org/apache/hudi/index/bloom/KeyRangeNode.java | 4 +- .../bloom/ListBasedGlobalIndexFileFilter.java | 19 +- .../hudi/index/bloom/ListBasedIndexFileFilter.java | 8 +- .../org/apache/hudi/index/hbase/HBaseIndex.java | 75 ++-- .../org/apache/hudi/io/HoodieAppendHandle.java | 7 +- .../java/org/apache/hudi/io/HoodieCleanHelper.java | 34 +- .../org/apache/hudi/io/HoodieCommitArchiveLog.java | 13 +- .../org/apache/hudi/io/HoodieCreateHandle.java | 1 + .../java/org/apache/hudi/io/HoodieMergeHandle.java | 6 + .../io/compact/HoodieRealtimeTableCompactor.java | 4 +- .../BoundedPartitionAwareCompactionStrategy.java | 15 +- .../strategy/DayBasedCompactionStrategy.java | 6 +- .../LogFileSizeBasedCompactionStrategy.java | 2 +- .../hudi/io/storage/HoodieParquetWriter.java | 1 + .../org/apache/hudi/metrics/HoodieMetrics.java | 4 +- .../apache/hudi/metrics/JmxMetricsReporter.java | 8 +- .../apache/hudi/table/HoodieCopyOnWriteTable.java | 20 +- .../apache/hudi/table/HoodieMergeOnReadTable.java | 35 +- .../java/org/apache/hudi/table/HoodieTable.java | 2 +- .../org/apache/hudi/table/RollbackExecutor.java | 22 +- hudi-client/src/test/java/HoodieClientExample.java | 6 +- .../org/apache/hudi/HoodieClientTestHarness.java | 6 +- .../src/test/java/org/apache/hudi/TestCleaner.java | 65 +-- .../java/org/apache/hudi/TestClientRollback.java | 6 +- .../org/apache/hudi/TestCompactionAdminClient.java | 49 +- .../java/org/apache/hudi/TestHoodieClientBase.java | 225 +++++----- .../hudi/TestHoodieClientOnCopyOnWriteStorage.java | 82 +++- .../java/org/apache/hudi/TestHoodieReadClient.java | 7 +- .../test/java/org/apache/hudi/TestWriteStatus.java | 4 +- .../apache/hudi/common/HoodieClientTestUtils.java | 8 +- .../hudi/common/HoodieMergeOnReadTestUtils.java | 3 +- .../hudi/common/HoodieTestDataGenerator.java | 124 ++++-- .../hudi/func/TestBoundedInMemoryExecutor.java | 2 +- .../apache/hudi/func/TestBoundedInMemoryQueue.java | 2 +- .../apache/hudi/func/TestUpdateMapFunction.java | 2 - .../java/org/apache/hudi/index/TestHbaseIndex.java | 28 +- .../org/apache/hudi/index/TestHoodieIndex.java | 7 +- .../bloom/TestBucketizedBloomCheckPartitioner.java | 4 +- .../hudi/index/bloom/TestHoodieBloomIndex.java | 5 +- .../index/bloom/TestHoodieGlobalBloomIndex.java | 15 +- .../hudi/index/bloom/TestKeyRangeLookupTree.java | 3 +- .../apache/hudi/io/TestHoodieCommitArchiveLog.java | 8 +- .../org/apache/hudi/io/TestHoodieMergeHandle.java | 2 - .../io/strategy/TestHoodieCompactionStrategy.java | 9 +- .../apache/hudi/metrics/TestHoodieJmxMetrics.java | 6 +- .../org/apache/hudi/metrics/TestHoodieMetrics.java | 4 +- .../apache/hudi/table/TestCopyOnWriteTable.java | 11 +- .../apache/hudi/table/TestMergeOnReadTable.java | 135 ++---- .../apache/hudi/common/model}/BaseAvroPayload.java | 13 +- .../hudi/common/model/HoodieAvroPayload.java | 2 +- .../hudi/common/model/HoodieCommitMetadata.java | 7 +- .../hudi/common/model/HoodieRecordPayload.java | 2 +- .../common/model/HoodieRollingStatMetadata.java | 2 +- .../apache/hudi/common/model/HoodieTableType.java | 6 +- .../model}/OverwriteWithLatestAvroPayload.java | 24 +- .../hudi/common/table/HoodieTableConfig.java | 24 +- .../hudi/common/table/HoodieTableMetaClient.java | 43 +- .../apache/hudi/common/table/HoodieTimeline.java | 2 +- .../hudi/common/table/log/HoodieLogFileReader.java | 1 + .../common/table/log/HoodieLogFormatWriter.java | 2 + .../table/timeline/HoodieActiveTimeline.java | 10 +- .../table/timeline/HoodieArchivedTimeline.java | 2 +- .../hudi/common/table/timeline/HoodieInstant.java | 2 +- .../table/view/AbstractTableFileSystemView.java | 5 +- .../common/table/view/FileSystemViewManager.java | 22 +- .../table/view/HoodieTableFileSystemView.java | 2 + .../view/RemoteHoodieTableFileSystemView.java | 4 +- .../table/view/RocksDbBasedFileSystemView.java | 4 +- .../view/SpillableMapBasedFileSystemView.java | 2 + .../apache/hudi/common/util/HoodieAvroUtils.java | 5 +- .../hudi/common/util/ObjectSizeCalculator.java | 1 + .../org/apache/hudi/common/util/RocksDBDAO.java | 2 +- .../hudi/common/util/RocksDBSchemaHelper.java | 2 +- .../hudi/common/util/collection/DiskBasedMap.java | 1 + .../common/util/collection/LazyFileIterable.java | 1 + .../apache/hudi/exception/HoodieIOException.java | 2 +- ...etException.java => InvalidTableException.java} | 8 +- ...dException.java => TableNotFoundException.java} | 20 +- .../hudi/common/bloom/filter/TestBloomFilter.java | 9 +- .../hudi/common/minicluster/HdfsTestService.java | 23 +- .../apache/hudi/common/model/HoodieTestUtils.java | 44 +- .../common/model/TestHoodieCommitMetadata.java | 4 +- .../hudi/common/model/TestHoodieWriteStat.java | 2 - .../common/table/TestHoodieTableMetaClient.java | 4 +- .../hudi/common/table/log/TestHoodieLogFormat.java | 29 +- .../log/TestHoodieLogFormatAppendFailure.java | 2 +- .../common/table/string/MockHoodieTimeline.java | 11 +- .../table/string/TestHoodieActiveTimeline.java | 7 +- .../table/view/TestHoodieTableFileSystemView.java | 51 +-- .../table/view/TestIncrementalFSViewSync.java | 73 ++- .../hudi/common/util/CompactionTestUtils.java | 2 +- .../apache/hudi/common/util/SchemaTestUtil.java | 8 +- .../hudi/common/util/SpillableMapTestUtils.java | 2 +- .../util/TestDFSPropertiesConfiguration.java | 12 +- .../org/apache/hudi/common/util/TestFSUtils.java | 27 +- .../hudi/common/util/TestHoodieAvroUtils.java | 6 +- .../apache/hudi/common/util/TestNumericUtils.java | 18 +- .../apache/hudi/common/util/TestParquetUtils.java | 11 - .../hudi/common/util/TestRocksDBManager.java | 20 +- .../common/util/collection/TestDiskBasedMap.java | 2 +- .../util/collection/TestExternalSpillableMap.java | 9 +- ...imple-test.avsc => timestamp-test-evolved.avsc} | 17 +- hudi-hadoop-mr/pom.xml | 21 +- .../org/apache/hudi/hadoop/HoodieHiveUtil.java | 25 ++ .../hudi/hadoop/HoodieParquetInputFormat.java | 290 ++++++++---- .../hudi/hadoop/HoodieROTablePathFilter.java | 10 +- .../org/apache/hudi/hadoop/InputPathHandler.java | 139 ++++++ .../hadoop/hive/HoodieCombineHiveInputFormat.java | 6 + .../realtime/HoodieParquetRealtimeInputFormat.java | 2 +- .../apache/hudi/hadoop/InputFormatTestUtil.java | 10 +- .../apache/hudi/hadoop/InputPathHandlerTest.java | 181 ++++++++ ...rmat.java => TestHoodieParquetInputFormat.java} | 78 +++- .../hudi/hadoop/TestRecordReaderValueIterator.java | 11 +- .../realtime/TestHoodieRealtimeRecordReader.java | 70 +-- .../java/org/apache/hudi/hive/HiveSyncConfig.java | 2 +- .../java/org/apache/hudi/hive/HiveSyncTool.java | 13 +- .../org/apache/hudi/hive/HoodieHiveClient.java | 20 +- .../java/org/apache/hudi/hive/util/SchemaUtil.java | 4 +- .../org/apache/hudi/hive/TestHiveSyncTool.java | 30 +- .../test/java/org/apache/hudi/hive/TestUtil.java | 20 +- .../org/apache/hudi/hive/util/HiveTestService.java | 25 -- .../java/org/apache/hudi/integ/ITTestBase.java | 5 +- hudi-spark/pom.xml | 17 +- .../java/org/apache/hudi/ComplexKeyGenerator.java | 4 +- .../main/java/org/apache/hudi/DataSourceUtils.java | 63 +-- .../org/apache/hudi/HoodieDataSourceHelpers.java | 2 +- .../apache/hudi/NonpartitionedKeyGenerator.java | 2 +- .../main/java/org/apache/hudi/QuickstartUtils.java | 1 + .../java/org/apache/hudi/SimpleKeyGenerator.java | 4 +- .../org/apache/hudi/payload/AWSDmsAvroPayload.java | 4 +- .../org/apache/hudi/AvroConversionHelper.scala | 103 ++++- .../org/apache/hudi/AvroConversionUtils.scala | 18 +- .../scala/org/apache/hudi/DataSourceOptions.scala | 8 +- .../main/scala/org/apache/hudi/DefaultSource.scala | 2 +- .../org/apache/hudi/HoodieSparkSqlWriter.scala | 182 +++----- .../org/apache/hudi/IncrementalRelation.scala | 6 +- hudi-spark/src/test/java/DataSourceTestUtils.java | 2 +- hudi-spark/src/test/java/HoodieJavaApp.java | 8 +- .../src/test/java/HoodieJavaStreamingApp.java | 4 +- .../src/test/scala/TestDataSourceDefaults.scala | 4 +- hudi-timeline-service/pom.xml | 1 - .../timeline/service/FileSystemViewHandler.java | 16 +- .../hudi/timeline/service/TimelineService.java | 4 +- .../timeline/service/handlers/DataFileHandler.java | 2 +- .../service/handlers/FileSliceHandler.java | 4 +- .../timeline/service/handlers/TimelineHandler.java | 2 +- hudi-utilities/pom.xml | 3 +- .../apache/hudi/utilities/HDFSParquetImporter.java | 10 +- .../hudi/utilities/HiveIncrementalPuller.java | 6 +- .../org/apache/hudi/utilities/HoodieCleaner.java | 4 +- .../hudi/utilities/HoodieCompactionAdminTool.java | 4 +- .../org/apache/hudi/utilities/HoodieCompactor.java | 6 +- .../hudi/utilities/HoodieSnapshotCopier.java | 4 +- .../hudi/utilities/HoodieWithTimelineServer.java | 2 +- .../org/apache/hudi/utilities/UtilHelpers.java | 10 +- .../adhoc/UpgradePayloadFromUberToApache.java | 4 +- .../hudi/utilities/deltastreamer/DeltaSync.java | 38 +- .../deltastreamer/HoodieDeltaStreamer.java | 17 +- .../deltastreamer/SourceFormatAdapter.java | 11 +- .../keygen/TimestampBasedKeyGenerator.java | 20 +- .../hudi/utilities/perf/TimelineServerPerf.java | 2 +- .../schema/NullTargetSchemaRegistryProvider.java | 2 +- .../hudi/utilities/schema/SchemaProvider.java | 2 +- .../hudi/utilities/TestHDFSParquetImporter.java | 2 +- .../hudi/utilities/TestHiveIncrementalPuller.java | 27 ++ .../hudi/utilities/TestHoodieDeltaStreamer.java | 283 +++++++----- .../utilities/TestTimestampBasedKeyGenerator.java | 83 ++++ .../apache/hudi/utilities/UtilitiesTestBase.java | 2 +- .../utilities/sources/AbstractBaseTestSource.java | 25 +- .../sources/DistributedTestDataSource.java | 4 +- .../resources/delta-streamer-config/source.avsc | 8 +- .../sql-transformer.properties | 2 +- .../resources/delta-streamer-config/target.avsc | 12 +- packaging/hudi-hadoop-mr-bundle/pom.xml | 14 +- packaging/hudi-spark-bundle/pom.xml | 6 - packaging/hudi-timeline-server-bundle/pom.xml | 1 - pom.xml | 136 +++++- style/checkstyle.xml | 1 + 227 files changed, 3567 insertions(+), 2650 deletions(-) delete mode 100644 _config.yml create mode 100644 doap_HUDI.rdf rename hudi-cli/src/main/java/org/apache/hudi/cli/commands/{DatasetsCommand.java => TableCommand.java} (71%) create mode 100644 hudi-client/src/main/java/org/apache/hudi/AbstractHoodieWriteClient.java rename {hudi-spark/src/main/java/org/apache/hudi => hudi-common/src/main/java/org/apache/hudi/common/model}/BaseAvroPayload.java (83%) rename {hudi-spark/src/main/java/org/apache/hudi => hudi-common/src/main/java/org/apache/hudi/common/model}/OverwriteWithLatestAvroPayload.java (77%) rename hudi-common/src/main/java/org/apache/hudi/exception/{InvalidDatasetException.java => InvalidTableException.java} (80%) rename hudi-common/src/main/java/org/apache/hudi/exception/{DatasetNotFoundException.java => TableNotFoundException.java} (67%) copy hudi-common/src/test/resources/{simple-test.avsc => timestamp-test-evolved.avsc} (77%) create mode 100644 hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/InputPathHandler.java create mode 100644 hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/InputPathHandlerTest.java rename hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/{TestHoodieInputFormat.java => TestHoodieParquetInputFormat.java} (75%) create mode 100644 hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHiveIncrementalPuller.java create mode 100644 hudi-utilities/src/test/java/org/apache/hudi/utilities/TestTimestampBasedKeyGenerator.java