zhangyue19921010 opened a new pull request #3623:
URL: https://github.com/apache/hudi/pull/3623
## What is the purpose of the pull request
Execute `hbase-server` and `hbase-client` dependency in Hudi-presto-bundle.
Add `hbase-shaded-client` and `hbase-shaded-server` in Hudi-presto-bundle.
## Brief change log
According to current hudi master branch, Hudi support metadata table feature
which is an internal MOR table. Also metadata table uses HFile as its' base
file, which means will introduce HBase related dependcies.
Although hudi-presto-bundle's pom.xml already shaded guava but maybe did not
take care of the dependencies between HBase and guava. So that we need
hbase-shaded-server here.
Without this, presto is throwing NoSuchMethodError when querying with
metadata table enable.
```
2021-09-08T16:22:21.089+0800 INFO pool-54-thread-1 cache-stats
[cache: projector-compiler:hoover] request: 0, hit: 0/1.00, miss: 0/0.00,
loadTime: 0.00ms, eviction: 0
2021-09-08T16:22:21.090+0800 INFO pool-54-thread-1 cache-stats
[cache: page-writer-compiler] request: 0, hit: 0/1.00, miss: 0/0.00, loadTime:
0.00ms, eviction: 0
2021-09-08T16:22:21.090+0800 INFO pool-54-thread-1 cache-stats
[cache: projector-compiler:fw] request: 0, hit: 0/1.00, miss: 0/0.00,
loadTime: 0.00ms, eviction: 0
2021-09-08T16:22:21.090+0800 INFO pool-54-thread-1 cache-stats
[cache: page-writer-compiler] request: 0, hit: 0/1.00, miss: 0/0.00, loadTime:
0.00ms, eviction: 0
2021-09-08T16:22:29.156+0800 INFO hive-hudi-2
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:22:30.272+0800 INFO hive-hudi-2
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/hoodie.properties
2021-09-08T16:22:30.548+0800 INFO hive-hudi-2
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:22:30.548+0800 INFO hive-hudi-2
org.apache.hudi.common.table.HoodieTableMetaClient Loading Active commit
timeline for s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:22:30.836+0800 INFO hive-hudi-2
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
upto : Option{val=[20210906070619__replacecommit__COMPLETED]}
2021-09-08T16:22:30.846+0800 INFO hive-hudi-2
org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based
view for basePath s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:22:34.730+0800 INFO hive-hudi-2
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 3881 ms to
read 11 instants, 957 replaced file groups
2021-09-08T16:22:35.620+0800 INFO hive-hudi-2
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-09-08T16:22:35.625+0800 INFO hive-hudi-2
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:22:36.476+0800 INFO hive-hudi-2
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/hoodie.properties
2021-09-08T16:22:36.835+0800 INFO hive-hudi-2
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:22:36.836+0800 INFO hive-hudi-2
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata
2021-09-08T16:22:37.844+0800 INFO hive-hudi-2
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata/.hoodie/hoodie.properties
2021-09-08T16:22:38.253+0800 INFO hive-hudi-2
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata
2021-09-08T16:22:38.581+0800 INFO hive-hudi-2
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
upto : Option{val=[20210906065610__deltacommit__COMPLETED]}
2021-09-08T16:22:38.581+0800 INFO hive-hudi-2
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-09-08T16:22:39.484+0800 INFO hive-hudi-2
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-09-08T16:22:39.485+0800 INFO hive-hudi-2
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (files)
2021-09-08T16:22:40.782+0800 INFO hive-hudi-2
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=3, NumFileGroups=1, FileGroupsCreationTime=6, StoreTimeTaken=0
2021-09-08T16:22:41.126+0800 INFO hive-hudi-2
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
upto : Option{val=[20210906070619__replacecommit__COMPLETED]}
2021-09-08T16:22:41.428+0800 INFO hive-hudi-2
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
upto : Option{val=[20210906065610__deltacommit__COMPLETED]}
2021-09-08T16:22:41.880+0800 INFO hive-hudi-2
org.apache.hudi.metadata.HoodieTableMetadataUtil Updating at
20210906070619 from Commit/CLUSTER. #partitions_updated=2
2021-09-08T16:22:41.880+0800 INFO hive-hudi-2
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (2021080100)
2021-09-08T16:22:42.764+0800 INFO hive-hudi-2
org.apache.hadoop.hbase.io.hfile.CacheConfig Allocating LruBlockCache
size=819.20 MB, blockSize=64 KB
2021-09-08T16:22:42.778+0800 WARN hive-hudi-2
com.facebook.presto.hive.util.ResumableTasks ResumableTask completed
exceptionally
java.lang.NoSuchMethodError:
com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
at
org.apache.hadoop.hbase.io.hfile.LruBlockCache.toString(LruBlockCache.java:704)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at
org.apache.hadoop.hbase.io.hfile.CacheConfig.toString(CacheConfig.java:502)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at
org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:260)
at
org.apache.hudi.io.storage.HoodieFileReaderFactory.newHFileFileReader(HoodieFileReaderFactory.java:56)
at
org.apache.hudi.io.storage.HoodieFileReaderFactory.getFileReader(HoodieFileReaderFactory.java:42)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.openReadersIfNeeded(HoodieBackedTableMetadata.java:210)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.openReadersIfNeededOrThrow(HoodieBackedTableMetadata.java:177)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKeyFromMetadata(HoodieBackedTableMetadata.java:129)
at
org.apache.hudi.metadata.BaseTableMetadata.getMergedRecordByKey(BaseTableMetadata.java:280)
at
org.apache.hudi.metadata.BaseTableMetadata.fetchAllFilesInPartition(BaseTableMetadata.java:217)
at
org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:126)
at
org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:65)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:280)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:269)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestBaseFiles(AbstractTableFileSystemView.java:430)
at
org.apache.hudi.hadoop.HoodieROTablePathFilter.accept(HoodieROTablePathFilter.java:181)
at
com.facebook.presto.hive.util.HiveFileIterator.lambda$getLocatedFileStatusRemoteIterator$0(HiveFileIterator.java:105)
at com.google.common.collect.Iterators$5.computeNext(Iterators.java:639)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at
com.facebook.presto.hive.util.HiveFileIterator.computeNext(HiveFileIterator.java:71)
at
com.facebook.presto.hive.util.HiveFileIterator.computeNext(HiveFileIterator.java:39)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at
java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1811)
at
java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
at
java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
at
java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
at
java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)
at java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
at
com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:288)
at
com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:106)
at
com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:214)
at
com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
at
com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
at
com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
at
com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
With this patch, presto query works fine
```
2021-09-08T16:33:56.123+0800 INFO main
com.facebook.presto.server.PrestoServer ======== SERVER STARTED ========
2021-09-08T16:34:03.617+0800 INFO dispatcher-query-1
com.facebook.presto.event.QueryMonitor TIMELINE: Query
20210908_083403_00000_c47wx ::
Transaction:[c1dceda6-9e42-4932-b4e5-00e778763f22] :: elapsed 81ms :: planning
0ms :: scheduling 81ms :: running 0ms :: finishing 81ms :: begin
2021-09-08T16:34:03.511+08:00 :: end 2021-09-08T16:34:03.592+08:00
2021-09-08T16:34:12.348+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:34:13.492+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/hoodie.properties
2021-09-08T16:34:13.789+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:34:13.790+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableMetaClient Loading Active commit
timeline for s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:34:14.095+0800 INFO hive-hudi-1
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
upto : Option{val=[20210906070619__replacecommit__COMPLETED]}
2021-09-08T16:34:14.107+0800 INFO hive-hudi-1
org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based
view for basePath s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:34:17.760+0800 INFO hive-hudi-1
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 3650 ms to
read 11 instants, 957 replaced file groups
2021-09-08T16:34:18.684+0800 INFO hive-hudi-1
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-09-08T16:34:18.688+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:34:19.586+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/hoodie.properties
2021-09-08T16:34:19.859+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://xxxx/sos/mars/sample_xxxx
2021-09-08T16:34:19.859+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata
2021-09-08T16:34:20.805+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata/.hoodie/hoodie.properties
2021-09-08T16:34:21.105+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata
2021-09-08T16:34:21.383+0800 INFO hive-hudi-1
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
upto : Option{val=[20210906065610__deltacommit__COMPLETED]}
2021-09-08T16:34:21.384+0800 INFO hive-hudi-1
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 1 ms to
read 0 instants, 0 replaced file groups
2021-09-08T16:34:22.196+0800 INFO hive-hudi-1
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-09-08T16:34:22.197+0800 INFO hive-hudi-1
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (files)
2021-09-08T16:34:23.395+0800 INFO hive-hudi-1
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=3, NumFileGroups=1, FileGroupsCreationTime=7, StoreTimeTaken=1
2021-09-08T16:34:23.683+0800 INFO hive-hudi-1
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
upto : Option{val=[20210906070619__replacecommit__COMPLETED]}
2021-09-08T16:34:24.003+0800 INFO hive-hudi-1
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
upto : Option{val=[20210906065610__deltacommit__COMPLETED]}
2021-09-08T16:34:24.448+0800 INFO hive-hudi-1
org.apache.hudi.metadata.HoodieTableMetadataUtil Updating at
20210906070619 from Commit/CLUSTER. #partitions_updated=2
2021-09-08T16:34:24.449+0800 INFO hive-hudi-1
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (2021080100)
2021-09-08T16:34:25.276+0800 INFO hive-hudi-1
org.apache.hadoop.hbase.io.hfile.CacheConfig Allocating LruBlockCache
size=819.20 MB, blockSize=64 KB
2021-09-08T16:34:25.304+0800 INFO hive-hudi-1
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=882176, freeSize=858111296,
maxSize=858993472, heapSize=882176, minSize=816043776, minFactor=0.95,
multiSize=408021888, multiFactor=0.5, singleSize=204010944, singleFactor=0.25},
cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
prefetchOnOpen=false
2021-09-08T16:34:27.479+0800 INFO hive-hudi-1
org.apache.hadoop.io.compress.CodecPool Got brand-new decompressor [.gz]
2021-09-08T16:34:27.532+0800 INFO hive-hudi-1
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base
file from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata/files/f55a2608-8b79-4967-ba6a-6067501c5067-0_0-7-246_20210831035109.hfile
at instant 20210831035109 in 2261 ms
2021-09-08T16:34:27.550+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata
2021-09-08T16:34:29.191+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata/.hoodie/hoodie.properties
2021-09-08T16:34:29.497+0800 INFO hive-hudi-1
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata
2021-09-08T16:34:29.783+0800 INFO hive-hudi-1
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
upto : Option{val=[20210906065610__deltacommit__COMPLETED]}
2021-09-08T16:34:30.087+0800 INFO hive-hudi-1
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata/files/.f55a2608-8b79-4967-ba6a-6067501c5067-0_20210831035109.log.1_0-24-263',
fileLen=0}
2021-09-08T16:34:31.458+0800 INFO hive-hudi-1
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata/files/.f55a2608-8b79-4967-ba6a-6067501c5067-0_20210831035109.log.1_0-24-263
at instant 20210906065610
2021-09-08T16:34:31.458+0800 INFO hive-hudi-1
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the
final data blocks
2021-09-08T16:34:31.459+0800 INFO hive-hudi-1
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-09-08T16:34:31.479+0800 INFO hive-hudi-1
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=882176, freeSize=858111296,
maxSize=858993472, heapSize=882176, minSize=816043776, minFactor=0.95,
multiSize=408021888, multiFactor=0.5, singleSize=204010944, singleFactor=0.25},
cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
prefetchOnOpen=false
2021-09-08T16:34:31.526+0800 INFO hive-hudi-1
org.apache.hudi.common.util.collection.BitCaskDiskMap Spilling to file
location
/tmp/hudi-BITCASK-24f97006-41bf-4037-b1cf-0c4852c590a5/bf7fe07c-6ccd-40a0-9a89-d2642e5dd31a
in host (172.24.10.19) with hostname (yuezhang-mac)
2021-09-08T16:34:31.528+0800 INFO hive-hudi-1
org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated
Payload size => 664
2021-09-08T16:34:31.528+0800 INFO hive-hudi-1
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log
files scanned => 1
2021-09-08T16:34:31.528+0800 INFO hive-hudi-1
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner
MaxMemoryInBytes allowed for compaction => 1073741824
2021-09-08T16:34:31.529+0800 INFO hive-hudi-1
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in MemoryBasedMap in ExternalSpillableMap => 2
2021-09-08T16:34:31.529+0800 INFO hive-hudi-1
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in
bytes of MemoryBasedMap in ExternalSpillableMap => 1328
2021-09-08T16:34:31.529+0800 INFO hive-hudi-1
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in BitCaskDiskMap in ExternalSpillableMap => 0
2021-09-08T16:34:31.529+0800 INFO hive-hudi-1
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file
spilled to disk => 0
2021-09-08T16:34:31.529+0800 INFO hive-hudi-1
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log
files from
[s3a://xxxx/sos/mars/sample_xxxx/.hoodie/metadata/files/.f55a2608-8b79-4967-ba6a-6067501c5067-0_20210831035109.log.1_0-24-263]
at instant (dataset instant=20210906070619, metadata instant=20210906065610)
in 3997 ms
2021-09-08T16:34:32.072+0800 INFO hive-hudi-1
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key
2021080100 took [baseFileRead, logMerge] [542, 0] ms
2021-09-08T16:34:32.090+0800 INFO hive-hudi-1
org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from
metadata: partition=2021080100, #files=6
2021-09-08T16:34:32.091+0800 INFO hive-hudi-1
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=6, NumFileGroups=6, FileGroupsCreationTime=1, StoreTimeTaken=0
2021-09-08T16:34:32.093+0800 INFO hive-hudi-1
org.apache.hudi.hadoop.HoodieROTablePathFilter Based on hoodie metadata from
base path: s3a://xxxx/sos/mars/sample_xxxx, caching 2 files under
s3a://xxxx/sos/mars/sample_xxxx/2021080100
2021-09-08T16:34:32.375+0800 INFO
20210908_083405_00001_c47wx.2.0.0-22-118
org.apache.hadoop.conf.Configuration.deprecation io.bytes.per.checksum
is deprecated. Instead, use dfs.bytes-per-checksum
2021-09-08T16:34:47.735+0800 INFO dispatcher-query-4
com.facebook.presto.event.QueryMonitor TIMELINE: Query
20210908_083405_00001_c47wx ::
Transaction:[f928abbf-b7ef-4999-93c4-2246b6fd8fc8] :: elapsed 41703ms ::
planning 1257ms :: scheduling 25235ms :: running 15152ms :: finishing 59ms ::
begin 2021-09-08T16:34:05.882+08:00 :: end 2021-09-08T16:34:47.585+08:00
```
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test
coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please
describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
