James601232 edited a comment on issue #3963:
URL: https://github.com/apache/hudi/issues/3963#issuecomment-966696290
fully logs:
com.tx.bigdata.HoodieJavaWriteClientExample
0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
20607 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to
server: tob23.bigdata.net/101.162.133.154:9000. Already tried 0 time(s);
maxRetries=45
40622 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to
server: tob23.bigdata.net/101.162.133.154:9000. Already tried 1 time(s);
maxRetries=45
60626 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to
server: tob23.bigdata.net/101.162.133.154:9000. Already tried 2 time(s);
maxRetries=45
80709 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to
server: tob23.bigdata.net/101.162.133.154:9000. Already tried 3 time(s);
maxRetries=45
81136 [main] INFO org.apache.hudi.client.embedded.EmbeddedTimelineService
- Starting Timeline service !!
81136 [main] WARN org.apache.hudi.client.embedded.EmbeddedTimelineService
- Unable to find driver bind address from spark config
81147 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating View Manager with storage type :MEMORY
81147 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating in-memory based Table View
81160 [main] INFO org.apache.hudi.org.apache.jetty.util.log - Logging
initialized @81966ms to org.apache.hudi.org.apache.jetty.util.log.Slf4jLog
81371 [main] INFO io.javalin.Javalin -
__ __ _
/ /____ _ _ __ ____ _ / /(_)____
__ / // __ `/| | / // __ `// // // __ \
/ /_/ // /_/ / | |/ // /_/ // // // / / /
\____/ \__,_/ |___/ \__,_//_//_//_/ /_/
https://javalin.io/documentation
81372 [main] INFO io.javalin.Javalin - Starting Javalin ...
81497 [main] INFO io.javalin.Javalin - Listening on http://localhost:50730/
81497 [main] INFO io.javalin.Javalin - Javalin started in 128ms \o/
81497 [main] INFO org.apache.hudi.timeline.service.TimelineService -
Starting Timeline server on port :50730
81497 [main] INFO org.apache.hudi.client.embedded.EmbeddedTimelineService
- Started embedded timeline server at 0.0.0.0:50730
81507 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading HoodieTableMetaClient from /home/work/hudi_catalog4
81557 [main] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading
table properties from /home/work/hudi_catalog4/.hoodie/hoodie.properties
81833 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET)
from /home/work/hudi_catalog4
81833 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading Active commit timeline for /home/work/hudi_catalog4
82094 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20211111203048__commit__COMPLETED]}
82096 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating View Manager with storage type :REMOTE_FIRST
82096 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating remote first table view
82142 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading HoodieTableMetaClient from /home/work/hudi_catalog4
82191 [main] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading
table properties from /home/work/hudi_catalog4/.hoodie/hoodie.properties
82313 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET)
from /home/work/hudi_catalog4
82313 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading Active commit timeline for /home/work/hudi_catalog4
82435 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20211111203048__commit__COMPLETED]}
82439 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading HoodieTableMetaClient from /home/work/hudi_catalog4
82482 [main] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading
table properties from /home/work/hudi_catalog4/.hoodie/hoodie.properties
82638 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET)
from /home/work/hudi_catalog4
82638 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading Active commit timeline for /home/work/hudi_catalog4
82679 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20211111203048__commit__COMPLETED]}
82681 [main] INFO org.apache.hudi.client.AbstractHoodieWriteClient -
Generate a new instant time: 20211112071954 action: commit
82682 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Creating a new
instant [==>20211112071954__commit__REQUESTED]
82826 [main] INFO com.tx.bigdata.HoodieJavaWriteClientExample - Starting
commit 20211112071954
82878 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading HoodieTableMetaClient from /home/work/hudi_catalog4
83122 [main] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading
table properties from /home/work/hudi_catalog4/.hoodie/hoodie.properties
83201 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET)
from /home/work/hudi_catalog4
83201 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading Active commit timeline for /home/work/hudi_catalog4
83246 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[==>20211112071954__commit__REQUESTED]}
83246 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating View Manager with storage type :REMOTE_FIRST
83246 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating remote first table view
83445 [main] INFO org.apache.hudi.client.AsyncCleanerService - Async auto
cleaning is not enabled. Not running cleaner now
83619 [main] INFO
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor - Workload
profile :WorkloadProfile {globalStat=WorkloadStat {numInserts=10,
numUpdates=0}, partitionStat={2020/01/02=WorkloadStat {numInserts=5,
numUpdates=0}, 2020/01/01=WorkloadStat {numInserts=4, numUpdates=0},
2020/01/03=WorkloadStat {numInserts=1, numUpdates=0}}, operationType=null}
83658 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Checking for file
exists ?/home/work/hudi_catalog4/.hoodie/20211112071954.commit.requested
84409 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Create new file
for toInstant ?/home/work/hudi_catalog4/.hoodie/20211112071954.inflight
84869 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- AvgRecordSize => 1024
84873 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating remote view for basePath /home/work/hudi_catalog4.
Server=0.0.0.0:50730, Timeout=300
84873 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating InMemory based view for basePath /home/work/hudi_catalog4
84879 [main] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 2 ms to
read 0 instants, 0 replaced file groups
84949 [main] INFO org.apache.hudi.common.util.ClusteringUtils - Found 0
files in pending clustering operations
84960 [main] INFO
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending
request :
(http://0.0.0.0:50730/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2020%2F01%2F02&maxinstant=20211111203048&basepath=%2Fhome%2Fwork%2Fhudi_catalog4&lastinstantts=20211111203048&timelinehash=66173038ff0a2b772212dd1e06335c5c002d206bc91f381edacca9d5db467745)
85340 [qtp633079302-25] INFO
org.apache.hudi.common.table.HoodieTableMetaClient - Loading
HoodieTableMetaClient from /home/work/hudi_catalog4
85399 [qtp633079302-25] INFO org.apache.hudi.common.table.HoodieTableConfig
- Loading table properties from
/home/work/hudi_catalog4/.hoodie/hoodie.properties
85485 [qtp633079302-25] INFO
org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of
type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
/home/work/hudi_catalog4
85485 [qtp633079302-25] INFO
org.apache.hudi.common.table.view.FileSystemViewManager - Creating InMemory
based view for basePath /home/work/hudi_catalog4
85618 [qtp633079302-25] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[==>20211112071954__commit__INFLIGHT]}
85619 [qtp633079302-25] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 0 ms to
read 0 instants, 0 replaced file groups
85671 [qtp633079302-25] INFO org.apache.hudi.common.util.ClusteringUtils -
Found 0 files in pending clustering operations
85675 [qtp633079302-25] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file
system view for partition (2020/01/02)
85766 [qtp633079302-25] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView -
addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=7,
StoreTimeTaken=1
85800 [main] INFO
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending
request :
(http://0.0.0.0:50730/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2020%2F01%2F01&maxinstant=20211111203048&basepath=%2Fhome%2Fwork%2Fhudi_catalog4&lastinstantts=20211111203048&timelinehash=66173038ff0a2b772212dd1e06335c5c002d206bc91f381edacca9d5db467745)
85802 [qtp633079302-19] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file
system view for partition (2020/01/01)
85882 [qtp633079302-19] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView -
addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=0,
StoreTimeTaken=0
85885 [main] INFO
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending
request :
(http://0.0.0.0:50730/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2020%2F01%2F03&maxinstant=20211111203048&basepath=%2Fhome%2Fwork%2Fhudi_catalog4&lastinstantts=20211111203048&timelinehash=66173038ff0a2b772212dd1e06335c5c002d206bc91f381edacca9d5db467745)
85886 [qtp633079302-33] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file
system view for partition (2020/01/03)
85965 [qtp633079302-33] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView -
addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=0,
StoreTimeTaken=0
85967 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- For partitionPath : 2020/01/02 Small Files => [SmallFile
{location=HoodieRecordLocation {instantTime=20211111203048,
fileId=1cef849a-6dbe-4e4d-8894-1e196252a854-0}, sizeBytes=437290}]
85968 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- Assigning 5 inserts to new update bucket 0
85969 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- Total insert buckets for partition path 2020/01/02 => [(InsertBucket
{bucketNumber=0, weight=1.0},1.0)]
85969 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- For partitionPath : 2020/01/01 Small Files => [SmallFile
{location=HoodieRecordLocation {instantTime=20211111203048,
fileId=10658ff8-0e7d-4da2-9455-6d76b80518d4-0}, sizeBytes=437106}]
85969 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- Assigning 4 inserts to new update bucket 1
85969 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- Total insert buckets for partition path 2020/01/01 => [(InsertBucket
{bucketNumber=1, weight=1.0},1.0)]
85969 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- For partitionPath : 2020/01/03 Small Files => [SmallFile
{location=HoodieRecordLocation {instantTime=20211111203048,
fileId=158308ae-8ac1-474c-85bc-6f0c4f88243d-0}, sizeBytes=436359}]
85969 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- Assigning 1 inserts to new update bucket 2
85969 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- Total insert buckets for partition path 2020/01/03 => [(InsertBucket
{bucketNumber=2, weight=1.0},1.0)]
85969 [main] INFO org.apache.hudi.table.action.commit.JavaUpsertPartitioner
- Total Buckets :3, buckets info => {0=BucketInfo {bucketType=UPDATE,
fileIdPrefix=1cef849a-6dbe-4e4d-8894-1e196252a854-0, partitionPath=2020/01/02},
1=BucketInfo {bucketType=UPDATE,
fileIdPrefix=10658ff8-0e7d-4da2-9455-6d76b80518d4-0, partitionPath=2020/01/01},
2=BucketInfo {bucketType=UPDATE,
fileIdPrefix=158308ae-8ac1-474c-85bc-6f0c4f88243d-0,
partitionPath=2020/01/03}},
Partition to insert buckets => {2020/01/02=[(InsertBucket {bucketNumber=0,
weight=1.0},1.0)], 2020/01/01=[(InsertBucket {bucketNumber=1,
weight=1.0},1.0)], 2020/01/03=[(InsertBucket {bucketNumber=2,
weight=1.0},1.0)]},
UpdateLocations mapped to buckets
=>{10658ff8-0e7d-4da2-9455-6d76b80518d4-0=1,
158308ae-8ac1-474c-85bc-6f0c4f88243d-0=2,
1cef849a-6dbe-4e4d-8894-1e196252a854-0=0}
85974 [main] INFO
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending
request :
(http://0.0.0.0:50730/v1/hoodie/view/datafile/latest/partition?partition=2020%2F01%2F02&basepath=%2Fhome%2Fwork%2Fhudi_catalog4&fileid=1cef849a-6dbe-4e4d-8894-1e196252a854-0&lastinstantts=20211111203048&timelinehash=66173038ff0a2b772212dd1e06335c5c002d206bc91f381edacca9d5db467745)
85983 [main] INFO org.apache.hudi.io.HoodieMergeHandle -
MaxMemoryPerPartitionMerge => 1073741824
85993 [main] INFO
org.apache.hudi.common.util.collection.ExternalSpillableMap - Estimated
Payload size => 728
85996 [main] INFO org.apache.hudi.common.util.collection.BitCaskDiskMap -
Spilling to file location
/tmp/hudi-BITCASK-8a6950c0-9295-48d7-a860-c3323564f437/cb95c4a4-74a6-4024-9c73-0fcb6ce98f64
in host (127.0.0.1) with hostname (qihoodeiMac.local)
85996 [main] INFO org.apache.hudi.io.HoodieMergeHandle - Number of entries
in MemoryBasedMap => 5Total size in bytes of MemoryBasedMap => 3640Number of
entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
85996 [main] INFO org.apache.hudi.io.HoodieMergeHandle -
partitionPath:2020/01/02, fileId to be
merged:1cef849a-6dbe-4e4d-8894-1e196252a854-0
86343 [main] INFO org.apache.hudi.io.HoodieMergeHandle - Merging new data
into oldPath
/home/work/hudi_catalog4/2020/01/02/1cef849a-6dbe-4e4d-8894-1e196252a854-0_0-0-0_20211111203048.parquet,
as newPath
/home/work/hudi_catalog4/2020/01/02/1cef849a-6dbe-4e4d-8894-1e196252a854-0_0-0-0_20211112071954.parquet
86476 [main] INFO org.apache.hudi.table.marker.DirectWriteMarkers -
Creating Marker
Path=/home/work/hudi_catalog4/.hoodie/.temp/20211112071954/2020/01/02/1cef849a-6dbe-4e4d-8894-1e196252a854-0_0-0-0_20211112071954.parquet.marker.MERGE
86882 [main] INFO org.apache.hudi.table.marker.DirectWriteMarkers -
[direct] Created marker file
/home/work/hudi_catalog4/.hoodie/.temp/20211112071954/2020/01/02/1cef849a-6dbe-4e4d-8894-1e196252a854-0_0-0-0_20211112071954.parquet.marker.MERGE
in 535 ms
87362 [main] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new
compressor [.gz]
87851 [pool-5-thread-1] INFO
org.apache.hudi.common.util.queue.IteratorBasedQueueProducer - starting to
buffer records
87852 [pool-5-thread-2] INFO
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor - starting consumer
thread
88931 [pool-5-thread-1] INFO
org.apache.parquet.hadoop.InternalParquetRecordReader - RecordReader
initialized will read a total of 15 records.
88931 [pool-5-thread-1] INFO
org.apache.parquet.hadoop.InternalParquetRecordReader - at row 0. reading next
block
89067 [pool-5-thread-1] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new decompressor [.gz]
89074 [pool-5-thread-1] INFO
org.apache.parquet.hadoop.InternalParquetRecordReader - block read in memory
in 143 ms. row count = 15
89103 [pool-5-thread-1] INFO
org.apache.hudi.common.util.queue.IteratorBasedQueueProducer - finished
buffering records
89108 [pool-5-thread-2] INFO
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor - Queue Consumption
is done; notifying producer threads
89134 [main] INFO org.apache.parquet.hadoop.InternalParquetRecordWriter -
Flushing mem columnStore to file. allocated memory: 3934
90533 [main] INFO org.apache.hudi.io.HoodieMergeHandle - MergeHandle for
partitionPath 2020/01/02 fileID 1cef849a-6dbe-4e4d-8894-1e196252a854-0, took
4552 ms.
90534 [main] INFO
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending
request :
(http://0.0.0.0:50730/v1/hoodie/view/datafile/latest/partition?partition=2020%2F01%2F01&basepath=%2Fhome%2Fwork%2Fhudi_catalog4&fileid=10658ff8-0e7d-4da2-9455-6d76b80518d4-0&lastinstantts=20211111203048&timelinehash=66173038ff0a2b772212dd1e06335c5c002d206bc91f381edacca9d5db467745)
90538 [main] INFO org.apache.hudi.io.HoodieMergeHandle -
MaxMemoryPerPartitionMerge => 1073741824
90539 [main] INFO
org.apache.hudi.common.util.collection.ExternalSpillableMap - Estimated
Payload size => 728
90540 [main] INFO org.apache.hudi.common.util.collection.BitCaskDiskMap -
Spilling to file location
/tmp/hudi-BITCASK-23c3dfa4-00d3-425a-8b74-583d4b5a4f67/20756bf2-d7c3-4934-8c32-2262b97ea5b2
in host (127.0.0.1) with hostname (qihoodeiMac.local)
90541 [main] INFO org.apache.hudi.io.HoodieMergeHandle - Number of entries
in MemoryBasedMap => 4Total size in bytes of MemoryBasedMap => 2912Number of
entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
90541 [main] INFO org.apache.hudi.io.HoodieMergeHandle -
partitionPath:2020/01/01, fileId to be
merged:10658ff8-0e7d-4da2-9455-6d76b80518d4-0
90581 [main] INFO org.apache.hudi.io.HoodieMergeHandle - Merging new data
into oldPath
/home/work/hudi_catalog4/2020/01/01/10658ff8-0e7d-4da2-9455-6d76b80518d4-0_0-0-0_20211111203048.parquet,
as newPath
/home/work/hudi_catalog4/2020/01/01/10658ff8-0e7d-4da2-9455-6d76b80518d4-0_0-0-0_20211112071954.parquet
90689 [main] INFO org.apache.hudi.table.marker.DirectWriteMarkers -
Creating Marker
Path=/home/work/hudi_catalog4/.hoodie/.temp/20211112071954/2020/01/01/10658ff8-0e7d-4da2-9455-6d76b80518d4-0_0-0-0_20211112071954.parquet.marker.MERGE
91114 [main] INFO org.apache.hudi.table.marker.DirectWriteMarkers -
[direct] Created marker file
/home/work/hudi_catalog4/.hoodie/.temp/20211112071954/2020/01/01/10658ff8-0e7d-4da2-9455-6d76b80518d4-0_0-0-0_20211112071954.parquet.marker.MERGE
in 532 ms
91183 [main] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new
compressor [.gz]
91238 [pool-6-thread-1] INFO
org.apache.hudi.common.util.queue.IteratorBasedQueueProducer - starting to
buffer records
91239 [pool-6-thread-2] INFO
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor - starting consumer
thread
92624 [pool-6-thread-1] INFO
org.apache.parquet.hadoop.InternalParquetRecordReader - RecordReader
initialized will read a total of 12 records.
92624 [pool-6-thread-1] INFO
org.apache.parquet.hadoop.InternalParquetRecordReader - at row 0. reading next
block
92813 [pool-6-thread-1] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new decompressor [.gz]
92814 [pool-6-thread-1] INFO
org.apache.parquet.hadoop.InternalParquetRecordReader - block read in memory
in 190 ms. row count = 12
92817 [pool-6-thread-1] INFO
org.apache.hudi.common.util.queue.IteratorBasedQueueProducer - finished
buffering records
92818 [pool-6-thread-2] INFO
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor - Queue Consumption
is done; notifying producer threads
92819 [main] INFO org.apache.parquet.hadoop.InternalParquetRecordWriter -
Flushing mem columnStore to file. allocated memory: 3262
94360 [main] INFO org.apache.hudi.io.HoodieMergeHandle - MergeHandle for
partitionPath 2020/01/01 fileID 10658ff8-0e7d-4da2-9455-6d76b80518d4-0, took
3822 ms.
94361 [main] INFO
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending
request :
(http://0.0.0.0:50730/v1/hoodie/view/datafile/latest/partition?partition=2020%2F01%2F03&basepath=%2Fhome%2Fwork%2Fhudi_catalog4&fileid=158308ae-8ac1-474c-85bc-6f0c4f88243d-0&lastinstantts=20211111203048&timelinehash=66173038ff0a2b772212dd1e06335c5c002d206bc91f381edacca9d5db467745)
94365 [main] INFO org.apache.hudi.io.HoodieMergeHandle -
MaxMemoryPerPartitionMerge => 1073741824
94366 [main] INFO
org.apache.hudi.common.util.collection.ExternalSpillableMap - Estimated
Payload size => 728
94368 [main] INFO org.apache.hudi.common.util.collection.BitCaskDiskMap -
Spilling to file location
/tmp/hudi-BITCASK-4eaafa7d-2820-48cd-8641-5f03c31b9ba4/435d474b-1beb-4ddc-b1bf-cb590d1db53a
in host (127.0.0.1) with hostname (qihoodeiMac.local)
94368 [main] INFO org.apache.hudi.io.HoodieMergeHandle - Number of entries
in MemoryBasedMap => 1Total size in bytes of MemoryBasedMap => 728Number of
entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
94368 [main] INFO org.apache.hudi.io.HoodieMergeHandle -
partitionPath:2020/01/03, fileId to be
merged:158308ae-8ac1-474c-85bc-6f0c4f88243d-0
94684 [main] INFO org.apache.hudi.io.HoodieMergeHandle - Merging new data
into oldPath
/home/work/hudi_catalog4/2020/01/03/158308ae-8ac1-474c-85bc-6f0c4f88243d-0_0-0-0_20211111203048.parquet,
as newPath
/home/work/hudi_catalog4/2020/01/03/158308ae-8ac1-474c-85bc-6f0c4f88243d-0_0-0-0_20211112071954.parquet
94796 [main] INFO org.apache.hudi.table.marker.DirectWriteMarkers -
Creating Marker
Path=/home/work/hudi_catalog4/.hoodie/.temp/20211112071954/2020/01/03/158308ae-8ac1-474c-85bc-6f0c4f88243d-0_0-0-0_20211112071954.parquet.marker.MERGE
95208 [main] INFO org.apache.hudi.table.marker.DirectWriteMarkers -
[direct] Created marker file
/home/work/hudi_catalog4/.hoodie/.temp/20211112071954/2020/01/03/158308ae-8ac1-474c-85bc-6f0c4f88243d-0_0-0-0_20211112071954.parquet.marker.MERGE
in 524 ms
95289 [main] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new
compressor [.gz]
95345 [pool-7-thread-1] INFO
org.apache.hudi.common.util.queue.IteratorBasedQueueProducer - starting to
buffer records
95345 [pool-7-thread-2] INFO
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor - starting consumer
thread
96362 [pool-7-thread-1] INFO
org.apache.parquet.hadoop.InternalParquetRecordReader - RecordReader
initialized will read a total of 3 records.
96362 [pool-7-thread-1] INFO
org.apache.parquet.hadoop.InternalParquetRecordReader - at row 0. reading next
block
96477 [pool-7-thread-1] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new decompressor [.gz]
96478 [pool-7-thread-1] INFO
org.apache.parquet.hadoop.InternalParquetRecordReader - block read in memory
in 116 ms. row count = 3
96480 [pool-7-thread-1] INFO
org.apache.hudi.common.util.queue.IteratorBasedQueueProducer - finished
buffering records
96480 [pool-7-thread-2] INFO
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor - Queue Consumption
is done; notifying producer threads
96481 [main] INFO org.apache.parquet.hadoop.InternalParquetRecordWriter -
Flushing mem columnStore to file. allocated memory: 1250
97867 [main] INFO org.apache.hudi.io.HoodieMergeHandle - MergeHandle for
partitionPath 2020/01/03 fileID 158308ae-8ac1-474c-85bc-6f0c4f88243d-0, took
3502 ms.
97868 [main] INFO
org.apache.hudi.table.action.commit.BaseCommitActionExecutor - Auto commit
enabled: Committing 20211112071954
97868 [main] INFO
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor - Committing
20211112071954, action Type commit
98378 [main] INFO
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor - Committing
20211112071954, action Type commit
98378 [main] INFO org.apache.hudi.common.util.CommitUtils - Creating
metadata for INSERT numWriteStats:3numReplaceFileIds:0
98383 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Marking instant
complete [==>20211112071954__commit__INFLIGHT]
98383 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Checking for file
exists ?/home/work/hudi_catalog4/.hoodie/20211112071954.inflight
99035 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Create new file
for toInstant ?/home/work/hudi_catalog4/.hoodie/20211112071954.commit
99035 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Completed
[==>20211112071954__commit__INFLIGHT]
99035 [main] INFO
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor - Committed
20211112071954
99501 [main] INFO org.apache.hudi.table.marker.DirectWriteMarkers -
Removing marker directory at
/home/work/hudi_catalog4/.hoodie/.temp/20211112071954
99556 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20211112071954__commit__COMPLETED]}
99557 [main] INFO org.apache.hudi.table.HoodieTimelineArchiveLog - No
Instants to archive
99557 [main] INFO org.apache.hudi.client.AbstractHoodieWriteClient - Auto
cleaning is enabled. Running cleaner now
99558 [main] INFO org.apache.hudi.client.AbstractHoodieWriteClient -
Scheduling cleaning at instant time :20211112072011
99558 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading HoodieTableMetaClient from /home/work/hudi_catalog4
99594 [main] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading
table properties from /home/work/hudi_catalog4/.hoodie/hoodie.properties
99923 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET)
from /home/work/hudi_catalog4
99923 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading Active commit timeline for /home/work/hudi_catalog4
99965 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20211112071954__commit__COMPLETED]}
99966 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating View Manager with storage type :REMOTE_FIRST
99966 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating remote first table view
99970 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating remote view for basePath /home/work/hudi_catalog4.
Server=0.0.0.0:50730, Timeout=300
99971 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating InMemory based view for basePath /home/work/hudi_catalog4
99971 [main] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 0 ms to
read 0 instants, 0 replaced file groups
100010 [main] INFO org.apache.hudi.common.util.ClusteringUtils - Found 0
files in pending clustering operations
100012 [main] INFO
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending
request :
(http://0.0.0.0:50730/v1/hoodie/view/compactions/pending/?basepath=%2Fhome%2Fwork%2Fhudi_catalog4&lastinstantts=20211112071954&timelinehash=531158aa753faba88dee696a03c6ca809cff47a29a8b9533422b1a7c9fa2c711)
100015 [qtp633079302-33] INFO
org.apache.hudi.timeline.service.RequestHandler - Syncing view as client
passed last known instant 20211112071954 as last known instant but server has
the following last instant on timeline
:Option{val=[20211111203048__commit__COMPLETED]}
100070 [qtp633079302-33] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20211112071954__commit__COMPLETED]}
100071 [qtp633079302-33] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 0 ms to
read 0 instants, 0 replaced file groups
100112 [qtp633079302-33] INFO org.apache.hudi.common.util.ClusteringUtils
- Found 0 files in pending clustering operations
100116 [main] INFO org.apache.hudi.table.action.clean.CleanPlanner - No
earliest commit to retain. No need to scan partitions !!
100116 [main] INFO org.apache.hudi.table.action.clean.CleanPlanner -
Nothing to clean here. It is already clean
100134 [main] INFO org.apache.hudi.client.AbstractHoodieWriteClient -
Cleaner started
100134 [main] INFO org.apache.hudi.client.AbstractHoodieWriteClient -
Cleaned failed attempts if any
100135 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading HoodieTableMetaClient from /home/work/hudi_catalog4
100389 [main] INFO org.apache.hudi.common.table.HoodieTableConfig -
Loading table properties from /home/work/hudi_catalog4/.hoodie/hoodie.properties
100467 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET)
from /home/work/hudi_catalog4
100467 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient -
Loading Active commit timeline for /home/work/hudi_catalog4
100520 [main] INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20211112071954__commit__COMPLETED]}
100520 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating View Manager with storage type :REMOTE_FIRST
100520 [main] INFO org.apache.hudi.common.table.view.FileSystemViewManager
- Creating remote first table view
100523 [main] INFO org.apache.hudi.client.AbstractHoodieClient - Stopping
Timeline service !!
100523 [main] INFO org.apache.hudi.client.embedded.EmbeddedTimelineService
- Closing Timeline server
100523 [main] INFO org.apache.hudi.timeline.service.TimelineService -
Closing Timeline Service
100523 [main] INFO io.javalin.Javalin - Stopping Javalin ...
100531 [main] INFO io.javalin.Javalin - Javalin has stopped
100532 [main] INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 0 ms to
read 0 instants, 0 replaced file groups
100573 [main] INFO org.apache.hudi.common.util.ClusteringUtils - Found 0
files in pending clustering operations
100573 [main] INFO org.apache.hudi.timeline.service.TimelineService -
Closed Timeline Service
100573 [main] INFO org.apache.hudi.client.embedded.EmbeddedTimelineService
- Closed Timeline server
Process finished with exit code 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
