rshanmugam1 opened a new issue #2609:
URL: https://github.com/apache/hudi/issues/2609
**Describe the problem you faced**
Presto query performance with hudi table takes ~2x extra time when compared
to parquet for simple query . data stored in s3. hudi metadata store enabled.
note, spark sql much faster with hudi dataset.
*parquet query first attempt*: 23 secs
*parquet query second attempt*: 0.6 secs
*hudi query first attempt*: 40 sec
*hudi query second attempt*: 40 sec
data size : 200gb
also, parquet query faster after first run. we aer in poc to migrate parquet
to Hudi datalake. it is one of the blocker. help appreciated.
**Environment Description**
* Hudi version : master (4a5683d54a3910e1c860f26b291e1e99931b5949)
* Spark version : 2.4.0
* Hive version : 2.3.3
* Hadoop version :2.8.5
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : yes, presto runs on k8
**Additional context**
Presto (trino 352) runs on k8. hudi bundle added.
**Stacktrace**
```
parquet Query (first attempt)
======================================
presto> select y, m, d, h, count(*) from hive.parquet_warehouse.sample_data
where y=2021 and m=2 and d=20 and h between 0 and 4 group by y, m, d, h;
Query 20210227_043723_00000_52ygg, RUNNING, 8 nodes, 6,122 splits
0:23 [ 450M rows, 0B] [19.5M rows/s, 0B/s]
[====================================>>> ] 85%
STAGES ROWS ROWS/s BYTES BYTES/s QUEUED RUN DONE
0.........R 0 0 0B 0B 0 16 0
1.......R 4.81K 207 685K 29.6K 0 256 0
2.....R 450M 19.4M 0B 0B 524 80 5246
y | m | d | h | _col4
------+---+----+---+-----------
2021 | 2 | 20 | 4 | 109245879
2021 | 2 | 20 | 0 | 76442859
2021 | 2 | 20 | 1 | 94755366
2021 | 2 | 20 | 2 | 101223162
2021 | 2 | 20 | 3 | 103258290
(5 rows)
Query 20210227_043723_00000_52ygg, FINISHED, 8 nodes
Splits: 6,183 total, 6,183 done (100.00%)
0:23 [485M rows, 0B] [20.9M rows/s, 0B/s]
parquet query (second attempt)
======================================
presto> select y, m, d, h, count(*) from hive.parquet_warehouse.sample_data
where y=2021 and m=2 and d=20 and h between 0 and 4 group by y, m, d, h;
Query 20210227_043810_00001_52ygg, RUNNING, 10 nodes, 6,215 splits
0:05 [ 475M rows, 0B] [95.6M rows/s, 0B/s]
[=======================================>>>] 91%
STAGES ROWS ROWS/s BYTES BYTES/s QUEUED RUN DONE
0.........R 0 0 0B 0B 0 16 0
1.......R 4.81K 966 686K 138K 0 320 0
2.....R 475M 95.3M 0B 0B 120 94 5665
y | m | d | h | _col4
------+---+----+---+-----------
2021 | 2 | 20 | 3 | 103258290
2021 | 2 | 20 | 2 | 101223162
2021 | 2 | 20 | 0 | 76442859
2021 | 2 | 20 | 1 | 94755366
2021 | 2 | 20 | 4 | 109245879
(5 rows)
Query 20210227_043810_00001_52ygg, FINISHED, 10 nodes
Splits: 6,215 total, 6,215 done (100.00%)
0:05 [485M rows, 0B] [94.2M rows/s, 0B/s]
hudi query (first attempt)
==========================================
presto> select y, m, d, h, count(*) from hive.hudi_warehouse.sample_data
group by y, m, d, h;
Query 20210227_042722_00016_9dket, RUNNING, 10 nodes, 8,443 splits
0:40 [ 473M rows, 0B] [ 12M rows/s, 0B/s] [
<=> ]
STAGES ROWS ROWS/s BYTES BYTES/s QUEUED RUN DONE
0.........R 0 0 0B 0B 0 16 0
1.......R 3.26K 82 464K 11.7K 0 320 0
2.....S 473M 12M 0B 0B 43 18 8046
y | m | d | h | _col4
------+---+----+---+-----------
2021 | 2 | 20 | 1 | 94755289
2021 | 2 | 20 | 4 | 109245869
2021 | 2 | 20 | 0 | 76442808
2021 | 2 | 20 | 2 | 101223154
2021 | 2 | 20 | 3 | 103258285
(5 rows)
Query 20210227_042722_00016_9dket, FINISHED, 10 nodes
Splits: 8,572 total, 8,572 done (100.00%)
0:40 [485M rows, 0B] [12.1M rows/s, 0B/s]
presto logs for hudi Query
================================
2021-02-27T04:27:23.112Z DEBUG dispatcher-query-24
io.prestosql.execution.QueryStateMachine Query
20210227_042722_00016_9dket is QUEUED
2021-02-27T04:27:23.114Z DEBUG dispatcher-query-28
io.prestosql.execution.QueryStateMachine Query
20210227_042722_00016_9dket is WAITING_FOR_RESOURCES
2021-02-27T04:27:23.167Z DEBUG Query-20210227_042722_00016_9dket-664
io.prestosql.plugin.hive.metastore.thrift.ThriftHiveMetastoreClient
Invocation of get_table_req(req=GetTableRequest(dbName:hudi_warehouse,
tblName:sample_data_cluster_2,
capabilities:ClientCapabilities(values:[INSERT_ONLY_TABLES]))) succeeded in
51.56ms
2021-02-27T04:27:23.194Z DEBUG dispatcher-query-25
io.prestosql.execution.QueryStateMachine Query
20210227_042722_00016_9dket is DISPATCHING
2021-02-27T04:27:23.194Z DEBUG dispatcher-query-26
io.prestosql.execution.QueryStateMachine Query
20210227_042722_00016_9dket is PLANNING
2021-02-27T04:27:23.276Z DEBUG Query-20210227_042722_00016_9dket-667
io.prestosql.plugin.hive.metastore.thrift.ThriftHiveMetastoreClient
Invocation of get_partition_names_ps(db_name='hudi_warehouse',
tbl_name='sample_data_cluster_2', part_vals=[, , , ], max_parts=-1) succeeded
in 60.66ms
2021-02-27T04:27:23.364Z DEBUG hive-hive-16
io.prestosql.plugin.hive.metastore.thrift.ThriftHiveMetastoreClient
Invocation of get_partitions_by_names(db_name='hudi_warehouse',
tbl_name='sample_data_cluster_2', names=[y=2021/m=2/d=20/h=4,
y=2021/m=2/d=20/h=3, y=2021/m=2/d=20/h=2, y=2021/m=2/d=20/h=1,
y=2021/m=2/d=20/h=0]) succeeded in 58.46ms
2021-02-27T04:27:23.369Z DEBUG query-execution-13
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.0 is PLANNED
2021-02-27T04:27:23.369Z DEBUG query-execution-13
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.1 is PLANNED
2021-02-27T04:27:23.373Z DEBUG query-execution-14
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.2 is PLANNED
2021-02-27T04:27:23.373Z DEBUG dispatcher-query-25
io.prestosql.execution.QueryStateMachine Query
20210227_042722_00016_9dket is STARTING
2021-02-27T04:27:23.378Z DEBUG query-execution-16
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.2 is SCHEDULING
2021-02-27T04:27:23.379Z DEBUG query-execution-14
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.1 is SCHEDULING
2021-02-27T04:27:23.420Z DEBUG dispatcher-query-26
io.prestosql.execution.QueryStateMachine Query
20210227_042722_00016_9dket is RUNNING
2021-02-27T04:27:23.425Z INFO hive-hive-19
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Reading hoodie metadata
from path s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.426Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.427Z DEBUG query-execution-14
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.1 is SCHEDULED
2021-02-27T04:27:23.427Z INFO hive-hive-19
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.428Z DEBUG query-execution-14
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.0 is SCHEDULING
2021-02-27T04:27:23.428Z DEBUG query-execution-14
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.1 is RUNNING
2021-02-27T04:27:23.429Z INFO hive-hive-17
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Reading hoodie metadata
from path s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.429Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.430Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.432Z INFO hive-hive-18
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Reading hoodie metadata
from path s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.433Z INFO hive-hive-16
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Reading hoodie metadata
from path s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.433Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.433Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.434Z INFO hive-hive-18
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.434Z DEBUG query-execution-14
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.0 is SCHEDULED
2021-02-27T04:27:23.437Z DEBUG query-execution-16
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.0 is RUNNING
2021-02-27T04:27:23.437Z INFO hive-hive-16
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.473Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.473Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.474Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.490Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.502Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.503Z INFO hive-hive-16
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Found a total of 1
groups
2021-02-27T04:27:23.503Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.503Z INFO hive-hive-16
org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based
view for basePath s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.503Z INFO hive-hive-19
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Found a total of 1
groups
2021-02-27T04:27:23.503Z INFO hive-hive-19
org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based
view for basePath s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.523Z INFO hive-hive-19
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED],
[20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED],
[20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.523Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.529Z INFO hive-hive-16
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED],
[20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED],
[20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.530Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.539Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.540Z INFO hive-hive-18
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Found a total of 1
groups
2021-02-27T04:27:23.540Z INFO hive-hive-18
org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based
view for basePath s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.541Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.542Z INFO hive-hive-17
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Found a total of 1
groups
2021-02-27T04:27:23.542Z INFO hive-hive-17
org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based
view for basePath s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.561Z INFO hive-hive-19
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-02-27T04:27:23.561Z INFO hive-hive-18
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED],
[20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED],
[20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.561Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.562Z INFO hive-hive-17
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED],
[20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED],
[20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.562Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.562Z INFO hive-hive-19
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.562Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.567Z INFO hive-hive-16
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-02-27T04:27:23.568Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.569Z INFO hive-hive-16
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.598Z INFO hive-hive-18
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-02-27T04:27:23.598Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.599Z INFO hive-hive-17
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-02-27T04:27:23.599Z INFO hive-hive-18
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.600Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.601Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.604Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.606Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.619Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.619Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.620Z INFO hive-hive-19
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.620Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.620Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.621Z INFO hive-hive-16
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.638Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.639Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.652Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.653Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.655Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.655Z INFO hive-hive-18
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.656Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.657Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.659Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:23.670Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:23.674Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.687Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.695Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:23.696Z INFO hive-hive-16
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[0000000000000__deltacommit__COMPLETED],
[20210227020224__deltacommit__COMPLETED],
[20210227021755__deltacommit__COMPLETED],
[20210227024056__deltacommit__COMPLETED],
[20210227030430__deltacommit__COMPLETED],
[20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.697Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.700Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:23.710Z INFO hive-hive-19
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[0000000000000__deltacommit__COMPLETED],
[20210227020224__deltacommit__COMPLETED],
[20210227021755__deltacommit__COMPLETED],
[20210227024056__deltacommit__COMPLETED],
[20210227030430__deltacommit__COMPLETED],
[20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.710Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.711Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.714Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.729Z INFO hive-hive-18
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[0000000000000__deltacommit__COMPLETED],
[20210227020224__deltacommit__COMPLETED],
[20210227021755__deltacommit__COMPLETED],
[20210227024056__deltacommit__COMPLETED],
[20210227030430__deltacommit__COMPLETED],
[20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.730Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.732Z INFO hive-hive-17
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[0000000000000__deltacommit__COMPLETED],
[20210227020224__deltacommit__COMPLETED],
[20210227021755__deltacommit__COMPLETED],
[20210227024056__deltacommit__COMPLETED],
[20210227030430__deltacommit__COMPLETED],
[20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.733Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.737Z INFO hive-hive-16
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-02-27T04:27:23.737Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (files)
2021-02-27T04:27:23.746Z INFO hive-hive-19
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-02-27T04:27:23.747Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (files)
2021-02-27T04:27:23.767Z INFO hive-hive-18
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-02-27T04:27:23.767Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (files)
2021-02-27T04:27:23.771Z INFO hive-hive-17
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-02-27T04:27:23.772Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (files)
2021-02-27T04:27:23.793Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in
partition (files) =7, Time taken =19
2021-02-27T04:27:23.795Z INFO hive-hive-16
org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding
file-groups for partition :files, #FileGroups=1
2021-02-27T04:27:23.795Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=7, NumFileGroups=1, FileGroupsCreationTime=2, StoreTimeTaken=0
2021-02-27T04:27:23.795Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load
partition (files) =58
2021-02-27T04:27:23.795Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView Pending
Compaction instant for (FileSlice
{fileGroupId=HoodieFileGroupId{partitionPath='files',
fileId='7e172a24-c399-4f87-a9a7-75b3d2879e0b-0'}, baseCommitTime=0000000000000,
baseFile='HoodieBaseFile{fullPath=s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile,
fileLen=5696, BootstrapBaseFile=null}',
logFiles='[HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=25302},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=23561},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metad
ata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=23243},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=21732},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553',
fileLen=27253}]'}) is :Optional.empty
2021-02-27T04:27:23.805Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in
partition (files) =7, Time taken =20
2021-02-27T04:27:23.807Z INFO hive-hive-19
org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding
file-groups for partition :files, #FileGroups=1
2021-02-27T04:27:23.807Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=7, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=1
2021-02-27T04:27:23.807Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load
partition (files) =60
2021-02-27T04:27:23.807Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView Pending
Compaction instant for (FileSlice
{fileGroupId=HoodieFileGroupId{partitionPath='files',
fileId='7e172a24-c399-4f87-a9a7-75b3d2879e0b-0'}, baseCommitTime=0000000000000,
baseFile='HoodieBaseFile{fullPath=s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile,
fileLen=5696, BootstrapBaseFile=null}',
logFiles='[HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=25302},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=23561},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metad
ata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=23243},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=21732},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553',
fileLen=27253}]'}) is :Optional.empty
2021-02-27T04:27:23.815Z INFO hive-hive-16
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED],
[20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED],
[20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.827Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in
partition (files) =7, Time taken =22
2021-02-27T04:27:23.828Z INFO hive-hive-19
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED],
[20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED],
[20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.829Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in
partition (files) =7, Time taken =17
2021-02-27T04:27:23.829Z INFO hive-hive-18
org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding
file-groups for partition :files, #FileGroups=1
2021-02-27T04:27:23.829Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=7, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
2021-02-27T04:27:23.829Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load
partition (files) =62
2021-02-27T04:27:23.830Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView Pending
Compaction instant for (FileSlice
{fileGroupId=HoodieFileGroupId{partitionPath='files',
fileId='7e172a24-c399-4f87-a9a7-75b3d2879e0b-0'}, baseCommitTime=0000000000000,
baseFile='HoodieBaseFile{fullPath=s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile,
fileLen=5696, BootstrapBaseFile=null}',
logFiles='[HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=25302},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=23561},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metad
ata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=23243},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=21732},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553',
fileLen=27253}]'}) is :Optional.empty
2021-02-27T04:27:23.831Z INFO hive-hive-17
org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding
file-groups for partition :files, #FileGroups=1
2021-02-27T04:27:23.831Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=7, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=1
2021-02-27T04:27:23.831Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load
partition (files) =59
2021-02-27T04:27:23.831Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Pending
Compaction instant for (FileSlice
{fileGroupId=HoodieFileGroupId{partitionPath='files',
fileId='7e172a24-c399-4f87-a9a7-75b3d2879e0b-0'}, baseCommitTime=0000000000000,
baseFile='HoodieBaseFile{fullPath=s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile,
fileLen=5696, BootstrapBaseFile=null}',
logFiles='[HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=25302},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=23561},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metad
ata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=23243},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=21732},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553',
fileLen=27253}]'}) is :Optional.empty
2021-02-27T04:27:23.834Z INFO hive-hive-16
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[0000000000000__deltacommit__COMPLETED],
[20210227020224__deltacommit__COMPLETED],
[20210227021755__deltacommit__COMPLETED],
[20210227024056__deltacommit__COMPLETED],
[20210227030430__deltacommit__COMPLETED],
[20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.835Z INFO hive-hive-16
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata table is
disabled.
2021-02-27T04:27:23.835Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (y=2021/m=2/d=20/h=4)
2021-02-27T04:27:23.849Z INFO hive-hive-18
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED],
[20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED],
[20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.849Z INFO hive-hive-19
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[0000000000000__deltacommit__COMPLETED],
[20210227020224__deltacommit__COMPLETED],
[20210227021755__deltacommit__COMPLETED],
[20210227024056__deltacommit__COMPLETED],
[20210227030430__deltacommit__COMPLETED],
[20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.849Z INFO hive-hive-19
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata table is
disabled.
2021-02-27T04:27:23.849Z INFO hive-hive-17
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED],
[20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED],
[20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.850Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (y=2021/m=2/d=20/h=3)
2021-02-27T04:27:23.869Z INFO hive-hive-17
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[0000000000000__deltacommit__COMPLETED],
[20210227020224__deltacommit__COMPLETED],
[20210227021755__deltacommit__COMPLETED],
[20210227024056__deltacommit__COMPLETED],
[20210227030430__deltacommit__COMPLETED],
[20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.870Z INFO hive-hive-17
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata table is
disabled.
2021-02-27T04:27:23.870Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (y=2021/m=2/d=20/h=1)
2021-02-27T04:27:23.872Z INFO hive-hive-18
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[0000000000000__deltacommit__COMPLETED],
[20210227020224__deltacommit__COMPLETED],
[20210227021755__deltacommit__COMPLETED],
[20210227024056__deltacommit__COMPLETED],
[20210227030430__deltacommit__COMPLETED],
[20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.873Z INFO hive-hive-18
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata table is
disabled.
2021-02-27T04:27:23.873Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (y=2021/m=2/d=20/h=2)
2021-02-27T04:27:23.952Z INFO hive-hive-16
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:23.953Z INFO hive-hive-16
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.972Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:23.973Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.983Z INFO hive-hive-19
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:23.983Z INFO hive-hive-19
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.995Z INFO hive-hive-16
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base
file from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile
at instant 0000000000000
2021-02-27T04:27:23.995Z INFO hive-hive-18
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:23.995Z INFO hive-hive-18
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.996Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.997Z INFO hive-hive-16
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:24.034Z INFO hive-hive-17
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base
file from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile
at instant 0000000000000
2021-02-27T04:27:24.036Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.037Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:24.050Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:24.051Z INFO hive-hive-19
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base
file from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile
at instant 0000000000000
2021-02-27T04:27:24.052Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.053Z INFO hive-hive-19
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:24.068Z INFO hive-hive-18
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base
file from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile
at instant 0000000000000
2021-02-27T04:27:24.070Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.070Z INFO hive-hive-16
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.071Z INFO hive-hive-18
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:24.075Z INFO hive-hive-16
org.apache.hudi.common.util.collection.DiskBasedMap Spilling to file
location /tmp/view_map/de468b42-7a11-47fb-a360-fb966c8f8b15 in host
(100.97.190.93) with hostname (presto-coordinator-5fbd7bd6bd-bhx56)
2021-02-27T04:27:24.096Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:24.106Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:24.111Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:24.112Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553',
fileLen=0}
2021-02-27T04:27:24.114Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.114Z INFO hive-hive-17
org.apache.hudi.common.util.collection.DiskBasedMap Spilling to file
location /tmp/view_map/287eb55d-674d-4f7f-82c1-7a766c596e64 in host
(100.97.190.93) with hostname (presto-coordinator-5fbd7bd6bd-bhx56)
2021-02-27T04:27:24.122Z INFO hive-hive-19
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.123Z INFO hive-hive-19
org.apache.hudi.common.util.collection.DiskBasedMap Spilling to file
location /tmp/view_map/2a5b2fb5-5287-4b3b-8c9c-8ce4ebfed736 in host
(100.97.190.93) with hostname (presto-coordinator-5fbd7bd6bd-bhx56)
2021-02-27T04:27:24.125Z INFO hive-hive-18
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.125Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553
at instant 20210227020224
2021-02-27T04:27:24.126Z INFO hive-hive-18
org.apache.hudi.common.util.collection.DiskBasedMap Spilling to file
location /tmp/view_map/52ad2374-1441-48f2-8856-f4838ca8bec5 in host
(100.97.190.93) with hostname (presto-coordinator-5fbd7bd6bd-bhx56)
2021-02-27T04:27:24.140Z INFO hive-hive-16
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=0}
2021-02-27T04:27:24.145Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553',
fileLen=0}
2021-02-27T04:27:24.151Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553',
fileLen=0}
2021-02-27T04:27:24.155Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=0}
2021-02-27T04:27:24.156Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553',
fileLen=0}
2021-02-27T04:27:24.164Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553
at instant 20210227020224
2021-02-27T04:27:24.168Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553
at instant 20210227020224
2021-02-27T04:27:24.171Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553
at instant 20210227020224
2021-02-27T04:27:24.171Z INFO hive-hive-19
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=0}
2021-02-27T04:27:24.176Z INFO hive-hive-18
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=0}
2021-02-27T04:27:24.177Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183
at instant 20210227021755
2021-02-27T04:27:24.178Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.179Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=0}
2021-02-27T04:27:24.184Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=0}
2021-02-27T04:27:24.188Z INFO hive-hive-16
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.189Z INFO hive-hive-16
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.190Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=0}
2021-02-27T04:27:24.191Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=0}
2021-02-27T04:27:24.191Z INFO hive-hive-16
org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated
Payload size => 792
2021-02-27T04:27:24.198Z INFO hive-hive-16
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=0}
2021-02-27T04:27:24.200Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183
at instant 20210227021755
2021-02-27T04:27:24.200Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.204Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183
at instant 20210227021755
2021-02-27T04:27:24.204Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.207Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183
at instant 20210227021755
2021-02-27T04:27:24.208Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.213Z INFO hive-hive-19
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.213Z INFO hive-hive-19
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.215Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=0}
2021-02-27T04:27:24.216Z INFO hive-hive-19
org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated
Payload size => 792
2021-02-27T04:27:24.218Z INFO hive-hive-18
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.218Z INFO hive-hive-18
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.220Z INFO hive-hive-18
org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated
Payload size => 792
2021-02-27T04:27:24.223Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.223Z INFO hive-hive-19
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=0}
2021-02-27T04:27:24.223Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.228Z INFO hive-hive-18
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=0}
2021-02-27T04:27:24.233Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049
at instant 20210227024056
2021-02-27T04:27:24.233Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.235Z INFO hive-hive-17
org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated
Payload size => 792
2021-02-27T04:27:24.239Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=0}
2021-02-27T04:27:24.244Z INFO hive-hive-16
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.245Z INFO hive-hive-16
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.245Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=0}
2021-02-27T04:27:24.247Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=0}
2021-02-27T04:27:24.255Z INFO hive-hive-16
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=0}
2021-02-27T04:27:24.258Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049
at instant 20210227024056
2021-02-27T04:27:24.258Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.262Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049
at instant 20210227024056
2021-02-27T04:27:24.262Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.270Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=0}
2021-02-27T04:27:24.271Z INFO hive-hive-19
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.271Z INFO hive-hive-19
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.275Z INFO hive-hive-18
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.275Z INFO hive-hive-18
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.281Z INFO hive-hive-19
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=0}
2021-02-27T04:27:24.285Z INFO hive-hive-18
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=0}
2021-02-27T04:27:24.288Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=0}
2021-02-27T04:27:24.289Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049
at instant 20210227024056
2021-02-27T04:27:24.289Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.297Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=0}
2021-02-27T04:27:24.301Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=0}
2021-02-27T04:27:24.302Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088
at instant 20210227030430
2021-02-27T04:27:24.302Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.303Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.303Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.313Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=0}
2021-02-27T04:27:24.314Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088
at instant 20210227030430
2021-02-27T04:27:24.314Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.315Z INFO hive-hive-16
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.316Z INFO hive-hive-16
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.326Z INFO hive-hive-16
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=0}
2021-02-27T04:27:24.326Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088
at instant 20210227030430
2021-02-27T04:27:24.326Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.327Z INFO hive-hive-19
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.327Z INFO hive-hive-19
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.333Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=0}
2021-02-27T04:27:24.338Z INFO hive-hive-19
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=0}
2021-02-27T04:27:24.340Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=0}
2021-02-27T04:27:24.348Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088
at instant 20210227030430
2021-02-27T04:27:24.348Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.353Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093
at instant 20210227032808
2021-02-27T04:27:24.354Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.356Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=0}
2021-02-27T04:27:24.356Z INFO hive-hive-18
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.357Z INFO hive-hive-18
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.372Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.372Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.373Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093
at instant 20210227032808
2021-02-27T04:27:24.373Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.382Z INFO hive-hive-16
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.382Z INFO hive-hive-16
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.383Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=0}
2021-02-27T04:27:24.384Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the
final data blocks
2021-02-27T04:27:24.384Z INFO hive-hive-16
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.387Z INFO hive-hive-19
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.387Z INFO hive-hive-19
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.390Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the
final data blocks
2021-02-27T04:27:24.390Z INFO hive-hive-19
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.397Z INFO hive-hive-18
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=0}
2021-02-27T04:27:24.398Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=0}
2021-02-27T04:27:24.413Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=0}
2021-02-27T04:27:24.416Z INFO hive-hive-16
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.416Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093
at instant 20210227032808
2021-02-27T04:27:24.416Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.416Z INFO hive-hive-16
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.418Z INFO hive-hive-16
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log
files scanned => 5
2021-02-27T04:27:24.419Z INFO hive-hive-16
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner
MaxMemoryInBytes allowed for compaction => 0
2021-02-27T04:27:24.419Z INFO hive-hive-16
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in MemoryBasedMap in ExternalSpillableMap => 6
2021-02-27T04:27:24.419Z INFO hive-hive-16
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in
bytes of MemoryBasedMap in ExternalSpillableMap => 4752
2021-02-27T04:27:24.419Z INFO hive-hive-16
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in DiskBasedMap in ExternalSpillableMap => 0
2021-02-27T04:27:24.419Z INFO hive-hive-16
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file
spilled to disk => 0
2021-02-27T04:27:24.419Z INFO hive-hive-16
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log
files from
[s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093]
at instant 20210227032808(dataset instant=20210227032808, metadata
instant=20210227032808)
2021-02-27T04:27:24.422Z INFO hive-hive-19
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.422Z INFO hive-hive-19
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.425Z INFO hive-hive-19
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log
files scanned => 5
2021-02-27T04:27:24.425Z INFO hive-hive-19
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner
MaxMemoryInBytes allowed for compaction => 0
2021-02-27T04:27:24.425Z INFO hive-hive-19
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in MemoryBasedMap in ExternalSpillableMap => 6
2021-02-27T04:27:24.425Z INFO hive-hive-19
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in
bytes of MemoryBasedMap in ExternalSpillableMap => 4752
2021-02-27T04:27:24.425Z INFO hive-hive-19
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in DiskBasedMap in ExternalSpillableMap => 0
2021-02-27T04:27:24.426Z INFO hive-hive-19
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file
spilled to disk => 0
2021-02-27T04:27:24.426Z INFO hive-hive-19
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log
files from
[s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093]
at instant 20210227032808(dataset instant=20210227032808, metadata
instant=20210227032808)
2021-02-27T04:27:24.430Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093
at instant 20210227032808
2021-02-27T04:27:24.431Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.435Z INFO hive-hive-16
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key
y=2021/m=2/d=20/h=4 took [open, baseFileRead, logMerge] [467, 16, 0] ms
2021-02-27T04:27:24.436Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.436Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.439Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the
final data blocks
2021-02-27T04:27:24.439Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.440Z INFO hive-hive-19
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key
y=2021/m=2/d=20/h=3 took [open, baseFileRead, logMerge] [443, 14, 0] ms
2021-02-27T04:27:24.445Z INFO hive-hive-16
org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from
metadata: partition=y=2021/m=2/d=20/h=4, #files=761
2021-02-27T04:27:24.445Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in
partition (y=2021/m=2/d=20/h=4) =761, Time taken =493
2021-02-27T04:27:24.450Z INFO hive-hive-19
org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from
metadata: partition=y=2021/m=2/d=20/h=3, #files=699
2021-02-27T04:27:24.450Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in
partition (y=2021/m=2/d=20/h=3) =699, Time taken =467
2021-02-27T04:27:24.452Z INFO hive-hive-18
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.452Z INFO hive-hive-18
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.459Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.460Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.462Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the
final data blocks
2021-02-27T04:27:24.462Z INFO hive-hive-18
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:24.462Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log
files scanned => 5
2021-02-27T04:27:24.463Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner
MaxMemoryInBytes allowed for compaction => 0
2021-02-27T04:27:24.463Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in MemoryBasedMap in ExternalSpillableMap => 6
2021-02-27T04:27:24.463Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in
bytes of MemoryBasedMap in ExternalSpillableMap => 4752
2021-02-27T04:27:24.463Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in DiskBasedMap in ExternalSpillableMap => 0
2021-02-27T04:27:24.463Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file
spilled to disk => 0
2021-02-27T04:27:24.463Z INFO hive-hive-17
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log
files from
[s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093]
at instant 20210227032808(dataset instant=20210227032808, metadata
instant=20210227032808)
2021-02-27T04:27:24.480Z INFO hive-hive-17
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key
y=2021/m=2/d=20/h=1 took [open, baseFileRead, logMerge] [491, 17, 0] ms
2021-02-27T04:27:24.487Z INFO hive-hive-18
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.487Z INFO hive-hive-18
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.491Z INFO hive-hive-17
org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from
metadata: partition=y=2021/m=2/d=20/h=1, #files=647
2021-02-27T04:27:24.491Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in
partition (y=2021/m=2/d=20/h=1) =647, Time taken =519
2021-02-27T04:27:24.501Z INFO hive-hive-16
org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding
file-groups for partition :y=2021/m=2/d=20/h=4, #FileGroups=761
2021-02-27T04:27:24.501Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=761, NumFileGroups=761, FileGroupsCreationTime=54, StoreTimeTaken=1
2021-02-27T04:27:24.501Z INFO hive-hive-16
org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load
partition (y=2021/m=2/d=20/h=4) =666
2021-02-27T04:27:24.526Z INFO hive-hive-18
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log
files scanned => 5
2021-02-27T04:27:24.527Z INFO hive-hive-18
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner
MaxMemoryInBytes allowed for compaction => 0
2021-02-27T04:27:24.527Z INFO hive-hive-18
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in MemoryBasedMap in ExternalSpillableMap => 6
2021-02-27T04:27:24.527Z INFO hive-hive-18
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in
bytes of MemoryBasedMap in ExternalSpillableMap => 4752
2021-02-27T04:27:24.527Z INFO hive-hive-18
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in DiskBasedMap in ExternalSpillableMap => 0
2021-02-27T04:27:24.527Z INFO hive-hive-18
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file
spilled to disk => 0
2021-02-27T04:27:24.531Z INFO hive-hive-18
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log
files from
[s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093]
at instant 20210227032808(dataset instant=20210227032808, metadata
instant=20210227032808)
2021-02-27T04:27:24.551Z INFO hive-hive-18
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key
y=2021/m=2/d=20/h=2 took [open, baseFileRead, logMerge] [537, 19, 0] ms
2021-02-27T04:27:24.560Z INFO hive-hive-18
org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from
metadata: partition=y=2021/m=2/d=20/h=2, #files=691
2021-02-27T04:27:24.560Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in
partition (y=2021/m=2/d=20/h=2) =691, Time taken =566
2021-02-27T04:27:24.563Z INFO hive-hive-19
org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding
file-groups for partition :y=2021/m=2/d=20/h=3, #FileGroups=699
2021-02-27T04:27:24.563Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=699, NumFileGroups=699, FileGroupsCreationTime=113, StoreTimeTaken=0
2021-02-27T04:27:24.563Z INFO hive-hive-19
org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load
partition (y=2021/m=2/d=20/h=3) =713
2021-02-27T04:27:24.584Z INFO hive-hive-17
org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding
file-groups for partition :y=2021/m=2/d=20/h=1, #FileGroups=647
2021-02-27T04:27:24.584Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=647, NumFileGroups=647, FileGroupsCreationTime=92, StoreTimeTaken=1
2021-02-27T04:27:24.584Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load
partition (y=2021/m=2/d=20/h=1) =714
2021-02-27T04:27:24.609Z INFO hive-hive-19
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Total paths to process
after hoodie filter 699
2021-02-27T04:27:24.615Z INFO hive-hive-16
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Total paths to process
after hoodie filter 761
2021-02-27T04:27:24.631Z INFO hive-hive-17
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Total paths to process
after hoodie filter 647
2021-02-27T04:27:24.670Z INFO hive-hive-18
org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding
file-groups for partition :y=2021/m=2/d=20/h=2, #FileGroups=691
2021-02-27T04:27:24.670Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=691, NumFileGroups=691, FileGroupsCreationTime=110, StoreTimeTaken=0
2021-02-27T04:27:24.671Z INFO hive-hive-18
org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load
partition (y=2021/m=2/d=20/h=2) =798
2021-02-27T04:27:24.714Z INFO hive-hive-18
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Total paths to process
after hoodie filter 691
2021-02-27T04:27:45.364Z INFO hive-hive-17
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Reading hoodie metadata
from path s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.364Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.365Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:45.410Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:45.426Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.427Z INFO hive-hive-17
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Found a total of 1
groups
2021-02-27T04:27:45.427Z INFO hive-hive-17
org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based
view for basePath s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.453Z INFO hive-hive-17
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED],
[20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED],
[20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:45.453Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-02-27T04:27:45.492Z INFO hive-hive-17
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-02-27T04:27:45.492Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.493Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:45.531Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:45.547Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.547Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:45.548Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:45.585Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:45.599Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:45.618Z INFO hive-hive-17
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[0000000000000__deltacommit__COMPLETED],
[20210227020224__deltacommit__COMPLETED],
[20210227021755__deltacommit__COMPLETED],
[20210227024056__deltacommit__COMPLETED],
[20210227030430__deltacommit__COMPLETED],
[20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:45.618Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to
read 0 instants, 0 replaced file groups
2021-02-27T04:27:45.652Z INFO hive-hive-17
org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending
clustering operations
2021-02-27T04:27:45.652Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (files)
2021-02-27T04:27:45.709Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in
partition (files) =7, Time taken =18
2021-02-27T04:27:45.710Z INFO hive-hive-17
org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding
file-groups for partition :files, #FileGroups=1
2021-02-27T04:27:45.710Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=7, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
2021-02-27T04:27:45.711Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load
partition (files) =59
2021-02-27T04:27:45.711Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Pending
Compaction instant for (FileSlice
{fileGroupId=HoodieFileGroupId{partitionPath='files',
fileId='7e172a24-c399-4f87-a9a7-75b3d2879e0b-0'}, baseCommitTime=0000000000000,
baseFile='HoodieBaseFile{fullPath=s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile,
fileLen=5696, BootstrapBaseFile=null}',
logFiles='[HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=25302},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=23561},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metad
ata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=23243},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=21732},
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553',
fileLen=27253}]'}) is :Optional.empty
2021-02-27T04:27:45.732Z INFO hive-hive-17
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED],
[20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED],
[20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:45.756Z INFO hive-hive-17
org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants
[[0000000000000__deltacommit__COMPLETED],
[20210227020224__deltacommit__COMPLETED],
[20210227021755__deltacommit__COMPLETED],
[20210227024056__deltacommit__COMPLETED],
[20210227030430__deltacommit__COMPLETED],
[20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:45.756Z INFO hive-hive-17
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata table is
disabled.
2021-02-27T04:27:45.756Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file
system view for partition (y=2021/m=2/d=20/h=0)
2021-02-27T04:27:45.899Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:45.899Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:45.953Z INFO hive-hive-17
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base
file from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile
at instant 0000000000000
2021-02-27T04:27:45.954Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Loading
HoodieTableMetaClient from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:45.954Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[hdfs://hdfs-service], Config:[Configuration: ], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:46.012Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableConfig Loading table properties from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:46.030Z INFO hive-hive-17
org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table
of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:46.035Z INFO hive-hive-17
org.apache.hudi.common.util.collection.DiskBasedMap Spilling to file
location /tmp/view_map/c949fce3-e000-4ef4-b7c6-17ee139de8e3 in host
(100.97.190.93) with hostname (presto-coordinator-5fbd7bd6bd-bhx56)
2021-02-27T04:27:46.064Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553',
fileLen=0}
2021-02-27T04:27:46.082Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553
at instant 20210227020224
2021-02-27T04:27:46.086Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=0}
2021-02-27T04:27:46.102Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183',
fileLen=0}
2021-02-27T04:27:46.121Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183
at instant 20210227021755
2021-02-27T04:27:46.121Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:46.134Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:46.134Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:46.137Z INFO hive-hive-17
org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated
Payload size => 792
2021-02-27T04:27:46.140Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=0}
2021-02-27T04:27:46.154Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049',
fileLen=0}
2021-02-27T04:27:46.168Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049
at instant 20210227024056
2021-02-27T04:27:46.168Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:46.179Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:46.180Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:46.184Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=0}
2021-02-27T04:27:46.200Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088',
fileLen=0}
2021-02-27T04:27:46.214Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088
at instant 20210227030430
2021-02-27T04:27:46.214Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:46.225Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:46.226Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:46.230Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next
reader for logfile
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=0}
2021-02-27T04:27:46.245Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log
file
HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093',
fileLen=0}
2021-02-27T04:27:46.264Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data
block from file
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093
at instant 20210227032808
2021-02-27T04:27:46.265Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:46.276Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:46.277Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:46.279Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the
final data blocks
2021-02-27T04:27:46.279Z INFO hive-hive-17
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of
remaining logblocks to merge 1
2021-02-27T04:27:46.291Z INFO hive-hive-17
org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS:
[file:///], Config:[Configuration: core-default.xml, core-site.xml,
hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml], FileSystem:
[org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:46.291Z INFO hive-hive-17
org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=12497808,
freeSize=12174472304, maxSize=12186970112, heapSize=12497808,
minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5,
singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:46.293Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log
files scanned => 5
2021-02-27T04:27:46.293Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner
MaxMemoryInBytes allowed for compaction => 0
2021-02-27T04:27:46.293Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in MemoryBasedMap in ExternalSpillableMap => 6
2021-02-27T04:27:46.293Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in
bytes of MemoryBasedMap in ExternalSpillableMap => 4752
2021-02-27T04:27:46.293Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of
entries in DiskBasedMap in ExternalSpillableMap => 0
2021-02-27T04:27:46.293Z INFO hive-hive-17
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file
spilled to disk => 0
2021-02-27T04:27:46.294Z INFO hive-hive-17
org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log
files from
[s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088,
s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093]
at instant 20210227032808(dataset instant=20210227032808, metadata
instant=20210227032808)
2021-02-27T04:27:46.309Z INFO hive-hive-17
org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key
y=2021/m=2/d=20/h=0 took [open, baseFileRead, logMerge] [396, 15, 0] ms
2021-02-27T04:27:46.314Z INFO hive-hive-17
org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from
metadata: partition=y=2021/m=2/d=20/h=0, #files=623
2021-02-27T04:27:46.315Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in
partition (y=2021/m=2/d=20/h=0) =623, Time taken =417
2021-02-27T04:27:46.337Z INFO hive-hive-17
org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding
file-groups for partition :y=2021/m=2/d=20/h=0, #FileGroups=623
2021-02-27T04:27:46.338Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView:
NumFiles=623, NumFileGroups=623, FileGroupsCreationTime=22, StoreTimeTaken=1
2021-02-27T04:27:46.338Z INFO hive-hive-17
org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load
partition (y=2021/m=2/d=20/h=0) =582
2021-02-27T04:27:46.360Z INFO hive-hive-17
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Total paths to process
after hoodie filter 623
2021-02-27T04:28:02.931Z DEBUG query-execution-16
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.2 is SCHEDULED
2021-02-27T04:28:02.934Z DEBUG query-execution-16
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.2 is RUNNING
2021-02-27T04:28:03.016Z DEBUG query-execution-16
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.2 is FINISHED
2021-02-27T04:28:03.024Z DEBUG query-execution-16
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.1 is FINISHED
2021-02-27T04:28:03.026Z DEBUG query-execution-16
io.prestosql.execution.StageStateMachine Stage
20210227_042722_00016_9dket.0 is FINISHED
2021-02-27T04:28:03.026Z DEBUG dispatcher-query-26
io.prestosql.execution.QueryStateMachine Query
20210227_042722_00016_9dket is FINISHING
2021-02-27T04:28:03.027Z DEBUG dispatcher-query-26
io.prestosql.execution.QueryStateMachine Query
20210227_042722_00016_9dket is FINISHED
2021-02-27T04:28:03.053Z INFO dispatcher-query-26
io.prestosql.event.QueryMonitor TIMELINE: Query 20210227_042722_00016_9dket ::
Transaction:[13b23144-6f70-42e6-b876-4a69ca8744f3] :: elapsed 39914ms ::
planning 179ms :: waiting 80ms :: scheduling 1724ms :: running 37991ms ::
finishing 20ms :: begin 2021-02-27T04:27:23.112Z :: end 2021-02-27T04:28:03.026Z
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
