jiaojietao opened a new issue, #8964:
URL: https://github.com/apache/hudi/issues/8964
**To Reproduce**
Steps to reproduce the behavior:
1.**Expected behavior**
1)behavior
```2023-06-14 13:45:30,484 INFO org.apache.hadoop.io.compress.CodecPool
[] - Got brand-new compressor [.gz]
2023-06-14 13:45:30,922 INFO org.apache.hudi.io.HoodieCreateHandle
[] - New CreateHandle for partition : with fileId
1530c350-1cc7-474b-b36a-2089799a2ea5
2023-06-14 13:45:30,928 INFO org.apache.hudi.io.HoodieCreateHandle
[] - Closing the file 1530c350-1cc7-474b-b36a-2089799a2ea5 as we
are done with all the records 3
2023-06-14 13:45:30,941 ERROR org.apache.hudi.sink.compact.CompactFunction
[] - Executor executes action [Execute compaction for instant
20230606150932822 from task 0] error
java.lang.NoSuchMethodError:
shaded.parquet.org.apache.thrift.EncodingUtils.setBit(BIZ)B
at
org.apache.parquet.format.PageHeader.setUncompressed_page_sizeIsSet(PageHeader.java:340)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at org.apache.parquet.format.PageHeader.<init>(PageHeader.java:230)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.parquet.format.converter.ParquetMetadataConverter.newDataPageHeader(ParquetMetadataConverter.java:1746)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.parquet.format.converter.ParquetMetadataConverter.writeDataPageV1Header(ParquetMetadataConverter.java:1828)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:186)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:59)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.parquet.column.impl.ColumnWriterBase.writePage(ColumnWriterBase.java:387)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.parquet.column.impl.ColumnWriteStoreBase.flush(ColumnWriteStoreBase.java:186)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:29)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:185)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:124)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:319)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.hudi.io.storage.HoodieAvroParquetWriter.close(HoodieAvroParquetWriter.java:84)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.hudi.io.HoodieCreateHandle.close(HoodieCreateHandle.java:211)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleInsert(HoodieFlinkCopyOnWriteTable.java:419)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:231)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.hudi.sink.compact.CompactFunction.doCompaction(CompactFunction.java:110)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.hudi.sink.compact.CompactFunction.lambda$processElement$0(CompactFunction.java:95)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
~[hudi-flink1.14-bundle-0.12.0.jar:0.12.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_301]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_301]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
2023-06-14 13:45:30,946 INFO
org.apache.hudi.common.table.HoodieTableMetaClient [] - Loading
HoodieTableMetaClient from hdfs:///flink_hudi/hudi_shipments
```
2) use example
```create table hudi_shipments(
shipment_id INT,
order_id INT,
origin STRING,
destination STRING,
is_arrived BOOLEAN,
PRIMARY KEY (shipment_id) NOT ENFORCED
)
with(
'connector'='hudi',
'path'='hdfs:///flink_hudi/hudi_shipments',
'hoodie.datasource.write.recordkey.field'='id',
'hoodie.parquet.max.file.size'='268435456',
'write.tasks'='4',
'write.bucket_assign.tasks'='1',
'write.task.max.size'='1024',
'write.rate.limit'='30000',
'table.type'='MERGE_ON_READ',
'compaction.tasks'='1',
'compaction.async.enabled'='true',
'compaction.delta_commits'='5',
'compaction.max_memory'='1024',
'compaction.schedule.enabled' = 'true',
'compaction.trigger.strategy' = 'num_commits',
'changelog.enabled' = 'true',
'read.streaming.enabled' = 'true',
'read.streaming.skip_compaction' = 'true'
);
insert into hudi_shipments select* from shipments;
```
2.**Environment Description**
* Hudi version :
0.12.0
* Hadoop version :
CDH5.13.3
* Storage (HDFS/S3/GCS..) :
HDFS
* Running on Docker? (yes/no) :
no
* flink
1.14
3.**Additional context**
Manual compression successful, automatic compression failed
4.**Stacktrace**
Heap and stack have no memory overflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
