[
https://issues.apache.org/jira/browse/HUDI-3660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537510#comment-17537510
]
Sagar Sumit commented on HUDI-3660:
-----------------------------------
[~yuangzhang] the config takes value in bytes. So, if you want to set to 10mb
then set
'hoodie.logfile.max.size'='10485760'
However, that's too small value for log file. With just 10mb you will end up
writing too many small log files and thus more compaction work needs to be
done. Any particular reason why such small value for max log file size?
> config hoodie.logfile.max.size not work
> ---------------------------------------
>
> Key: HUDI-3660
> URL: https://issues.apache.org/jira/browse/HUDI-3660
> Project: Apache Hudi
> Issue Type: Bug
> Components: configs
> Reporter: YuAngZhang
> Priority: Blocker
> Fix For: 0.11.1
>
> Attachments: log.jpg
>
>
> log file not rollover when file size more than 10M
> it seems the method HoodieLogFormatWriter.rolloverIfNeeded not work, the
> file system wrapped to HoodieWrapperFileSystem，the pos of
> FSDataInputStream always set to 0
> {code:java}
> SET 'execution.checkpointing.interval' = '30min';
> CREATE TABLE sink(
> role_id VARCHAR(20),
> log_id VARCHAR(10),
> origin_json string,
> ts TIMESTAMP(3),
> ds date,
> `ds` date
> )
> PARTITIONED BY (`ds`)
> WITH (
> 'connector' = 'hudi',
> 'path' = 'hdfs:///user/dl/hudi_nsh/',
> 'table.type' = 'MERGE_ON_READ',
> 'compaction.trigger.strategy'='num_commits',
> 'compaction.delta_commits'='5',
> 'hoodie.cleaner.commits.retained'='1',
> 'hoodie.datasource.write.recordkey.field'='role_id,log_id,ts',
> 'write.batch.size'='10',
> 'hoodie.logfile.max.size'='10',
> 'hive_sync.enable'='true',
> 'hive_sync.mode' = 'hms',
> 'hive_sync.metastore.uris' = 'thrift://fuxi-luoge-148:9083',
> 'hive_sync.jdbc_url'='jdbc:hive2://',
> 'hive_sync.table'='sink5',
> 'hive_sync.db'='test',
> 'hive_sync.username'='',
> 'hive_sync.password'=''
> ); {code}
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
