glory9211 commented on issue #6107:
URL: https://github.com/apache/hudi/issues/6107#issuecomment-1195603690
> > looks like something to do with meta sync where RO is not getting
sync'ed. please provide scripts and configs for reproducing then we can help
from there.
>
> please find below configuration which we are using currently
>
> hudi_options = { 'hoodie.datasource.write.table.type':
self._write_table_type, 'hoodie.table.name': self._table_name,
'hoodie.datasource.write.recordkey.field': self._record_key,
'hoodie.datasource.write.partitionpath.field': self._partition_field,
'hoodie.datasource.write.precombine.field': self._combine_key,
'hoodie.datasource.write.keygenerator.class':
'org.apache.hudi.keygen.ComplexKeyGenerator', 'hoodie.parquet.max.file.size':
"20971520", 'hoodie.datasource.hive_sync.enable': 'true',
'hoodie.datasource.hive_sync.table': self._table_name.lower(),
'hoodie.datasource.hive_sync.partition_fields': self._partition_field,
'hoodie.datasource.hive_sync.partition_extractor_class':
'org.apache.hudi.hive.MultiPartKeysValueExtractor',
'hoodie.datasource.hive_sync.database': self._hive_database.lower(),
'hoodie.datasource.write.hive_style_partitioning': 'true',
'hoodie.datasource.hive_sync.mode': 'hms',
'hoodie.datasource.hive_sync.support_timestamp': 'true' }
As mentioned by @KnightChess RT and RO tables are synced when you run
compaction on Hudi MOR tables.
i.e. The Delta (Avro) Files are merged into the Parquet Files. In HUDI
COW Tables == Data in Parquet Files
MOR Tables == Data in Avro + Parquet Files
You can read the configs from the docs
[here](https://hudi.apache.org/docs/configurations/)
Some sample configs you should provide
```
## Compaction
'hoodie.compact.inline.max.delta.seconds' : 60,
'hoodie.compact.inline.max.delta.commits' : 4,
'hoodie.compact.inline.trigger.strategy' : 'NUM_OR_TIME',
'hoodie.compact.inline' : True,
'hoodie.datasource.compaction.async.enable' : True,
```
This will trigger compaction after every 60 seconds or 4 delta commits for a
streaming job.
Read more about what is compaction in Hudi
[here](https://hudi.apache.org/docs/compaction)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
