aznwarmonkey commented on issue #4541:
URL: https://github.com/apache/hudi/issues/4541#issuecomment-1009479559
> sorry I thought you had removed clustering for now. or did you add it back
once it started working?
I have removed it, this is the latest config that i am running with
```python
hudi_options = {
'hoodie.table.name': table_name,
'hoodie.datasource.write.keygenerator.class':
'org.apache.hudi.keygen.ComplexKeyGenerator',
'hoodie.datasource.write.recordkey.field': keys,
'hoodie.datasource.write.partitionpath.field':
','.join(partitions),
'hoodie.datasource.write.hive_style_partitioning': True,
'hoodie.datasource.write.table.name': table_name,
'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
'hoodie.datasource.write.precombine.field': timestamp_col,
'hoodie.index.type': 'BLOOM',
'hoodie.consistency.check.enabled': True,
'hoodie.parquet.small.file.limit': 134217728,
'hoodie.parquet.max.file.size': 1073741824,
'write.bulk_insert.shuffle_by_partition': True,
'hoodie.datasource.write.row.writer.enable': True,
'hoodie.bulkinsert.sort.mode': 'PARTITION_SORT',
'hoodie.bulkinsert.shuffle.parallelism': num_partitons,
'hoodie.cleaner.commits.retained': '1',
'hoodie.clean.async': True,
}
df.write.format('org.apache.hudi') \
.option('hoodie.datasource.write.operation', 'bulk_insert') \
.options(**hudi_options).mode('append').save(output_path)
```
Running on hudi version 0.7 deployed on AWS EMR cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
