asheeshgarg commented on issue #1825:
URL: https://github.com/apache/hudi/issues/1825#issuecomment-659001436
@bvaradar Balaji I tried the mentioned property but doesn't see the impact
still see parquet generated
2020-07-15 20:41:40 478.6 KiB
65254296-10d0-49d4-b168-6708e6274712-0_0-30-724_20200715204132.parquet
2020-07-15 20:29:35 456.3 KiB
65254296-10d0-49d4-b168-6708e6274712-0_0-30-819_20200715202927.parquet
Here are my writer hudi configs
hudi_options = {
| "hoodie.table.name": self.table_name,
| "hoodie.datasource.write.table.type": "COPY_ON_WRITE",
| "hoodie.datasource.write.keygenerator.class":
"org.apache.hudi.keygen.ComplexKeyGenerator",
| "hoodie.datasource.write.recordkey.field":
"dl_snapshot_date,dl_dataset,column",
| "hoodie.datasource.write.precombine.field": "column",
| "hoodie.datasource.write.table.name": self.table_name,
| "hoodie.copyonwrite.record.size.estimate":128,
| "hoodie.parquet.small.file.limit": 500*1024*1024,
| "hoodie.parquet.max.file.size": 2*1024*1024*1024,
| "hoodie.upsert.shuffle.parallelism": 2,
| "hoodie.insert.shuffle.parallelism": 2,
| "hoodie.embed.timeline.server": False,
| "hoodie.datasource.write.partitionpath.field": "dl_snapshot_date",
| }
Could you please suggest if anything is wrong?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
