zyclove commented on issue #10131: URL: https://github.com/apache/hudi/issues/10131#issuecomment-1818173326
@ad1happy2go The table bucket is 128. hoodie.bucket.index.num.buckets=128 When I use the bulk model, why do the result tables generate so many small files (some hundreds of thousands of small files) and they are not generated according to the default 128, which has a huge impact on performance and cost. How should I optimize this?  ``` set hoodie.metadata.table=false; set hoodie.sql.insert.mode=non-strict; set hoodie.sql.bulk.insert.enable=true; set hoodie.populate.meta.fields=false; set hoodie.parquet.compression.codec=snappy; set hoodie.bloom.index.prune.by.ranges=false; set hoodie.file.listing.parallelism=200; set hoodie.cleaner.parallelism=200; set hoodie.insert.shuffle.parallelism=200; set hoodie.upsert.shuffle.parallelism=200; set hoodie.delete.shuffle.parallelism=200; set hoodie.bulkinsert.shuffle.parallelism=200; ``` But when I don't turn on the bulk mode, I can basically generate files according to the number of buckets normally, but the performance is also relatively slow, and OMM sometimes occurs.  ``` Remove the above configuration ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
