zyclove opened a new issue, #10131: URL: https://github.com/apache/hudi/issues/10131
**Describe the problem you faced** spark sql bulk insert data is too slow , how to turn performance. as https://hudi.apache.org/docs/performance I do change many config, but is not well. **To Reproduce** Steps to reproduce the behavior: 1. spark-sql set hudi config set hoodie.write.lock.zookeeper.lock_key=bi_ods_real.smart_datapoint_report_rw_clear_rt; set spark.sql.hive.filesourcePartitionFileCacheSize=524288000; set hoodie.metadata.table=false; set hoodie.sql.insert.mode=non-strict; set hoodie.sql.bulk.insert.enable=true; set hoodie.populate.meta.fields=false; set hoodie.parquet.compression.codec=snappy; set hoodie.bloom.index.prune.by.ranges=false; set hoodie.file.listing.parallelism=800; set hoodie.cleaner.parallelism=800; set hoodie.insert.shuffle.parallelism=800; set hoodie.upsert.shuffle.parallelism=800; set hoodie.delete.shuffle.parallelism=800; set hoodie.memory.compaction.max.size= 4294967296; set hoodie.memory.merge.max.size=107374182400; 2. sql ``` insert into bi_dw_real.dwd_smart_datapoint_report_rw_clear_rt select /*+ coalesce(${partitions}) */ md5(concat(coalesce(data_id,''),coalesce(dev_id,''),coalesce(gw_id,''),coalesce(product_id,''),coalesce(uid,''),coalesce(dp_code,''),coalesce(dp_id,''),if(dp_mode in ('ro','rw','wr'),dp_mode,'un'),coalesce(dp_name,''),coalesce(dp_time,''),coalesce(dp_type,''),coalesce(dp_value,''),coalesce(ct,''))) as id, _hoodie_record_key as uuid, data_id,dev_id,gw_id,product_id,uid, dp_code,dp_id,if(dp_mode in ('ro','rw','wr'),dp_mode,'un') as dp_mode ,dp_name,dp_time,dp_type,dp_value, ct as gmt_modified, case when length(ct)=10 then date_format(from_unixtime(ct),'yyyyMMddHH') when length(ct)=13 then date_format(from_unixtime(ct/1000),'yyyyMMddHH') else '1970010100' end as dt from hudi_table_changes('bi_ods_real.ods_log_smart_datapoint_report_batch_rt', 'latest_state', '${taskBeginTime}', '${next30minuteTime}') lateral view dataPointExplode(split(value,'\001')[0]) dps as ct, data_id, dev_id, gw_id, product_id, uid, dp_code, dp_id, gmtModified, dp_mode, dp_name, dp_time, dp_type, dp_value where _hoodie_commit_time >${taskBeginTime} and _hoodie_commit_time<=${next30minuteTime}; ``` 3. result table info tblproperties ( type = 'mor', primaryKey = 'id', preCombineField = 'gmt_modified', hoodie.combine.before.upsert='false', hoodie.bucket.index.num.buckets=128, hoodie.compact.inline='false', hoodie.common.spillable.diskmap.type='ROCKS_DB', hoodie.datasource.write.partitionpath.field='dt,dp_mode', hoodie.compaction.payload.class='org.apache.hudi.common.model.PartialUpdateAvroPayload' ) **Expected behavior** A clear and concise description of what you expected to happen. **Environment Description** * Hudi version :0.14.0 * Spark version :3.2.1 * Hive version :3.2.1 * Hadoop version :3.2.2 * Storage (HDFS/S3/GCS..) :s3 * Running on Docker? (yes/no) :no **Additional context**   how to change parallelism? I set spark-sql --conf spark.default.parallelism=800 is not work . The follow config in sql file is not work as expect. ``` set hoodie.file.listing.parallelism=800; set hoodie.cleaner.parallelism=800; set hoodie.insert.shuffle.parallelism=800; set hoodie.upsert.shuffle.parallelism=800; set hoodie.delete.shuffle.parallelism=800; ``` The follow issues are not bulk insert . #8189 #2620 Please take a look and give me some optimization suggestions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
