[PR] [HUDI-9334] Optimize Parallelism of show_invalid_parquet [hudi]

Tue, 22 Apr 2025 06:26:18 -0700 


fhan688 opened a new pull request, #13206:
URL: https://github.com/apache/hudi/pull/13206

   ### Change Logs
   
   the former parallelism is based on partition path which may not enough in 
some scenarios.
   this PR optimize this by building parallelism by number of files.
   
   before:
   
![101](https://github.com/user-attachments/assets/af440538-44dd-4878-b255-f882a9579199)
   
   
   after:
   
![102](https://github.com/user-attachments/assets/7bca495e-cfab-4a71-b99b-f11615c00f81)
   
   
   ### Impact
   
   hudi-spark-datasource
   
   ### Risk level (write none, low medium or high below)
   
   low
   
   ### Documentation Update
   none
   
   ### Contributor's checklist
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to