[GitHub] [hudi] BalaMahesh commented on issue #7595: [SUPPORT] Hudi Clean and Delta commits taking ~50 mins to finish frequently

[via GitHub]

BalaMahesh commented on issue #7595:
URL: https://github.com/apache/hudi/issues/7595#issuecomment-1464816071

   > > We were running with
   > > hoodie.metadata.index.bloom.filter.enable=false 
hoodie.metadata.index.column.stats.enable=false
   > > I hope that answers your question. We have set this false because, I 
have run into this issue when set to true. #7657
   > > I have changed the index type to simple and then restarted the 
application.
   > > Index look up duration has come down and uniform now. <img 
alt="Screenshot 2023-01-13 at 10 29 51 AM" width="671" 
src="https://user-images.githubusercontent.com/25053668/212241200-1284af49-d728-432f-a1ba-a72e1ed50dbe.png";>
   > > delta commit durations are uniform too except for the issue I mentioned 
here #7364. After the restart delta commit gets stuck and then later progresses.
   > > <img alt="Screenshot 2023-01-13 at 10 30 00 AM" width="672" 
src="https://user-images.githubusercontent.com/25053668/212241373-1f67ca15-7e5d-4bdc-b9a1-ced25a94ba68.png";>
   > 
   > @BalaMahesh Hello, BalaMahesh, we met same issue here, I have two 
questions here, hope you could have chance to response. Thanks in advance !
   > 
   > * how to find charts of "Index lookup duration" ?
   > * simple is the default index type on Spark engine, and we are using 
simple index type on our Custom Spark job, seems simple index type is not the 
key solution. do you have any more updates ?
   > 
   > Thanks in advance again !
   
   1. You have to push metrics to either Prometheus push gateway or any other 
monitoring solution for plotting these charts. 
   2. You can see the spark stages and jobs in the spark web ui to identify 
which task is taking long. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org