luffyd commented on issue #1913: URL: https://github.com/apache/hudi/issues/1913#issuecomment-669674913
Have built jars from master branch, latest commit at the time of build was ``` commit 8c4ff185f1752b5041c4e66ac595bd90c2693137 (HEAD -> master, 0r) Author: mabin001 <[email protected]> Date: Tue Jul 7 19:10:16 2020 +0800 [HUDI-1064]Trim hoodie table name (#1805) ``` I did see successful commits, after a certain point of time it crashed. I tried to look at the spark UI, it did not load and was slow. Moving from driver mode to cluster mode, I am not noticing this issue. But I am curious on what is driver doing by accessing Filesystem, even if there is cache. Why does driver need to look at these Files? s3://dev-anushb-emr/publicLogs/tooManyFiles-stderr.gz -- Please ignore log lines containing luffyd, I have added them to figure the s3 call pattern s3://dev-anushb-emr/publicLogs/tooManyFiles-hoodieLog ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
