shardulm94 commented on PR #40637:
URL: https://github.com/apache/spark/pull/40637#issuecomment-1496640034
Thanks @ShreyeshArangath for this! I think it helps clear a lot of
unnecessary noise from user logs and keeps the logs manageable.
One thing I noticed is that we set `spark.yarn.report.logging.frequency` to
`30` by default. I think it is a much more sensible default, especially given
a) Spark jobs submitted to YARN would usually run longer than a minute
including the overhead of launching Spark driver b) We always report any state
change immediately e.g. `ACCEPTED` -> `RUNNING`. I would be pro defaulting to
`30`. But if we want to eliminate behavior change, we can always default it to
`1`.
I saw the tests are failing with what seems to be a transient Github
connection issue.
```
/usr/bin/git -c protocol.version=2 fetch --prune --progress
--no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/*
+refs/tags/*:refs/tags/*
Error: fatal: unable to access 'https://github.com/apache/spark/': The
requested URL returned error: 429
The process '/usr/bin/git' failed with exit code 128
```
Can you push an empty commit to re-run the tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
