HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-455739953
@squito
I just added new config to toggle applying custom log url for incomplete
applications.
(`spark.histor
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-46750
Sorry to trigger build too frequently: now I'm done with modifying. Waiting
the build result.
---
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-455539926
Fixed UTs as well as added a new test to verify custom log URL work. Ended
up with adding `attributes` to Executor
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-455430715
I'll work on fixing UTs: not sure why `@JsonIgnore` doesn't work. Also I'm
testing manually on YARN cluster as wel
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-455388759
Also cc. to @gerashegalov to see whether this approach can also cover the
case in #20326 as well.
---
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-455387835
Just modified to let HistoryAppStatusStore intercept and modify executor log
URLs.
I guess this will avoid
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-455373732
@squito
> The one thing I am wondering about is if we should change it so that when
the SHS is looking at an i
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-455328233
@squito Yes, exactly. Nice summarization.
Btw, I didn't add variables for other resource managers as they do
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-455157050
Retest this, please
This is an automated message
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-455088037
Verified with YARN cluster as well.
> Don't set `spark.history.custom.executor.log.url` (default)
```
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-455032458
Updated with new approach:
- Only SHS provides custom executor log URLs (via instrumenting events a bit
whi
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-454656384
@vanzin Thanks for confirming! I'll come up with updates which applies your
suggestion.
(Btw I guess I found th
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-453805982
Sorry to revisit this lately: I had to tackle some other tasks as well, so
spent time to do simpler thing first.
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-451871129
I'm afraid I'm over-thinking about this, but still don't feel easy to tackle
it: not just moving logic to SHS, but
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-449665436
I'm OK to think about having general solution / different URLs between UI in
driver / SHS, except parameters.
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-448029536
OK. I guess I'm seeing what we concern about. Looks like we don't want to
set specific URL to env which could be a
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-447525509
@squito
If we don't touch the default value it is just running as it is for now
because the default URL is set
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-447197369
@squito
> I also wonder if this feature should not be so yarn-specific then -- in
fact, it almost seems mo
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-446753861
It's something private for now. Btw, I feel this as egg and chicken problem.
Once we open flexibility 3rd party ca
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-446738674
@squito
> Or maybe I'm still not quite following, and there is some 3rd party piece
here, outside of spark & y
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-446418872
@vanzin
IMHO we can't go with approach #20326 went since cluster ID is not available
in default log URL so can
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-446387143
> My understanding is that this allows pointing the Spark UI directly at the
history server (old JHS or new ATS) i
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom
log URL for stdout/stderr
URL: https://github.com/apache/spark/pull/23260#issuecomment-446103258
@vanzin Thanks for the detailed review! Addressed review comments.
---
23 matches
Mail list logo