GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/10971
[SPARK-13055] SQLHistoryListener throws ClassCastException
This is an existing issue uncovered recently by #10835. The reason for the
exception was because the `SQLHistoryListener` gets all sorts of accumulators,
not just the ones that represent SQL metrics. For example, the listener gets
the `internal.metrics.shuffleRead.remoteBlocksFetched`, which is an Int, then
it proceeds to cast the Int to a Long, which fails.
The fix is to mark accumulators representing SQL metrics using some
internal metadata. Then we can identify which ones are SQL metrics and only
process those in the `SQLHistoryListener`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark fix-sql-history
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10971.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10971
----
commit 4a56ed5ea53db3ca444417046f0581932e7ca189
Author: Andrew Or <[email protected]>
Date: 2016-01-28T21:24:14Z
Add metadata field to AccumulableInfo
This is currently not used, but will be populated by SQLMetrics
in the ensuing commits.
commit 4bf0b3a36353b531a4a82e4f496a41d1b8611c71
Author: Andrew Or <[email protected]>
Date: 2016-01-28T21:30:44Z
Add Accumulable#toInfo method
SQLMetrics will override this in the next commit to set some
custom metadata in the AccumulableInfo it returns.
commit 0d5eb4b8785cbfbed33358153fd54e231ac89bba
Author: Andrew Or <[email protected]>
Date: 2016-01-28T21:46:24Z
Load only SQL metrics in SQLHistoryListener
This should fix the ClassCastException since all SQL metrics
are (for now) Long.
commit 47a91ff11105586b357c06adbb906f557718e72d
Author: Andrew Or <[email protected]>
Date: 2016-01-28T23:38:38Z
Add tests
commit 14d34fe6bb40b47cb0d7d96699cfe15fbec45dee
Author: Andrew Or <[email protected]>
Date: 2016-01-28T23:39:01Z
Merge branch 'master' of github.com:apache/spark into fix-sql-history
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
