[
https://issues.apache.org/jira/browse/SPARK-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14647127#comment-14647127
]
Michal Malohlava commented on SPARK-3270:
-----------------------------------------
Hi Steve,
yes, agree, it would be really beneficial to handle an executor's LC in more
fine-grained way - at least to have well-defined points in LC which can be
enhanced by defined functionality (our typical use-case is to launch additional
services on all available executors - see closed PR 2691).
In our case, we solved the issue by having RDD with pre-defined locality, which
is used to invoke a given code on all available executor - see code here:
https://github.com/h2oai/sparkling-water/blob/master/core/src/main/scala/org/apache/spark/h2o/InvokeOnNodesRDD.scala
And in this context, we are using also Spark Streaming (see
https://github.com/h2oai/sparkling-water/blob/master/examples/src/main/scala/org/apache/spark/examples/h2o/CraigslistJobTitlesStreamingApp.scala),
however we do not handle executor failures right now. But I can imagine
solution based on `SparkListener` (i.e., reacting to a change in a cluster by
firing a new operation going around cluster and executing given code if it is
necessary).
> Spark API for Application Extensions
> ------------------------------------
>
> Key: SPARK-3270
> URL: https://issues.apache.org/jira/browse/SPARK-3270
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Reporter: Michal Malohlava
>
> Any application should be able to enrich spark infrastructure by services
> which are not available by default.
> Hence, to support such application extensions (aka "extesions"/"plugins")
> Spark platform should provide:
> - an API to register an extension
> - an API to register a "service" (meaning provided functionality)
> - well-defined points in Spark infrastructure which can be enriched/hooked
> by extension
> - a way of deploying extension (for example, simply putting the extension
> on classpath and using Java service interface)
> - a way to access extension from application
> Overall proposal is available here:
> https://docs.google.com/document/d/1dHF9zi7GzFbYnbV2PwaOQ2eLPoTeiN9IogUe4PAOtrQ/edit?usp=sharing
> Note: In this context, I do not mean reinventing OSGi (or another plugin
> platform) but it can serve as a good starting point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
