[
https://issues.apache.org/jira/browse/SPARK-26254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720241#comment-16720241
]
Gabor Somogyi edited comment on SPARK-26254 at 12/13/18 3:29 PM:
-----------------------------------------------------------------
{quote}There was concern about using ServiceLoader before, but if the interface
being loaded is private to Spark, it's fine with me.
{quote}
org.apache.spark.deploy.security.HadoopDelegationTokenProvider fulfils it.
{quote}Keep HDFS and HBase in core
{quote}
clear and same idea.
{quote}move the Kafka one to some Kafka package
{quote}
you mean module isn't it?
* If we move inside core then the kafka deps remain
* If we move to kafka-sql then DStreams will not reach it
My suggestion is to create a module something like kafka-token-provider and
kafka-sql (+ later DStreams) can depend on that.
{quote}the Hive one to the Hive module
{quote}
clear and same idea (sql/hive).
was (Author: gsomogyi):
{quote}There was concern about using ServiceLoader before, but if the interface
being loaded is private to Spark, it's fine with me.
{quote}
org.apache.spark.deploy.security.HadoopDelegationTokenProvider fulfils it.
{quote}Keep HDFS and HBase in core
{quote}
clear and same idea.
{quote}move the Kafka one to some Kafka package
{quote}
you mean module isn't it?
* If we move inside core then the kafka deps remain
* If we move to kafka-sql then DStreams will not reach it
My suggestion is to create a module something like kafka-token-provider and
kafka-sql (+ later DStreams) can depend on that.
{quote}the Hive one to the Hive module
{quote}
clear and same idea. For example hive-token-provider which extracts the ugly
dependencies from core.
> Move delegation token providers into a separate project
> -------------------------------------------------------
>
> Key: SPARK-26254
> URL: https://issues.apache.org/jira/browse/SPARK-26254
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Gabor Somogyi
> Priority: Major
>
> There was a discussion in
> [PR#22598|https://github.com/apache/spark/pull/22598] that there are several
> provided dependencies inside core project which shouldn't be there (for ex.
> hive and kafka). This jira is to solve this problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
