[jira] [Comment Edited] (BAHIR-8) Investigate weather we can use org.apache.bahir packages on Apache Spark extensions
[
https://issues.apache.org/jira/browse/BAHIR-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330676#comment-15330676
]
Christian Kadner edited comment on BAHIR-8 at 6/14/16 9:42 PM:
---
With my current approach, there is only one class per module which we need to
keep in the o.a.s namespace in order to maintain back-ward compatibility:
{noformat}
streaming-akka/src/main/scala/org/apache/spark/streaming/akka/AkkaUtils.scala
streaming-mqtt/src/main/scala/org/apache/spark/streaming/mqtt/MQTTUtils.scala
streaming-twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterUtils.scala
streaming-zeromq/src/main/scala/org/apache/spark/streaming/zeromq/ZeroMQUtils.scala
{noformat}
But your suggestion requires less new code and avoids unnecessary complexity,
so I am in favor of that.
was (Author: ckadner):
With my current approach, there is only one class per module which we need to
keep in the o.a.s namespace in order to maintain back-ward compatibility:
{noformat}
streaming-akka/src/main/scala/org/apache/spark/streaming/akka/AkkaUtils.scala
streaming-mqtt/src/main/scala/org/apache/spark/streaming/mqtt/MQTTUtils.scala
streaming-twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterUtils.scala
streaming-zeromq/src/main/scala/org/apache/spark/streaming/zeromq/ZeroMQUtils.scala
{noformat}
But there your suggestion requires less new code and avoids unnecessary
complexity, so I am in favor of that.
> Investigate weather we can use org.apache.bahir packages on Apache Spark
> extensions
> ---
>
> Key: BAHIR-8
> URL: https://issues.apache.org/jira/browse/BAHIR-8
> Project: Bahir
> Issue Type: Task
> Components: Spark Streaming Connectors
>Affects Versions: 2.0.0
>Reporter: Luciano Resende
>Assignee: Christian Kadner
>
> Experiment if we can move the current extensions to org.apache.bahir package.
> We might see issues related to the private[spark], etc restriction that scala
> imposes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BAHIR-8) Investigate weather we can use org.apache.bahir packages on Apache Spark extensions
[
https://issues.apache.org/jira/browse/BAHIR-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330676#comment-15330676
]
Christian Kadner commented on BAHIR-8:
--
With my current approach, there is only one class per module which we need to
keep in the o.a.s namespace in order to maintain back-ward compatibility:
{noformat}
streaming-akka/src/main/scala/org/apache/spark/streaming/akka/AkkaUtils.scala
streaming-mqtt/src/main/scala/org/apache/spark/streaming/mqtt/MQTTUtils.scala
streaming-twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterUtils.scala
streaming-zeromq/src/main/scala/org/apache/spark/streaming/zeromq/ZeroMQUtils.scala
{noformat}
But there your suggestion requires less new code and avoids unnecessary
complexity, so I am in favor of that.
> Investigate weather we can use org.apache.bahir packages on Apache Spark
> extensions
> ---
>
> Key: BAHIR-8
> URL: https://issues.apache.org/jira/browse/BAHIR-8
> Project: Bahir
> Issue Type: Task
> Components: Spark Streaming Connectors
>Affects Versions: 2.0.0
>Reporter: Luciano Resende
>Assignee: Christian Kadner
>
> Experiment if we can move the current extensions to org.apache.bahir package.
> We might see issues related to the private[spark], etc restriction that scala
> imposes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BAHIR-8) Investigate weather we can use org.apache.bahir packages on Apache Spark extensions
[ https://issues.apache.org/jira/browse/BAHIR-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330584#comment-15330584 ] Luciano Resende commented on BAHIR-8: - What I was thinking was to have a very thin layer on top of the current extension to enable new users to start using these connectors via org.apache.bahir code, which would then delegate back to the existing org.apache.spark code. This way users can start transitioning to bahir code, and we still wouldn't break existing applications that could continue to directly access the org.apache.spark. >From your comment above, are we going to loose this backward compatibility ? What others thing about the compatibility aspects ? > Investigate weather we can use org.apache.bahir packages on Apache Spark > extensions > --- > > Key: BAHIR-8 > URL: https://issues.apache.org/jira/browse/BAHIR-8 > Project: Bahir > Issue Type: Task > Components: Spark Streaming Connectors >Affects Versions: 2.0.0 >Reporter: Luciano Resende >Assignee: Christian Kadner > > Experiment if we can move the current extensions to org.apache.bahir package. > We might see issues related to the private[spark], etc restriction that scala > imposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BAHIR-8) Investigate weather we can use org.apache.bahir packages on Apache Spark extensions
[
https://issues.apache.org/jira/browse/BAHIR-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Kadner updated BAHIR-8:
-
Assignee: Christian Kadner
Affects Version/s: 2.0.0
Issue Type: Task (was: Bug)
The current set of streaming modules in Bahir (_akka, mqtt, zeromq, twitter_)
are using 7 different methods/constructors/fields which are
private/packaged-scoped to {{org.apache.spark.*}}.
{noformat}
core/src/main/scala/org/apache/spark/SparkContext.scala
- private[spark] def clean[F <: AnyRef](f: F, checkSerializable: Boolean =
true): F = { ...
core/src/main/scala/org/apache/spark/util/Utils.scala
- private[spark] object Utils extends Logging { ...
core/src/test/scala/org/apache/spark/SparkFunSuite.scala
- private[spark] abstract class SparkFunSuite { ...
core/src/main/scala/org/apache/spark/Logging.scala
- private[spark] trait Logging { ...
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
- private[streaming] val sc: SparkContext = { ...
- private[streaming] def withNamedScope[U](name: String)(body: => U): U = { ...
streaming/src/main/scala/org/apache/spark/streaming/dstream/InputDStream.scala
- private[streaming] def name: String = { ...
{noformat}
I suggest to create a _shim_ module named *common* or *common-streaming* to the
Bahir repository that will contains common utils and the necessary shim
classes/objects to make these "private APIs" accessible to the rest of the
classes/objects under the {{org.apache.bahir}} namespace. I am working on a PR
for that.
> Investigate weather we can use org.apache.bahir packages on Apache Spark
> extensions
> ---
>
> Key: BAHIR-8
> URL: https://issues.apache.org/jira/browse/BAHIR-8
> Project: Bahir
> Issue Type: Task
> Components: Spark Streaming Connectors
>Affects Versions: 2.0.0
>Reporter: Luciano Resende
>Assignee: Christian Kadner
>
> Experiment if we can move the current extensions to org.apache.bahir package.
> We might see issues related to the private[spark], etc restriction that scala
> imposes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
