Hi Gary,

Beam 2.3.0 and the Spark runner use Scala 2.11.

I can help you to have a smooth transition by creating a local branch using
Scala 2.10. However,  I strongly advice to upgrade to 2.11 as some other part of
Beam  (other runners and IOs) use 2.11 already.


On 02/22/2018 05:55 PM, Gary Dusbabek wrote:
> Hi,
> My apologies if this belongs on the dev list. If it does, let me know and I'll
> shoot things over that way...
> For the last day or so, I've been trying to create a Spark Runner that will 
> work
> on older deployments using Scala 2.10. I've taken a few approaches:
> 1. selectively changing a few dependencies in beam-runners-spark.pom (and a 
> few
> other places in the parent)
> 2. updating every dependency that references *_2.11 to be *_2.10
> In the former case the sticking point in both cases is that there is a library
> incompatibility with jackson-module-scala_2.xx. In the latter case there is a
> problem with SourceRDD.SourcePartitioning not [correctly] implementing
> `equals(...)` from the parent trait.
> Posts on the mailing list made me think that the move to Scala 2.11 started 
> only
> last fall, so I figured it should be easy to make the switch back.
> However, I have a feeling that it could be the case that I just don't 
> understand
> the Beam build system well enough to produce the right outcome (a custom 
> version
> that can be used with older Scala).
> Is there a correct or better way of achieving this?
> Kind Regards,
> Gary Dusbabek

Jean-Baptiste Onofré
Talend - http://www.talend.com

Reply via email to