[
https://issues.apache.org/jira/browse/DATAFU-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17621051#comment-17621051
]
Eyal Allweil commented on DATAFU-167:
-------------------------------------
Ok [~arpitbhardwaj] - I've merged some build changes/improvements. As far as I
can tell, the auth token API wasn't available in the py4j versions used in
Spark 2.2.0, 2.2.1, and 2.3.0.
I'm OK with dropping support for these versions in our next release, but I will
ask about it in our dev mailing list (dev@datafu.apache.org).
This means that with your fix you can change
[build_and_test_spark.sh|https://github.com/apache/datafu/blob/master/datafu-spark/build_and_test_spark.sh]
to remove these versions, and add 2.2.3, 2.3.3, 2.3.4, 2.4.4 and 2.4.5 which
will now become supported.
2.4.6 and up have a breaking change (for us) because of our
_collectLimitedList_ implementation.
You can run the script to test your branch - that's actually what the GitHub
PRs trigger.
> Fix Scala Python Bridge support in Spark 2 minor version updates
> -----------------------------------------------------------------
>
> Key: DATAFU-167
> URL: https://issues.apache.org/jira/browse/DATAFU-167
> Project: DataFu
> Issue Type: Bug
> Affects Versions: 1.6.1
> Environment: _emphasized text_
> Reporter: Eyal Allweil
> Priority: Major
> Labels: up-for-grabs
> Fix For: 1.7.0
>
>
> The Scala Python Bridge, which works for versions 2.2.2, 2.3.2 and 2.4.3,
> doesn't work for versions 2.2.3, 2.3.3, and 2.4.4 (and up).
> This can be reproduced in testing by running the tests with the command
> {code:bash}
> ./gradlew :datafu-spark:test -PscalaVersion=2.11 -PsparkVersion=2.2.3
> -PscalaCompatVersion=2.11
> {code}
> The error message is
> {noformat}
> AttributeError: 'GatewayParameters' object has no attribute 'auth_token'
> at
> org.apache.spark.datafu.deploy.SparkPythonRunner.execFile(SparkPythonRunner.scala:137)
> {noformat}
>
> Currently our code runs using the PYSPARK_ALLOW_INSECURE_GATEWAY parameter;
> it's possible/probable that using the auth_token parameter will both fix this
> problem and be better in general.
>
> Please note that in order to test Spark 2.4.4, you need to upgrade the
> scalatest version used.
>
> A description of using an auth token with py4j can be found here:
> [https://www.py4j.org/advanced_topics.html#authentication]
>
> It's likely that (at least some of the) changes will need to be made here:
> [https://github.com/apache/datafu/blob/master/datafu-spark/src/main/scala/spark/utils/overwrites/SparkPythonRunner.scala#L63]
> [https://github.com/apache/datafu/blob/master/datafu-spark/src/main/resources/pyspark_utils/bridge_utils.py#L43]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
