[
https://issues.apache.org/jira/browse/DATAFU-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eyal Allweil updated DATAFU-168:
--------------------------------
Fix Version/s: 1.8.0
(was: 1.7.0)
> Add support for Spark 2.4.6 and up
> ----------------------------------
>
> Key: DATAFU-168
> URL: https://issues.apache.org/jira/browse/DATAFU-168
> Project: DataFu
> Issue Type: Improvement
> Affects Versions: 1.6.1
> Reporter: Eyal Allweil
> Priority: Major
> Fix For: 1.8.0
>
>
> Once DATAFU-167 is merged, datafu-spark will support Spark versions up to
> 2.4.5. However, because our implementation of _collectLimitedList_ extends
> Spark's {_}collect{_}, and because its interface was changed in 2.4.6,
> compilation is broken for us.
>
> (here is the relevant line from collectLimitedList:
> [https://github.com/apache/datafu/blob/master/datafu-spark/src/main/scala/spark/utils/overwrites/SparkOverwriteUDAFs.scala#L104)]
>
> We need to either *1)* update our implementation, and drop support for older
> versions (and then release this in our version 1.8.0) or *2)* copy the code
> in a backwards compatible way.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
