[
https://issues.apache.org/jira/browse/SPARK-31943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jefferson V updated SPARK-31943:
--------------------------------
Description:
[31500|[https://github.com/apache/spark/pull/28351/files]] introduced
unimplemented fields of the `Collect` class that cause client extensions of
that class to fail unless the spark version is pinned to `2.4.5`. Since this
was a minor version bump, seems like this wasn't desired.
I believe we should be able to at least put default values:
`convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)`
`bufferElementType: DataType = child.dataType`
and restore the `override def eval` in `Collect`, to support compatibility with
2.4.5 while allowing implementers in that file to override them to fix the bug.
Since the abstract `Collect` is currently not designed to fix the bug (just
provide tools that can be implemented to fix it), this change wouldn't
undermine the bug fix, just add backwards compatibility to 2.4.5.
was:
[31500|[https://github.com/apache/spark/pull/28351/files]] introduced
unimplemented fields of the `Collect` class that cause extensions of that class
to fail unless the spark version is pinned to `2.4.5`. Since this was a minor
version bump, seems like this wasn't desired.
I believe we should be able to at least put default values:
`convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)`
`bufferElementType: DataType = child.dataType`
and restore the `override def eval` in `Collect`, to support compatibility with
2.4.5 while allowing implementers in that file to override them to fix the bug.
Since the abstract `Collect` is currently not designed to fix the bug (just
provide tools that can be implemented to fix it), this change wouldn't
undermine the bug fix, just add backwards compatibility to 2.4.5.
> SPARK-31500 introduces breaking changes in 2.4.6
> ------------------------------------------------
>
> Key: SPARK-31943
> URL: https://issues.apache.org/jira/browse/SPARK-31943
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.6
> Reporter: Jefferson V
> Priority: Major
> Labels: backward-incompatible
>
> [31500|[https://github.com/apache/spark/pull/28351/files]] introduced
> unimplemented fields of the `Collect` class that cause client extensions of
> that class to fail unless the spark version is pinned to `2.4.5`. Since this
> was a minor version bump, seems like this wasn't desired.
> I believe we should be able to at least put default values:
> `convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)`
> `bufferElementType: DataType = child.dataType`
> and restore the `override def eval` in `Collect`, to support compatibility
> with 2.4.5 while allowing implementers in that file to override them to fix
> the bug. Since the abstract `Collect` is currently not designed to fix the
> bug (just provide tools that can be implemented to fix it), this change
> wouldn't undermine the bug fix, just add backwards compatibility to 2.4.5.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
