[
https://issues.apache.org/jira/browse/SPARK-55087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Ott updated SPARK-55087:
-----------------------------
Description:
There is a usability issue in custom Python Data sources. I'm implementing
pushFilters for my source, but when I'm trying to use it, it fails with the
error:
pyspark.errors.exceptions.base.PySparkAssertionError:
[DATA_SOURCE_PUSHDOWN_DISABLED] CassandraBatchReader implements pushFilters()
but filter pushdown is disabled because configuration
'spark.sql.python.filterPushdown.enabled' is false. Set it to true to enable
filter pushdown.
If this functionality is disabled by default, why not simply skip calling the
pushFilters?
was:
There is a usability issue in custom Python Data sources. I'm implementing
pushFilters for my source, but when I'm trying to use it, it fails with the
error:
pyspark.errors.exceptions.base.PySparkAssertionError:
[DATA_SOURCE_PUSHDOWN_DISABLED] CassandraBatchReader implements pushFilters()
but filter pushdown is disabled because configuration
'spark.sql.python.filterPushdown.enabled' is false. Set it to true to enable
filter pushdown.
If this functionality is disabled by default, why not simply skip calling the
pushFilters?
P.S. This change was introduced in
https://issues.apache.org/jira/browse/SPARK-53978
> Reading with Python Data source fails by default if it implements pushFilters
> ------------------------------------------------------------------------------
>
> Key: SPARK-55087
> URL: https://issues.apache.org/jira/browse/SPARK-55087
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.1
> Reporter: Alex Ott
> Priority: Major
>
> There is a usability issue in custom Python Data sources. I'm implementing
> pushFilters for my source, but when I'm trying to use it, it fails with the
> error:
> pyspark.errors.exceptions.base.PySparkAssertionError:
> [DATA_SOURCE_PUSHDOWN_DISABLED] CassandraBatchReader implements pushFilters()
> but filter pushdown is disabled because configuration
> 'spark.sql.python.filterPushdown.enabled' is false. Set it to true to enable
> filter pushdown.
> If this functionality is disabled by default, why not simply skip calling the
> pushFilters?
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
