[
https://issues.apache.org/jira/browse/SPARK-19875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923152#comment-15923152
]
Sameer Agarwal commented on SPARK-19875:
----------------------------------------
I also verified with [~jay.pranavamurthi] that this is a constraint inference
bug. The underlying issue here is that our constraint inference rules aren't
bounded and can grow indefinitely making subsequent inference painfully slow.
IMO allowing users to disable constraint inference via a conf flag
(SPARK-19846) is a reasonable workaround in the short term. Longer term, we
should just intelligently bound their size.
> Map->filter on many columns gets stuck in constraint inference optimization
> code
> --------------------------------------------------------------------------------
>
> Key: SPARK-19875
> URL: https://issues.apache.org/jira/browse/SPARK-19875
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Jay Pranavamurthi
> Attachments: test10cols.csv, test50cols.csv, TestFilter.scala
>
>
> The attached code (TestFilter.scala) works with a 10-column csv dataset, but
> gets stuck with a 50-column csv dataset. Both datasets are attached.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
