Hi,
There have been reports of errors coming out when the following is set.
spark.conf.set("spark.sql.adaptive.enabled", "true")
Some reported in this forum. Please search the email list for
=> spark.sql.adaptive.enabled as you have not specified the nature of your
query causing error with this configuration parameter is turned on. You may
get additional info.
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Tue, 3 Aug 2021 at 19:37, Jesse Lord <jl...@vectra.ai.invalid> wrote:
> Hello spark users,
>
>
>
> I have an error that I would like to report as a spark 3.1.1 bug but I do
> not know how to create a reproducible example. I can provide a full stack
> trace if desired but the most useful information seems to be
>
>
>
> E py4j.protocol.Py4JJavaError: An error occurred while
> calling o3301.toJavaRDD.
>
> E : java.lang.IllegalStateException:
> UnspecifiedDistribution does not have default partitioning.
>
> E at
> org.apache.spark.sql.catalyst.plans.physical.UnspecifiedDistribution$.createPartitioning(partitioning.scala:52)
>
> E at
> org.apache.spark.sql.execution.exchange.EnsureRequirements$.$anonfun$ensureDistributionAndOrdering$1(EnsureRequirements.scala:54)
>
>
>
> This error happens when I have spark.sql.adaptive.enabled=true but does
> not happen when I change to false. It happens for both one of my unit tests
> (~30 rows) and with production data. Another work-around is to cache the
> dataframe before calling the collect/toJSON statement.
>
>
>
> I was not able to find any information about this kind of error on the
> jira or from stackexchange. I was wondering if anyone has seen this error
> before related to AQE and has any suggestions for trying to report it.
>
>
>
> Thanks,
>
> Jesse
>
>
>
