Re: Casting nested columns and updated nested struct fields.

2018-11-23 Thread Colin Williams
Seems like it's worthy of filing a bug against withColumn On Wed, Nov 21, 2018, 6:25 PM Colin Williams < colin.williams.seat...@gmail.com wrote: > Hello, > > I'm currently trying to update the schema for a dataframe with nested > columns. I would either like to update the schema itself or cast

[Spark ORC | SQL | Hive] Buffer size too small when using filterPushdown predicate=True (ref.: SPARK-25145)

2018-11-23 Thread Bjørnar Jensen
Hi, I submitted a bug report on this issue earlier (SPARK-25145), but it was closed as it could not be reproduced. Furthermore, it was stated that this error is usually caused my a misconfiguration. Well, we're still having this issue and would very much like to solve our misconfiguration.

Re: Casting nested columns and updated nested struct fields.

2018-11-23 Thread Colin Williams
Looks like it's been reported already. It's too bad it's been a year but should be released into spark 3: https://issues.apache.org/jira/browse/SPARK-22231 On Fri, Nov 23, 2018 at 8:42 AM Colin Williams wrote: > > Seems like it's worthy of filing a bug against withColumn > > On Wed, Nov 21, 2018,