> > * As a heavy user of complex data types I was wondering if there was > any plan to push those changes upstream? >
Yes, we intend to contribute this to open source. > * In addition, I was wondering if as part of this change it also tries > to solve the column pruning / filter pushdown issues with complex > datatypes? For parquet, this effort is primarily tracked via SPARK-4502 (see https://github.com/apache/spark/pull/16578) and is currently targeted for 2.3.
