Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/4434#issuecomment-103675457
@staslos, thanks for clarifying further. It is always useful to hear about
shortcomings in real usages. Spark SQL was alpha in 1.2, but as of 1.3 the
APIs are stable and we are using it at least at Databricks in production.
There was a regression in the first 1.3 RC for reading parquet from S3, but
that was fixed before the final release.
If there are limitations when doing schema evolution with datasources or
avro in general it would be great to open issues either in Spark or spark-avro.
I'd hope that writing projections would be a light weight way to do this in a
format agnostic way. Thought up until now most of our effort has been in
making sure that this work seamlessly for parquet, so its quite possible there
is still work to be done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
