[
https://issues.apache.org/jira/browse/PARQUET-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363558#comment-14363558
]
Oscar Boykin commented on PARQUET-218:
--------------------------------------
I think these macros are great, but I'd like to see if we can go a bit further.
I'd like Projection[T, U] to extend (or be implicitly convertible to
Function1[T, U]) so that we can use projections in scalding/spark without being
sure the underlying data supports parquet project (we have sources that are
mixed: for some time ranges they are parquet, but older data might just be
compressed thrift).
If we have this functionality, we can project as the IO layer when possible,
when not possible we can do it at scalding/spark layer.
> Macro based Scala APIs for Predicate and Projection push down
> -------------------------------------------------------------
>
> Key: PARQUET-218
> URL: https://issues.apache.org/jira/browse/PARQUET-218
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Reporter: Julien Le Dem
>
> There has been some experiments with Scala macros to make Predicate and
> Projection APIs in Scala.
> In particular by [~neville]:
> https://github.com/nevillelyh/parquet-avro-extra/blob/master/parquet-avro-examples/src/main/scala/me/lyh/parquet/avro/examples/ExampleApp.scala
> https://github.com/nevillelyh/parquet-avro-extra/commit/090f8d1d1d7d44489bb71e04465d4a12a46432c9#diff-89dca6adcdf105c6063661de43931c29R15
> There have also been some discussion at Twitter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
