Hi all! Spark 1.6.1.
Anyone know how to implement custom DF filter to be later pushed down to custom datasource? To be short, I've managed to create custom Expression, implicitly add methods with it to Column class, but I am stuck at the point where Expression must be converted to Filter by Spark before it is passed to "unhandledFilters" method at datasource Relation. Take a look at DataSourceStrategy.translateFilter - this is the place where mapping between Expression and Filter is hardcoded into Spark. Anyone know a good way to put just another custom filter in there? Or anyone implemented a custom filtering before + pushdown to datasource? Thanks in advance!
