Kirill Kozlov created BEAM-8343:
-----------------------------------
Summary: Add means for IO APIs to support predicate and/or project
push-down when running SQL pipelines
Key: BEAM-8343
URL: https://issues.apache.org/jira/browse/BEAM-8343
Project: Beam
Issue Type: New Feature
Components: dsl-sql
Reporter: Kirill Kozlov
Assignee: Kirill Kozlov
The objective is to create a universal way for Beam SQL IO APIs to support
predicate/project push-down.
A proposed way to achieve that is by introducing an interface responsible for
identifying what portion(s) of a Calc can be moved down to IO layer. Also,
adding following methods to a BeamSqlTable interface to pass necessary
parameters to IO APIs:
- BeamSqlTableFilter supportsFilter(RexProgram program, RexNode filter)
- Boolean supportsProjects()
- PCollection<Row> buildIOReader(PBegin begin, BeamSqlTableFilter filters,
List<String> fieldNames)
Design doc
[link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
