[GitHub] [flink] walterddr commented on a change in pull request #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate

2019-10-15 Thread GitBox 
walterddr commented on a change in pull request #8468: 
[FLINK-12399][table][table-planner] Fix FilterableTableSource does not change 
after applyPredicate
URL: https://github.com/apache/flink/pull/8468#discussion_r335180536
 
 

 ##
 File path: 
flink-connectors/flink-orc/src/main/java/org/apache/flink/orc/OrcTableSource.java
 ##
 @@ -213,7 +213,8 @@ public boolean isFilterPushedDown() {
 
@Override
public String explainSource() {
-   return "OrcFile[path=" + path + ", schema=" + orcSchema + ", 
filter=" + predicateString() + "]";
+   return "OrcFile[path=" + path + ", schema=" + orcSchema + ", 
filter=" + predicateString()
 
 Review comment:
   changed accordingly. actually the check for empty was missing


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] walterddr commented on a change in pull request #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate

2019-10-15 Thread GitBox 
walterddr commented on a change in pull request #8468: 
[FLINK-12399][table][table-planner] Fix FilterableTableSource does not change 
after applyPredicate
URL: https://github.com/apache/flink/pull/8468#discussion_r335181158
 
 

 ##
 File path: 
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/PushProjectIntoTableSourceScanRule.scala
 ##
 @@ -67,13 +67,23 @@ class PushProjectIntoTableSourceScanRule extends 
RelOptRule(
 val relOptTable = scan.getTable.asInstanceOf[FlinkRelOptTable]
 val tableSourceTable = relOptTable.unwrap(classOf[TableSourceTable[_]])
 val oldTableSource = tableSourceTable.tableSource
-val newTableSource = oldTableSource match {
+val (newTableSource, isProjectSuccess) = oldTableSource match {
   case nested: NestedFieldsProjectableTableSource[_] =>
 val nestedFields = RexNodeExtractor.extractRefNestedInputFields(
   project.getProjects, usedFields)
-nested.projectNestedFields(usedFields, nestedFields)
+(nested.projectNestedFields(usedFields, nestedFields), true)
   case projecting: ProjectableTableSource[_] =>
-projecting.projectFields(usedFields)
+(projecting.projectFields(usedFields), true)
+  case nonProjecting: TableSource[_] =>
+// projection cannot be pushed to TableSource
+(nonProjecting, false)
+}
+
+if (isProjectSuccess
+  && 
newTableSource.explainSource().equals(oldTableSource.explainSource())) {
 
 Review comment:
   I think the special case can be put outside of the scope of this PR since it 
is more related to how `*` is handled. I can open another JIRA for addressing 
this. what do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] walterddr commented on a change in pull request #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate

2019-10-12 Thread GitBox 
walterddr commented on a change in pull request #8468: 
[FLINK-12399][table][table-planner] Fix FilterableTableSource does not change 
after applyPredicate
URL: https://github.com/apache/flink/pull/8468#discussion_r334245784
 
 

 ##
 File path: 
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/api/stream/table/validation/TableSourceValidationTest.scala
 ##
 @@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.api.stream.table.validation
+
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.typeutils.RowTypeInfo
+import org.apache.flink.table.api.scala._
+import org.apache.flink.table.api.{TableException, TableSchema, Types}
+import org.apache.flink.table.utils.{TableTestBase, 
TestFilterableTableSourceWithoutExplainSourceOverride, 
TestProjectableTableSourceWithoutExplainSourceOverride}
+import org.hamcrest.Matchers
+import org.junit.Test
+
+class TableSourceValidationTest extends TableTestBase {
+
+  @Test
+  def testPushProjectTableSourceWithoutExplainSource(): Unit = {
+expectedException.expectCause(Matchers.isA(classOf[TableException]))
 
 Review comment:
   I actually tried using the annotation but it doesnt work since the wrapped 
around exception is not `TableException`. had to unparse the cause of the top 
level.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] walterddr commented on a change in pull request #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate

2019-09-30 Thread GitBox 
walterddr commented on a change in pull request #8468: 
[FLINK-12399][table][table-planner] Fix FilterableTableSource does not change 
after applyPredicate
URL: https://github.com/apache/flink/pull/8468#discussion_r329875995
 
 

 ##
 File path: 
flink-connectors/flink-orc/src/main/java/org/apache/flink/orc/OrcTableSource.java
 ##
 @@ -213,7 +213,8 @@ public boolean isFilterPushedDown() {
 
@Override
public String explainSource() {
-   return "OrcFile[path=" + path + ", schema=" + orcSchema + ", 
filter=" + predicateString() + "]";
+   return "OrcFile[path=" + path + ", schema=" + orcSchema + ", 
filter=" + predicateString()
 
 Review comment:
   had to change this override since in fact the OrcTableSource does not 
explain the pushed down predicate.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] walterddr commented on a change in pull request #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate

2019-09-30 Thread GitBox 
walterddr commented on a change in pull request #8468: 
[FLINK-12399][table][table-planner] Fix FilterableTableSource does not change 
after applyPredicate
URL: https://github.com/apache/flink/pull/8468#discussion_r329876013
 
 

 ##
 File path: 
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetTableSource.java
 ##
 @@ -223,7 +224,8 @@ public TableSchema getTableSchema() {
@Override
public String explainSource() {
return "ParquetFile[path=" + path + ", schema=" + parquetSchema 
+ ", filter=" + predicateString()
-   + ", typeInfo=" + typeInfo + "]";
+   + ", typeInfo=" + typeInfo + ", selectedFields=" + 
Arrays.toString(selectedFields)
 
 Review comment:
   same here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] walterddr commented on a change in pull request #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate

2019-09-30 Thread GitBox 
walterddr commented on a change in pull request #8468: 
[FLINK-12399][table][table-planner] Fix FilterableTableSource does not change 
after applyPredicate
URL: https://github.com/apache/flink/pull/8468#discussion_r329876088
 
 

 ##
 File path: 
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/plan/rules/logical/PushFilterIntoTableSourceScanRule.scala
 ##
 @@ -83,6 +84,14 @@ class PushFilterIntoTableSourceScanRule extends RelOptRule(
 
 val newTableSource = filterableSource.applyPredicate(remainingPredicates)
 
+if (remainingPredicates.size() > 0
+  && 
newTableSource.asInstanceOf[FilterableTableSource[_]].isFilterPushedDown
+  && 
newTableSource.explainSource().equals(scan.tableSource.explainSource())) {
+  throw new TableException("Failed to push filter into table source! "
 
 Review comment:
   throwing `TableException` here. do you think we should make this change a 
soft one to only generate a warning and still let the planner to go through?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] walterddr commented on a change in pull request #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate

2019-09-12 Thread GitBox 
walterddr commented on a change in pull request #8468: 
[FLINK-12399][table][table-planner] Fix FilterableTableSource does not change 
after applyPredicate
URL: https://github.com/apache/flink/pull/8468#discussion_r323980865
 
 

 ##
 File path: 
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/plan/nodes/PhysicalTableSourceScan.scala
 ##
 @@ -47,9 +47,25 @@ abstract class PhysicalTableSourceScan(
 val terms = super.explainTerms(pw)
 .item("fields", deriveRowType().getFieldNames.asScala.mkString(", "))
 
+val auxiliarySourceDesc = tableSource match {
+  case fts: FilterableTableSource[_] =>
+s"FilterPushDown=${fts.isFilterPushedDown.toString}"
+  case pts: ProjectableTableSource[_] =>
+// TODO: add isTableProjected, or getProjectedFieldIndices API to 
explain pushdown.
 
 Review comment:
   there's no way to determine currently for `ProjectableTableSource`. I will 
create a follow up ticket for this once the solution is accepted. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services