[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2485 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
Github user apivovarov commented on a diff in the pull request: https://github.com/apache/flink/pull/2485#discussion_r78270898 --- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/scala/stream/ExplainStreamTest.scala --- @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.scala.stream + +import org.apache.flink.api.scala._ +import org.apache.flink.api.scala.table._ +import org.apache.flink.api.table.TableEnvironment +import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment +import org.apache.flink.streaming.util.StreamingMultipleProgramsTestBase +import org.junit.Assert.assertEquals +import org.junit._ + +class ExplainStreamTest + extends StreamingMultipleProgramsTestBase { + + val testFilePath = ExplainStreamTest.this.getClass.getResource("/").getFile + + @Test + def testFilter() : Unit = { +val env = StreamExecutionEnvironment.getExecutionEnvironment +val tEnv = TableEnvironment.getTableEnvironment(env) + +val table = env.fromElements((1, "hello")) + .toTable(tEnv, 'a, 'b) + .filter("a % 2 = 0") + +val result = tEnv.explain(table).replaceAll("\\r\\n", "\n") +val source = scala.io.Source.fromFile(testFilePath + + "../../src/test/scala/resources/testFilterStream0.out").mkString.replaceAll("\\r\\n", "\n") +assertEquals(result, source) + } + + @Test + def testUnion() : Unit = { --- End diff -- no need for space after () on line 35 and 50 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
Github user apivovarov commented on a diff in the pull request: https://github.com/apache/flink/pull/2485#discussion_r78270846 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/StreamTableEnvironment.scala --- @@ -271,5 +271,21 @@ abstract class StreamTableEnvironment( } } + /* + * Returns the AST of the specified Table API and SQL queries and the execution plan to compute +* the result of the given [[Table]]. + * + * @param table The table for which the AST and execution plan will be returned. +* @param extended Flag to include detailed optimizer estimates. + */ + def explain(table: Table): String = { + +val ast = RelOptUtil.toString(table.getRelNode) + +s"== Abstract Syntax Tree ==" + + System.lineSeparator + + s"$ast" --- End diff -- No need for s on line 285 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
Github user apivovarov commented on a diff in the pull request: https://github.com/apache/flink/pull/2485#discussion_r78270826 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/StreamTableEnvironment.scala --- @@ -271,5 +271,21 @@ abstract class StreamTableEnvironment( } } + /* + * Returns the AST of the specified Table API and SQL queries and the execution plan to compute +* the result of the given [[Table]]. + * + * @param table The table for which the AST and execution plan will be returned. +* @param extended Flag to include detailed optimizer estimates. + */ + def explain(table: Table): String = { + +val ast = RelOptUtil.toString(table.getRelNode) + +s"== Abstract Syntax Tree ==" + + System.lineSeparator + + s"$ast" --- End diff -- Maybe this? ``` s"== Abstract Syntax Tree ==${System.lineSeparator}$ast" ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
Github user apivovarov commented on a diff in the pull request: https://github.com/apache/flink/pull/2485#discussion_r78270796 --- Diff: docs/dev/table_api.md --- @@ -2457,3 +2457,27 @@ The Table API provides a configuration (the so-called `TableConfig`) to modify r By default, the Table API supports `null` values. Null handling can be disabled to improve preformance by setting the `nullCheck` property in the `TableConfig` to `false`. {% top %} + +Explaining a Table + +The Table API provides a mechanism to describe the graph of operations that leads to the resulting output. This is done through the `TableEnvironment#explain(table)` method. It returns a string describing two graphs: the Abstract Syntax Tree of the relational algebra query and the Flink's Execution Plan of the equivalent Flink's Job. + +Table `explain` is supported for both `BatchTableEnvironment` and `StreamTableEnvironment`. Currently `StreamTableEnvironment` doesn't support the explanation of the Execution Plan. + + + +{% highlight scala %} + val env = StreamExecutionEnvironment.getExecutionEnvironment + val tEnv = TableEnvironment.getTableEnvironment(env) + + val table1 = env.fromElements((1, "hello")).toTable(tEnv, 'count, 'word) + val table2 = env.fromElements((1, "hello")).toTable(tEnv, 'count, 'word) + val table = table1.unionAll(table2) + + val explanation:String = tEnv.explain(table) --- End diff -- put space before String --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
GitHub user chobeat opened a pull request: https://github.com/apache/flink/pull/2485 [Flink 4599] - Add 'explain()' also to StreamTableEnvironment You can merge this pull request into a Git repository by running: $ git pull https://github.com/radicalbit/flink FLINK-4599 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2485.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2485 commit 29f0204308a8f871e7b27767efc3020e362218f4 Author: chobeat Date: 2016-09-09T09:27:41Z AST DataStream table explain commit 593569405bbd2b728f5e9edc82242019a7fbe9be Author: chobeat Date: 2016-09-09T09:31:12Z Merge remote-tracking branch 'origin/master' into FLINK-4599 commit 7bf135237cb24ed889b92bf5d8026de43d1038fc Author: f7753 Date: 2016-08-22T13:07:55Z [FLINK-4436] Unclosed DataOutputBuffer in Utils#setTokensFor() This closes #2402 commit ba043aaa51401f53c2868927a540ebf7a3493318 Author: Greg Hogan Date: 2016-07-25T13:09:27Z [FLINK-4257] [gelly] Handle delegating algorithm change of class Replaces Delegate with NoOpOperator. This closes #2474 commit 850fd5fec5133f7729bc6a5b2af00cb2decc229b Author: Till Rohrmann Date: 2016-08-31T15:58:09Z [FLINK-4458] Replace ForkableFlinkMiniCluster by LocalFlinkMiniCluster Rename _configuration to originalConfiguration Remove testing classes from main scope in flink-runtime Previously, the ForkableFlinkMiniCluster which resided in flink-test-utils required these files to be in the main scope of flink-runtime. With the removal of the ForkableFlinkMiniCluster, these classes are now no longer needed and can be moved back to the test scope. This closes #2450. commit 7eecc4bd7aa605051cc3c6abc1a87233b8695127 Author: Till Rohrmann Date: 2016-09-01T12:41:44Z [FLINK-4456] Replace Akka specific types by interfaces in Task Introduce TaskExecutionStateListener for Task Replace JobManagerGateway in Task by InputSplitProvider and CheckpointNotifier Replace the TaskManager ActorGateway by TaskManagerConnection in Task Rename taskmanager.CheckpointNotifier into CheckpointResponder; rename TaskExecutionStateListener.notifyTaskExecutionState into notifyTaskExecutionStateChanged Remove InputSplitProvider.start; add ClassLoader parameter to InputSplitProvider.getNextInputSplit Removes the unused class InputSplitIterator. Update InputSplitProvider JavaDocs This closes #2456. commit 45f842eb8ea7da4f0535cd29c345fd45fe3d3815 Author: Greg Hogan Date: 2016-09-02T14:42:30Z [FLINK-4522] [docs] Gelly link broken in homepage The Gelly documentation was recently split into multiple pages in FLINK-4104 but was missing a redirect. This commit updates the Gelly redirect to point to the old page. This closes #2464 commit 7c9d1679c8319d560c9032691ad05b723b852f66 Author: Greg Hogan Date: 2016-09-02T15:53:08Z [FLINK-4571] [gelly] Configurable little parallelism in Gelly drivers This closes #2475 commit 2ddb4fef89debd7cef740058cc79ddca097879e1 Author: Alexander Pivovarov Date: 2016-09-07T21:11:06Z [FLINK-4595] Close FileOutputStream in ParameterTool This closes #2478 commit 2d4c75e688cd143022fc1db2b209c71935003c7d Author: chobeat Date: 2016-09-09T09:32:13Z Merge branch 'FLINK-4599' of github.com:radicalbit/flink into FLINK-4599 commit f3ba22ba5d2901c25cdbd11689a1ca3cc50935cd Author: chobeat Date: 2016-09-09T12:30:50Z moved stream explain test resources to main folder --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---