[GitHub] flink pull request #5759: [FLINK-9067] [e2eTests] Add StreamSQLTestProgram a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5759


---

[GitHub] flink pull request #5759: [FLINK-9067] [e2eTests] Add StreamSQLTestProgram a...

[zhangminglei]

Github user zhangminglei commented on a diff in the pull request:

https://github.com/apache/flink/pull/5759#discussion_r177433238
  
--- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh ---
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash

+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+source "$(dirname "$0")"/common.sh
+

+TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar
+
+# copy flink-table jar into lib folder
+cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib
+
+start_cluster
+$FLINK_DIR/bin/taskmanager.sh start
+$FLINK_DIR/bin/taskmanager.sh start
+$FLINK_DIR/bin/taskmanager.sh start
+
+$FLINK_DIR/bin/flink run -p 4 $TEST_PROGRAM_JAR -outputPath 
$TEST_DATA_DIR/out/result
+
+stop_cluster
+$FLINK_DIR/bin/taskmanager.sh stop-all
+
+# remove flink-table from lib folder
+rm $FLINK_DIR/lib/flink-table*jar
--- End diff --

Exactly. 


---

[GitHub] flink pull request #5759: [FLINK-9067] [e2eTests] Add StreamSQLTestProgram a...

[twalthr]

Github user twalthr commented on a diff in the pull request:

https://github.com/apache/flink/pull/5759#discussion_r177416467
  
--- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh ---
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash

+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+source "$(dirname "$0")"/common.sh
+

+TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar
+
+# copy flink-table jar into lib folder
+cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib
+
+start_cluster
+$FLINK_DIR/bin/taskmanager.sh start
+$FLINK_DIR/bin/taskmanager.sh start
+$FLINK_DIR/bin/taskmanager.sh start
+
+$FLINK_DIR/bin/flink run -p 4 $TEST_PROGRAM_JAR -outputPath 
$TEST_DATA_DIR/out/result
+
+stop_cluster
+$FLINK_DIR/bin/taskmanager.sh stop-all
+
+# remove flink-table from lib folder
+rm $FLINK_DIR/lib/flink-table*jar
--- End diff --

We should convert this into a function and call `trap` to make sure this is 
always executed for clean up.


---

[GitHub] flink pull request #5759: [FLINK-9067] [e2eTests] Add StreamSQLTestProgram a...

[twalthr]

Github user twalthr commented on a diff in the pull request:

https://github.com/apache/flink/pull/5759#discussion_r177416248
  
--- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh ---
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash

+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+source "$(dirname "$0")"/common.sh
+

+TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar
+
+# copy flink-table jar into lib folder
+cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib
+
+start_cluster
+$FLINK_DIR/bin/taskmanager.sh start
+$FLINK_DIR/bin/taskmanager.sh start
+$FLINK_DIR/bin/taskmanager.sh start
+
+$FLINK_DIR/bin/flink run -p 4 $TEST_PROGRAM_JAR -outputPath 
$TEST_DATA_DIR/out/result
+
+stop_cluster
+$FLINK_DIR/bin/taskmanager.sh stop-all
+
+# remove flink-table from lib folder
+rm $FLINK_DIR/lib/flink-table*jar
+
+# collect results from files
+cat /tmp/xxx/part-0-0 /tmp/xxx/_part-0-1.pending > 
$TEST_DATA_DIR/out/result-complete
--- End diff --

I guess this line is not correct ;)


---

[GitHub] flink pull request #5759: [FLINK-9067] [e2eTests] Add StreamSQLTestProgram a...

[twalthr]

Github user twalthr commented on a diff in the pull request:

https://github.com/apache/flink/pull/5759#discussion_r177416632
  
--- Diff: flink-end-to-end-tests/flink-stream-sql-test/pom.xml ---
@@ -0,0 +1,126 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+   
+   flink-end-to-end-tests
+   org.apache.flink
+   1.6-SNAPSHOT
+   ..
+   
+
+   4.0.0
+
+   flink-stream-sql-test_${scala.binary.version}
+   flink-stream-sql-test
+   jar
+
+   
+   
+   org.apache.flink
+   flink-core
+   ${project.version}
+   provided
+   
+   
+   org.apache.flink
+   
flink-streaming-scala_${scala.binary.version}
+   ${project.version}
+   provided
+   
+   
+   org.apache.flink
+   
flink-table_${scala.binary.version}
+   ${project.version}
+   provided
+   
+   
+   org.apache.flink
+   
flink-connector-filesystem_${scala.binary.version}
+   ${project.version}
+   
+   
+
+   
+   
+   
+   org.apache.maven.plugins
+   maven-shade-plugin
+   3.0.0
+   
+   
+   package
+   
+   shade
+   
+   
+   
+   
+   
com.google.code.findbugs:jsr305
+   
org.slf4j:*
+   
log4j:*
+   
+   
+   
+   
+   
*:*
+   

+   
META-INF/*.SF
+   
META-INF/*.DSA
+   
META-INF/*.RSA
+   

+   
+   
+   
+   
+   
org.apache.flink.sql.tests.StreamSQLTestProgram
+   
+   
+   
+   
+   
+   
+   
+   
--- End diff --

This can be simiplified similar to the dataset allround test.


---

[GitHub] flink pull request #5759: [FLINK-9067] [e2eTests] Add StreamSQLTestProgram a...

[zhangminglei]

Github user zhangminglei commented on a diff in the pull request:

https://github.com/apache/flink/pull/5759#discussion_r177099106
  
--- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh ---
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash

+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+source "$(dirname "$0")"/common.sh
+

+TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar
+
+# copy flink-table jar into lib folder
+cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib
--- End diff --

Thanks @twalthr . I will take a look.


---

[GitHub] flink pull request #5759: [FLINK-9067] [e2eTests] Add StreamSQLTestProgram a...

[twalthr]

Github user twalthr commented on a diff in the pull request:

https://github.com/apache/flink/pull/5759#discussion_r177058370
  
--- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh ---
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash

+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+source "$(dirname "$0")"/common.sh
+

+TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar
+
+# copy flink-table jar into lib folder
+cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib
--- End diff --

@zhangminglei there is a known issue with a classloader that can not be 
removed. See the notice in 
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/#setup. 
That's why the optional Table API jar is moved to lib.


---

[GitHub] flink pull request #5759: [FLINK-9067] [e2eTests] Add StreamSQLTestProgram a...

[zhangminglei]

Github user zhangminglei commented on a diff in the pull request:

https://github.com/apache/flink/pull/5759#discussion_r176906809
  
--- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh ---
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash

+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+source "$(dirname "$0")"/common.sh
+

+TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar
+
+# copy flink-table jar into lib folder
+cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib
--- End diff --

Oh. I think because you start taskmanager.sh, so you have to copy that. Not 
sure about it.


---

[GitHub] flink pull request #5759: [FLINK-9067] [e2eTests] Add StreamSQLTestProgram a...

[zhangminglei]

Github user zhangminglei commented on a diff in the pull request:

https://github.com/apache/flink/pull/5759#discussion_r176906704
  
--- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh ---
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash

+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+source "$(dirname "$0")"/common.sh
+

+TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar
+
+# copy flink-table jar into lib folder
+cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib
--- End diff --

Hi, @fhueske I have some small questions here. I can see that there are 
some jar files in the example folder(```$FLINK_DIR/example```), some jar files 
in the ```$FLINK_DIR/lib``` folder, and other jar files in the 
```$FLINK_DIR/opt``` folder. Why did you copy this jar file from opt  folder to 
lib folder, instead of putting to ```$FLINK_DIR/example``` ? Thanks.


---

[GitHub] flink pull request #5759: [FLINK-9067] [e2eTests] Add StreamSQLTestProgram a...

[fhueske]

GitHub user fhueske opened a pull request:

https://github.com/apache/flink/pull/5759

[FLINK-9067] [e2eTests] Add StreamSQLTestProgram and test run script.

## What is the purpose of the change

* Adds an end-to-end test for a streaming SQL query
* The query fails at the first execution attempt, recovers, and completes 
in the second attempt

## Brief change log

* Add a Streaming SQL Test Job
* Add a script to setup the Flink cluster, run the job, stop the cluster, 
and verify the result
* Register the script in the nightly test jobs

## Verifying this change

* This PR adds a test.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): **no**
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: **no**
  - The serializers: **no**
  - The runtime per-record code paths (performance sensitive): **no**
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
  - The S3 file system connector: **no**

## Documentation

  - Does this pull request introduce a new feature? **no**
  - If yes, how is the feature documented? **n/a**


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhueske/flink e2eTestSQLStream

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5759.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5759


commit 7a14340827f5a31a11c71e05075732d3c455cda5
Author: Fabian Hueske 
Date:   2018-03-23T20:42:53Z

[FLINK-9067] [e2eTests] Add StreamSQLTestProgram and test run script.




---