[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6089


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[medcv]

Github user medcv commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r193581372
  
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -0,0 +1,98 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+   
+   org.apache.flink
+   flink-end-to-end-tests
+   1.6-SNAPSHOT
+   
+   4.0.0
+
+   flink-quickstart-test
+   flink-quickstart-test
+   jar
+
+   
+   2.11
+   
+
+   
+   
+   org.apache.flink
+   
flink-streaming-java_${scala.binary.version}
+   ${project.version}
+   provided
+   
+   
+   org.apache.flink
+   
flink-streaming-scala_${scala.binary.version}
+   ${project.version}
--- End diff --

Done!


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[medcv]

Github user medcv commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r193581841
  
--- Diff: flink-end-to-end-tests/test-scripts/test_quickstarts.sh ---
@@ -51,12 +59,12 @@ sed -i -e ''"$(($position + 1))"'i\
 ${flink.version}\
--- End diff --

I did some code change to get the ES dependency from 
flink-quickstart-test/pom.xml and removed the hardcoded values


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[medcv]

Github user medcv commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r193581567
  
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -0,0 +1,98 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+   
+   org.apache.flink
+   flink-end-to-end-tests
+   1.6-SNAPSHOT
+   
+   4.0.0
+
+   flink-quickstart-test
+   flink-quickstart-test
+   jar
+
+   
+   2.11
--- End diff --

Done!


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[medcv]

Github user medcv commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r193581591
  
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -0,0 +1,98 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+   
+   org.apache.flink
+   flink-end-to-end-tests
+   1.6-SNAPSHOT
+   
+   4.0.0
+
+   flink-quickstart-test
+   flink-quickstart-test
+   jar
+
+   
+   2.11
+   
+
+   
+   
+   org.apache.flink
+   
flink-streaming-java_${scala.binary.version}
+   ${project.version}
+   provided
+   
+   
+   org.apache.flink
+   
flink-streaming-scala_${scala.binary.version}
+   ${project.version}
+   
+   
+   org.apache.flink
+   
flink-connector-elasticsearch5_${scala.binary.version}
+   ${project.version}
+   
+   
+
+   
--- End diff --

Removed!


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r193392673
  
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -0,0 +1,98 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+   
+   org.apache.flink
+   flink-end-to-end-tests
+   1.6-SNAPSHOT
+   
+   4.0.0
+
+   flink-quickstart-test
+   flink-quickstart-test
+   jar
+
+   
+   2.11
--- End diff --

can be omitted as it is already defined in the root `pom.xml`


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r193392825
  
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -0,0 +1,98 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+   
+   org.apache.flink
+   flink-end-to-end-tests
+   1.6-SNAPSHOT
+   
+   4.0.0
+
+   flink-quickstart-test
+   flink-quickstart-test
+   jar
+
+   
+   2.11
+   
+
+   
+   
+   org.apache.flink
+   
flink-streaming-java_${scala.binary.version}
+   ${project.version}
+   provided
+   
+   
+   org.apache.flink
+   
flink-streaming-scala_${scala.binary.version}
+   ${project.version}
--- End diff --

set `scope` to `provided`


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r193395032
  
--- Diff: flink-end-to-end-tests/test-scripts/test_quickstarts.sh ---
@@ -51,12 +59,12 @@ sed -i -e ''"$(($position + 1))"'i\
 ${flink.version}\
--- End diff --

It would be _awesome_ if you could replace this hard-coded dependency with 
the one defined in the example project. But I wouldn't block the PR on it.


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r193394392
  
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -0,0 +1,98 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+   
+   org.apache.flink
+   flink-end-to-end-tests
+   1.6-SNAPSHOT
+   
+   4.0.0
+
+   flink-quickstart-test
+   flink-quickstart-test
+   jar
+
+   
+   2.11
+   
+
+   
+   
+   org.apache.flink
+   
flink-streaming-java_${scala.binary.version}
+   ${project.version}
+   provided
+   
+   
+   org.apache.flink
+   
flink-streaming-scala_${scala.binary.version}
+   ${project.version}
+   
+   
+   org.apache.flink
+   
flink-connector-elasticsearch5_${scala.binary.version}
+   ${project.version}
+   
+   
+
+   
--- End diff --

This section should be unnecessary.


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[medcv]

Github user medcv commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r191740715
  
--- Diff: flink-end-to-end-tests/test-scripts/elasticsearch-common.sh ---
@@ -75,6 +76,8 @@ function verify_result {
 }
 
 function shutdown_elasticsearch_cluster {
+   local index=$1
--- End diff --

+1


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[medcv]

Github user medcv commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r191740903
  
--- Diff: flink-end-to-end-tests/test-scripts/test_quickstarts.sh ---
@@ -18,29 +18,38 @@
 

 
 # End to end test for quick starts test.
+# Usage:
+# FLINK_DIR= 
flink-end-to-end-tests/test-scripts/test_quickstarts.sh  
--- End diff --

will change the name


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[medcv]

Github user medcv commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r191740667
  
--- Diff: flink-end-to-end-tests/test-scripts/elasticsearch-common.sh ---
@@ -56,13 +56,14 @@ function verify_elasticsearch_process_exist {
 
 function verify_result {
 local numRecords=$1
+local index=$2
--- End diff --

+1


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r191738239
  
--- Diff: flink-end-to-end-tests/test-scripts/elasticsearch-common.sh ---
@@ -75,6 +76,8 @@ function verify_result {
 }
 
 function shutdown_elasticsearch_cluster {
+   local index=$1
--- End diff --

adjust existing usages in `test_streaming_elasticsearch.sh`


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r191738228
  
--- Diff: flink-end-to-end-tests/test-scripts/elasticsearch-common.sh ---
@@ -56,13 +56,14 @@ function verify_elasticsearch_process_exist {
 
 function verify_result {
 local numRecords=$1
+local index=$2
--- End diff --

adjust existing usages in `test_streaming_elasticsearch.sh`


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r191737346
  
--- Diff: flink-end-to-end-tests/test-scripts/test_quickstarts.sh ---
@@ -18,29 +18,38 @@
 

 
 # End to end test for quick starts test.
+# Usage:
+# FLINK_DIR= 
flink-end-to-end-tests/test-scripts/test_quickstarts.sh  
--- End diff --

not sure how much value the `TestFileName` adds, given that it's the same 
for both tests and the package being hard-coded. At the very least it should be 
called `TestClassName`.


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[medcv]

Github user medcv commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r191733069
  
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -86,6 +102,21 @@ under the License.



+   
+   org.apache.maven.plugins
+   maven-enforcer-plugin
+   
+   
+   dependency-convergence
+   
+   enforce
+   
+   
+   true
--- End diff --

@zentol good point. this is not necessary anymore


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[medcv]

Github user medcv commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r191732783
  
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -20,27 +20,43 @@ under the License.
 http://maven.apache.org/POM/4.0.0;
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
-
-   4.0.0
-

org.apache.flink
flink-end-to-end-tests
1.6-SNAPSHOT
-   ..

+   4.0.0
 
-   flink-elasticsearch5-test
-   flink-elasticsearch5-test
+   flink-quickstart-test
+   flink-quickstart-test
jar
 
+   
+   2.11
+   
+


org.apache.flink

flink-streaming-java_${scala.binary.version}
${project.version}
provided

+   
+   org.apache.flink
+   
flink-streaming-scala_${scala.binary.version}
+   ${project.version}
+   
+   
+   org.apache.flink
+   
flink-connector-elasticsearch_${scala.binary.version}
+   ${project.version}
+   
+   
+   org.apache.flink
+   
flink-connector-elasticsearch2_${scala.binary.version}
--- End diff --

@zentol removed multiple connector versions


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r191677818
  
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -86,6 +102,21 @@ under the License.



+   
+   org.apache.maven.plugins
+   maven-enforcer-plugin
+   
+   
+   dependency-convergence
+   
+   enforce
+   
+   
+   true
--- End diff --

this was a good hint that what you were trying to do was problematic


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6089#discussion_r191677160
  
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -20,27 +20,43 @@ under the License.
 http://maven.apache.org/POM/4.0.0;
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
-
-   4.0.0
-

org.apache.flink
flink-end-to-end-tests
1.6-SNAPSHOT
-   ..

+   4.0.0
 
-   flink-elasticsearch5-test
-   flink-elasticsearch5-test
+   flink-quickstart-test
+   flink-quickstart-test
jar
 
+   
+   2.11
+   
+


org.apache.flink

flink-streaming-java_${scala.binary.version}
${project.version}
provided

+   
+   org.apache.flink
+   
flink-streaming-scala_${scala.binary.version}
+   ${project.version}
+   
+   
+   org.apache.flink
+   
flink-connector-elasticsearch_${scala.binary.version}
+   ${project.version}
+   
+   
+   org.apache.flink
+   
flink-connector-elasticsearch2_${scala.binary.version}
--- End diff --

depending multiple connector versions in one modules is a recipe for 
dependency conflicts and should be avoided at all costs. There's a reason why 
the existing examples and connector versions reside in different modules.


---

[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts

[medcv]

GitHub user medcv opened a pull request:

https://github.com/apache/flink/pull/6089

[FLINK-9451]End-to-end test: Scala Quickstarts

## What is the purpose of the change

Added an end-to-end test which verifies Flink's quickstarts scala. It does 
the following:

- create a new Flink project using the quickstarts scala archetype
- add a new Flink kafka connector dependency to the pom.xml 
- run mvn clean package
- verify that no core dependencies are contained in the jar file
- Run the program

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable / docs / 
JavaDocs / not documented)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/medcv/flink FLINK-9451

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/6089.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6089


commit 26f5948eff55fd54dbffc3125e0599479bee3bbe
Author: Yadan.JS 
Date:   2018-05-28T16:16:40Z

[FLINK-9451]End-to-end test: Scala Quickstarts




---