[GitHub] flink issue #5761: [FLINK-8989] [e2eTests] Elasticsearch1&2&5 end to end tes...
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5761 Thanks @tzulitai for review this patch! ~ ð ---
[GitHub] flink issue #5761: [FLINK-8989] [e2eTests] Elasticsearch1&2&5 end to end tes...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5761 +1, LGTM, merging ... ---
[GitHub] flink issue #5761: [FLINK-8989] [e2eTests] Elasticsearch1&2&5 end to end tes...
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/5761
Hi, @tzulitai Could you take another look on this PR ? It's the time for
having another review. Update the code according to your suggestions and I ran
the test on my local machine and below are the tests results. Looks good from
these results. Thank you very much.
```
Flink dist directory: /Users/zhangminglei/projects/flink/build-target
TEST_DATA_DIR:
/Users/zhangminglei/projects/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-20N
Downloading Elasticsearch1 from
https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.tar.gz
> Elasticsearch node is running.
Starting cluster.
Starting standalonesession daemon on host zhangmingleideMacBook-Pro.local.
Starting taskexecutor daemon on host zhangmingleideMacBook-Pro.local.
Waiting for dispatcher REST endpoint to come up...
Waiting for dispatcher REST endpoint to come up...
Waiting for dispatcher REST endpoint to come up...
Waiting for dispatcher REST endpoint to come up...
Dispatcher REST endpoint is up.
Starting execution of program
Program execution finished
Job with JobID dbd0dcbf09127ca1715da4bb3f4ca7fa has finished.
Job Runtime: 1791 ms
> Elasticsearch end to end test pass.
Stopping taskexecutor daemon (pid: 64950) on host
zhangmingleideMacBook-Pro.local.
Stopping standalonesession daemon (pid: 64658) on host
zhangmingleideMacBook-Pro.local.
No zookeeper daemon to stop on host zhangmingleideMacBook-Pro.local.
All tests PASS
Downloading Elasticsearch2 from
https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.5/elasticsearch-2.3.5.tar.gz
> Elasticsearch node is running.
Starting cluster.
Starting standalonesession daemon on host zhangmingleideMacBook-Pro.local.
[2018-05-22 09:16:16,773][INFO ][node ] [Orb]
version[2.3.5], pid[66176], build[90f439f/2016-07-27T10:36:52Z]
[2018-05-22 09:16:16,773][INFO ][node ] [Orb]
initializing ...
Starting taskexecutor daemon on host zhangmingleideMacBook-Pro.local.
Waiting for dispatcher REST endpoint to come up...
[2018-05-22 09:16:17,502][INFO ][plugins ] [Orb] modules
[reindex, lang-expression, lang-groovy], plugins [], sites []
[2018-05-22 09:16:17,533][INFO ][env ] [Orb] using [1]
data paths, mounts [[/ (/dev/disk1)]], net usable_space [287.8gb], net
total_space [464.6gb], spins? [unknown], types [hfs]
[2018-05-22 09:16:17,533][INFO ][env ] [Orb] heap size
[989.8mb], compressed ordinary object pointers [true]
[2018-05-22 09:16:17,534][WARN ][env ] [Orb] max file
descriptors [10240] for elasticsearch process likely too low, consider
increasing to at least [65536]
Waiting for dispatcher REST endpoint to come up...
Waiting for dispatcher REST endpoint to come up...
[2018-05-22 09:16:19,858][INFO ][node ] [Orb]
initialized
[2018-05-22 09:16:19,858][INFO ][node ] [Orb] starting
...
[2018-05-22 09:16:19,963][INFO ][transport] [Orb]
publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300},
{[::1]:9300}, {127.0.0.1:9300}
[2018-05-22 09:16:19,967][INFO ][discovery] [Orb]
elasticsearch/VmnVQsJKSDKtkXpxZY11Tg
Waiting for dispatcher REST endpoint to come up...
Dispatcher REST endpoint is up.
Starting execution of program
[2018-05-22 09:16:23,008][INFO ][cluster.service ] [Orb]
new_master {Orb}{VmnVQsJKSDKtkXpxZY11Tg}{127.0.0.1}{127.0.0.1:9300}, reason:
zen-disco-join(elected_as_master, [0] joins received)
[2018-05-22 09:16:23,023][INFO ][http ] [Orb]
publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200},
{[::1]:9200}, {127.0.0.1:9200}
[2018-05-22 09:16:23,023][INFO ][node ] [Orb] started
[2018-05-22 09:16:23,047][INFO ][gateway ] [Orb] recovered
[0] indices into cluster_state
[2018-05-22 09:16:24,579][INFO ][cluster.metadata ] [Orb] [index]
creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings
[type]
[2018-05-22 09:16:24,921][INFO ][cluster.routing.allocation] [Orb] Cluster
health status changed from [RED] to [YELLOW] (reason: [shards started
[[index][4]] ...]).
[2018-05-22 09:16:24,940][INFO ][cluster.metadata ] [Orb] [index]
update_mapping [type]
Program execution finished
Job with JobID ac42b71aeb6138100d9357b72bcf94f5 has finished.
Job Runtime: 1726 ms
> Elasticsearch end to end test pass.
[2018-05-22 09:16:26,552][INFO ][node ] [Orb] stopping
...
[2018-05-22 09:16:26,637][INFO ][node ] [Orb]
[GitHub] flink issue #5761: [FLINK-8989] [e2eTests] Elasticsearch1&2&5 end to end tes...
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5761 Hi, @tzulitai What is the state of this PR ? thanks. This PR is ready to have another review. ---
[GitHub] flink issue #5761: [FLINK-8989] [e2eTests] Elasticsearch1&2&5 end to end tes...
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5761 I think I found how to solve this issue. I will fix it as soon as possible. ---
[GitHub] flink issue #5761: [FLINK-8989] [e2eTests] Elasticsearch1&2&5 end to end tes...
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5761 I think I found how to solve this issue. I will fix it as soon as possible. ---
[GitHub] flink issue #5761: [FLINK-8989] [e2eTests] Elasticsearch1&2&5 end to end tes...
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5761 I think I found how to solve this issue. I will fix it as soon as possible. ---
[GitHub] flink issue #5761: [FLINK-8989] [e2eTests] Elasticsearch1&2&5 end to end tes...
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5761 I found an issue, I tried find it out but it seems useless. For elasticsearch5sink, I can run directly with the following command with flink cluster setup and elasticsearch5 cluster set up also without error. **[Flink cluster and ES cluster setup by manually.]** ``` $FLINK_DIR/bin/flink run -p 1 projects/flink/flink-end-to-end-tests/flink-elasticsearch5-test/target/Elasticsearch5SinkExample.jar --index index --type type ``` But when I change it to bash scripts to set up flink cluster and es5 cluster. Although it can be written to elasticsearch5 without error[we can see **Elasticsearch end to end test pass** in the following message]. In the end, it will throw an exception. ``` /Users/zhangminglei/projects/flink/flink-end-to-end-tests/test-scripts/../flink-elasticsearch5-test/target/Elasticsearch5SinkExample.jar Starting execution of program Program execution finished Job with JobID 2774e793a167a6bfba07031948fb5b14 has finished. Job Runtime: 6564 ms % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 4189 100 41890 0 19737 0 --:--:-- --:--:-- --:--:-- 19759 Elasticsearch end to end test pass. Stopping taskexecutor daemon (pid: 86784) on host zhangmingleideMacBook-Pro.local. Stopping standalonesession daemon (pid: 86475) on host zhangmingleideMacBook-Pro.local. Found non-empty .out files: Exception in thread "Thread-6" java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioWorker$RegisterTask at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioWorker.createRegisterTask(NioWorker.java:118) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.register(AbstractNioSelector.java:104) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioWorker.register(NioWorker.java:36) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:157) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:120) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 13 more One or more tests FAILED. ``` ---
