[
https://issues.apache.org/jira/browse/BAHIR-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701427#comment-16701427
]
ASF GitHub Bot commented on BAHIR-66:
-------------------------------------
GitHub user lukasz-antoniak opened a pull request:
https://github.com/apache/bahir/pull/71
[BAHIR-66] Switch to Java binding for ZeroMQ
Let me briefly describe the story behind hereby pull request.
Initially, I just wanted to implement integration test for BAHIR-66. Google
pointed me to JeroMQ, which provides official ZeroMQ binding for Java and does
not require native libraries. I have decided to give it a try, but quickly
realized that _akka-zeromq_ module (transient dependency from current Bahir
master) is not compatible with JeroMQ. Actually Akka team also wanted to move
to JeroMQ (https://github.com/akka/akka/issues/13856), but in the end decided
to remove _akka-zeromq_ project completely
(https://github.com/akka/akka/issues/15864,
https://www.lightbend.com/blog/akka-roadmap-update-2014). Having in mind that
_akka-zeromq_ does not support latest version of ZeroMQ protocol and further
development may come delayed, I have decided to refactor _streaming-zeromq_
implementation and leverage JeroMQ. With the change we receive various
benefits, such as support for PUB-SUB and PUSH-PULL messaging patterns and the
ability to bind the socket on whatever end of communication channel (see test
cases), subscription to multiple channels, etc. JeroMQ seems pretty reliable
and reconnection is handled out-of-the-box. Actually, we could even start the
ZeroMQ subscriber trying to connect to remote socket before other end created
and bound the socket. While I tried to preserve backward compatibility of
method signatures, there was no easy way to support Akka API and business logic
that users could put there (e.g. _akka.actor.ActorSystem_).
Comments welcome. I hope the pull request makes sense and you will like
Spark integration with ZeroMQ without dependency on Akka framework.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lukasz-antoniak/bahir BAHIR-66
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bahir/pull/71.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #71
----
commit 48a82946240ab36834e2d1eeab2c007daad9c53a
Author: Lukasz Antoniak <lukasz.antoniak@...>
Date: 2018-11-27T14:58:42Z
[BAHIR-66] Switch to Java binding for ZeroMQ
----
> Add test that ZeroMQ streaming connector can receive data
> ---------------------------------------------------------
>
> Key: BAHIR-66
> URL: https://issues.apache.org/jira/browse/BAHIR-66
> Project: Bahir
> Issue Type: Sub-task
> Components: Spark Streaming Connectors
> Reporter: Christian Kadner
> Priority: Major
> Labels: test
>
> Add test cases that verify that the *ZeroMQ streaming connector* can receive
> streaming data.
> See [BAHIR-63|https://issues.apache.org/jira/browse/BAHIR-63]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
