[jira] [Commented] (RATIS-72) Enable Ratis to run on high-performance RDMA networks
[ https://issues.apache.org/jira/browse/RATIS-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243288#comment-16243288 ] Tsz Wo Nicholas Sze commented on RATIS-72: -- [~atrivedi], please feel free to file JIRA if you see test failures. You may first check the output files (in ./ratis-Xxx/target/surefire-reports) and post the files in the JIRA. Thank you in advance! > Enable Ratis to run on high-performance RDMA networks > - > > Key: RATIS-72 > URL: https://issues.apache.org/jira/browse/RATIS-72 > Project: Ratis > Issue Type: New Feature >Reporter: Animesh Trivedi >Assignee: Animesh Trivedi >Priority: Minor > Attachments: failed-test > > > We want to enable Ratis to run on high-performance RDMA networks. There has > already been quite a bit of activity in leveraging RDMA networks in modern > data processing stacks. Distributed consensus is one of the problems that can > be accelerated using high-performance/RDMA messaging. > We start by implementing a Ratis RPC implementation using DiSNI and DaPRC > modules (https://github.com/zrlio/) for RDMA networks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RATIS-72) Enable Ratis to run on high-performance RDMA networks
[
https://issues.apache.org/jira/browse/RATIS-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243260#comment-16243260
]
Tsz Wo Nicholas Sze commented on RATIS-72:
--
{code}
TestRaftWithHadoopRpc>RaftBasicTests.testBasicAppendEntries:127->RaftBasicTests.lambda$testBasicAppendEntries$1:127
expected:<10> but was:<11>
{code}
This seems a bug in the test. Let me file a JIRA.
> Enable Ratis to run on high-performance RDMA networks
> -
>
> Key: RATIS-72
> URL: https://issues.apache.org/jira/browse/RATIS-72
> Project: Ratis
> Issue Type: New Feature
>Reporter: Animesh Trivedi
>Assignee: Animesh Trivedi
>Priority: Minor
> Attachments: failed-test
>
>
> We want to enable Ratis to run on high-performance RDMA networks. There has
> already been quite a bit of activity in leveraging RDMA networks in modern
> data processing stacks. Distributed consensus is one of the problems that can
> be accelerated using high-performance/RDMA messaging.
> We start by implementing a Ratis RPC implementation using DiSNI and DaPRC
> modules (https://github.com/zrlio/) for RDMA networks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (RATIS-72) Enable Ratis to run on high-performance RDMA networks
[
https://issues.apache.org/jira/browse/RATIS-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239858#comment-16239858
]
Jitendra Nath Pandey commented on RATIS-72:
---
[~atrivedi], Could you please run TestRaftWithSimulatedRpc individually?
{code}
mvn test -Dtest=TestRaftWithSimulatedRpc
{code}
I tried a few runs, and it passes for me on the latest code. Does it fail
consistently for you?
There have been some maven changes recently. You would need to do {{mvn package
-DskipTests}}, after {{mvn clean}} to get shaded jars correctly built. See
latest BUILDING.md file.
RATIS-132, once fixed, will get {{mvn clean install}} working.
> Enable Ratis to run on high-performance RDMA networks
> -
>
> Key: RATIS-72
> URL: https://issues.apache.org/jira/browse/RATIS-72
> Project: Ratis
> Issue Type: New Feature
>Reporter: Animesh Trivedi
>Assignee: Animesh Trivedi
>Priority: Minor
> Attachments: failed-test
>
>
> We want to enable Ratis to run on high-performance RDMA networks. There has
> already been quite a bit of activity in leveraging RDMA networks in modern
> data processing stacks. Distributed consensus is one of the problems that can
> be accelerated using high-performance/RDMA messaging.
> We start by implementing a Ratis RPC implementation using DiSNI and DaPRC
> modules (https://github.com/zrlio/) for RDMA networks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (RATIS-72) Enable Ratis to run on high-performance RDMA networks
[ https://issues.apache.org/jira/browse/RATIS-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239699#comment-16239699 ] Animesh Trivedi commented on RATIS-72: -- Hi [~jnp]], many thanks for offering your help. I am trying to finish this implementation now. One thing that would be helpful is to get the maven unit tests working. Currently, I when I run mvn test, I get : [...] Failed tests: TestRaftWithSimulatedRpc>RaftBasicTests.testBasicAppendEntries:127->RaftBasicTests.lambda$testBasicAppendEntries$1:127 expected:<10> but was:<11> Tests run: 65, Failures: 1, Errors: 0, Skipped: 0 [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Ratis ... SUCCESS [ 1.014 s] [INFO] Apache Ratis Proto Shaded .. SUCCESS [ 3.353 s] [INFO] Apache Ratis Common SUCCESS [ 5.008 s] [INFO] Apache Ratis Client SUCCESS [ 0.080 s] [INFO] Apache Ratis Hadoop Shaded . SUCCESS [ 1.397 s] [INFO] Apache Ratis Server FAILURE [01:41 min] [INFO] Apache Ratis Hadoop Support SKIPPED [INFO] Apache Ratis gRPC Support .. SKIPPED [INFO] Apache Ratis Netty Support . SKIPPED [INFO] Apache Ratis Replicated Map SKIPPED [INFO] Apache Ratis Project Assembly .. SKIPPED [INFO] Apache Ratis Examples .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] As I am trying get the similar functionality implemented as the gRPC or netty code, it would be helpful to have the basic tests working. Naturally, debugging in a distributed setting requires another level of testing. Thanks a lot ! > Enable Ratis to run on high-performance RDMA networks > - > > Key: RATIS-72 > URL: https://issues.apache.org/jira/browse/RATIS-72 > Project: Ratis > Issue Type: New Feature >Reporter: Animesh Trivedi >Assignee: Animesh Trivedi >Priority: Minor > Attachments: failed-test > > > We want to enable Ratis to run on high-performance RDMA networks. There has > already been quite a bit of activity in leveraging RDMA networks in modern > data processing stacks. Distributed consensus is one of the problems that can > be accelerated using high-performance/RDMA messaging. > We start by implementing a Ratis RPC implementation using DiSNI and DaPRC > modules (https://github.com/zrlio/) for RDMA networks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RATIS-72) Enable Ratis to run on high-performance RDMA networks
[ https://issues.apache.org/jira/browse/RATIS-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234954#comment-16234954 ] Jitendra Nath Pandey commented on RATIS-72: --- [~atrivedi], are you able to make progress on this? Let us know if you need any help. > Enable Ratis to run on high-performance RDMA networks > - > > Key: RATIS-72 > URL: https://issues.apache.org/jira/browse/RATIS-72 > Project: Ratis > Issue Type: New Feature >Reporter: Animesh Trivedi >Assignee: Animesh Trivedi >Priority: Minor > Attachments: failed-test > > > We want to enable Ratis to run on high-performance RDMA networks. There has > already been quite a bit of activity in leveraging RDMA networks in modern > data processing stacks. Distributed consensus is one of the problems that can > be accelerated using high-performance/RDMA messaging. > We start by implementing a Ratis RPC implementation using DiSNI and DaPRC > modules (https://github.com/zrlio/) for RDMA networks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RATIS-72) Enable Ratis to run on high-performance RDMA networks
[ https://issues.apache.org/jira/browse/RATIS-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195187#comment-16195187 ] Animesh Trivedi commented on RATIS-72: -- I just updated my 0.1 branch to github master and the compilation is broken for me. Any idea what am I missing? [INFO] - [INFO] - [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/atr/zrl/external/github/animeshtrivedi/incubator-ratis/ratis-common/src/main/java/org/apache/ratis/util/ProtoUtils.java:[103,43] cannot find symbol symbol: class RaftGroupIdProto location: class org.apache.ratis.util.ProtoUtils [ERROR] /home/atr/zrl/external/github/animeshtrivedi/incubator-ratis/ratis-common/src/main/java/org/apache/ratis/util/ProtoUtils.java:[107,33] package RaftGroupIdProto does not exist [ERROR] /home/atr/zrl/external/github/animeshtrivedi/incubator-ratis/ratis-common/src/main/java/org/apache/ratis/util/ProtoUtils.java:[111,39] cannot find symbol symbol: class RaftGroupProto location: class org.apache.ratis.util.ProtoUtils [ERROR] /home/atr/zrl/external/github/animeshtrivedi/incubator-ratis/ratis-common/src/main/java/org/apache/ratis/util/ProtoUtils.java:[116,31] package RaftGroupProto does not exist [ERROR] /home/atr/zrl/external/github/animeshtrivedi/incubator-ratis/ratis-common/src/main/java/org/apache/ratis/util/ProtoUtils.java:[108,12] cannot find symbol symbol: variable RaftGroupIdProto location: class org.apache.ratis.util.ProtoUtils [ERROR] /home/atr/zrl/external/github/animeshtrivedi/incubator-ratis/ratis-common/src/main/java/org/apache/ratis/util/ProtoUtils.java:[117,12] cannot find symbol symbol: variable RaftGroupProto location: class org.apache.ratis.util.ProtoUtils > Enable Ratis to run on high-performance RDMA networks > - > > Key: RATIS-72 > URL: https://issues.apache.org/jira/browse/RATIS-72 > Project: Ratis > Issue Type: New Feature >Reporter: Animesh Trivedi >Assignee: Animesh Trivedi >Priority: Minor > Attachments: failed-test > > > We want to enable Ratis to run on high-performance RDMA networks. There has > already been quite a bit of activity in leveraging RDMA networks in modern > data processing stacks. Distributed consensus is one of the problems that can > be accelerated using high-performance/RDMA messaging. > We start by implementing a Ratis RPC implementation using DiSNI and DaPRC > modules (https://github.com/zrlio/) for RDMA networks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RATIS-72) Enable Ratis to run on high-performance RDMA networks
[ https://issues.apache.org/jira/browse/RATIS-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194781#comment-16194781 ] Sen Zhang commented on RATIS-72: [~atrivedi] Hi Animesh, did you find a way to resolve this test failure? I have the same issue when I run maven test. > Enable Ratis to run on high-performance RDMA networks > - > > Key: RATIS-72 > URL: https://issues.apache.org/jira/browse/RATIS-72 > Project: Ratis > Issue Type: New Feature >Reporter: Animesh Trivedi >Assignee: Animesh Trivedi >Priority: Minor > Attachments: failed-test > > > We want to enable Ratis to run on high-performance RDMA networks. There has > already been quite a bit of activity in leveraging RDMA networks in modern > data processing stacks. Distributed consensus is one of the problems that can > be accelerated using high-performance/RDMA messaging. > We start by implementing a Ratis RPC implementation using DiSNI and DaPRC > modules (https://github.com/zrlio/) for RDMA networks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RATIS-72) Enable Ratis to run on high-performance RDMA networks
[ https://issues.apache.org/jira/browse/RATIS-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982517#comment-15982517 ] Animesh Trivedi commented on RATIS-72: -- [~jingzhao] I have attached the complete log for mvn test failure. https://issues.apache.org/jira/secure/attachment/12864913/failed-test Thanks > Enable Ratis to run on high-performance RDMA networks > - > > Key: RATIS-72 > URL: https://issues.apache.org/jira/browse/RATIS-72 > Project: Ratis > Issue Type: New Feature >Reporter: Animesh Trivedi >Assignee: Animesh Trivedi >Priority: Minor > Attachments: failed-test > > > We want to enable Ratis to run on high-performance RDMA networks. There has > already been quite a bit of activity in leveraging RDMA networks in modern > data processing stacks. Distributed consensus is one of the problems that can > be accelerated using high-performance/RDMA messaging. > We start by implementing a Ratis RPC implementation using DiSNI and DaPRC > modules (https://github.com/zrlio/) for RDMA networks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (RATIS-72) Enable Ratis to run on high-performance RDMA networks
[ https://issues.apache.org/jira/browse/RATIS-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981221#comment-15981221 ] Animesh Trivedi commented on RATIS-72: -- Hi [~szetszwo] - I am trying to setup a ratis test environment but am having issues to get it right. If I run `mvn test` (on the April 17th git master), the test fails with either HadoopRPC, GRPC or Netty Unit tests - one of them, but which one fails, is not fixed. Is this known? What is the mvn test command I should use? For example, when the test fails with Hadoop RPC: --- T E S T S --- Picked up JAVA_TOOL_OPTIONS: Running org.apache.ratis.hadooprpc.TestNotLeaderExceptionWithHadoopRpc Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.738 sec - in org.apache.ratis.hadooprpc.TestNotLeaderExceptionWithHadoopRpc Picked up JAVA_TOOL_OPTIONS: Running org.apache.ratis.hadooprpc.TestRaftReconfigurationWithHadoopRpc Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 52.511 sec <<< FAILURE! - in org.apache.ratis.hadooprpc.TestRaftReconfigurationWithHadoopRpc testRevertConfigurationChange(org.apache.ratis.hadooprpc.TestRaftReconfigurationWithHadoopRpc) Time elapsed: 3.108 sec <<< FAILURE! java.lang.AssertionError: expected:<0> but was:<2> Picked up JAVA_TOOL_OPTIONS: Running org.apache.ratis.hadooprpc.TestRaftWithHadoopRpc Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.544 sec - in org.apache.ratis.hadooprpc.TestRaftWithHadoopRpc Picked up JAVA_TOOL_OPTIONS: Running org.apache.ratis.hadooprpc.TestRetryCacheWithHadoopRpc Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.3 sec - in org.apache.ratis.hadooprpc.TestRetryCacheWithHadoopRpc Picked up JAVA_TOOL_OPTIONS: Running org.apache.ratis.hadooprpc.TestRaftSnapshotWithHadoopRpc Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.317 sec - in org.apache.ratis.hadooprpc.TestRaftSnapshotWithHadoopRpc Results : Failed tests: TestRaftReconfigurationWithHadoopRpc>RaftReconfigurationBaseTest.testRevertConfigurationChange:531 expected:<0> but was:<2> Tests run: 22, Failures: 1, Errors: 0, Skipped: 0 Or when the test fails with netty: --- T E S T S --- Picked up JAVA_TOOL_OPTIONS: -XX:+PreserveFramePointer -ea Running org.apache.ratis.netty.TestRaftSnapshotWithNetty Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.68 sec - in org.apache.ratis.netty.TestRaftSnapshotWithNetty Picked up JAVA_TOOL_OPTIONS: -XX:+PreserveFramePointer -ea Running org.apache.ratis.netty.TestRetryCacheWithNettyRpc Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.715 sec - in org.apache.ratis.netty.TestRetryCacheWithNettyRpc Picked up JAVA_TOOL_OPTIONS: -XX:+PreserveFramePointer -ea Running org.apache.ratis.netty.TestRaftWithNetty Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.9 sec <<< FAILURE! - in org.apache.ratis.netty.TestRaftWithNetty testBasicAppendEntries(org.apache.ratis.netty.TestRaftWithNetty) Time elapsed: 1.607 sec <<< FAILURE! java.lang.AssertionError: expected:<10> but was:<11> Picked up JAVA_TOOL_OPTIONS: -XX:+PreserveFramePointer -ea Running org.apache.ratis.netty.TestNotLeaderExceptionWithNetty Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.909 sec - in org.apache.ratis.netty.TestNotLeaderExceptionWithNetty Picked up JAVA_TOOL_OPTIONS: -XX:+PreserveFramePointer -ea Running org.apache.ratis.netty.TestRaftReconfigurationWithNetty Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.29 sec - in org.apache.ratis.netty.TestRaftReconfigurationWithNetty Results : Failed tests: TestRaftWithNetty>RaftBasicTests.testBasicAppendEntries:102->RaftBasicTests.lambda$testBasicAppendEntries$1:102 expected:<10> but was:<11> Tests run: 22, Failures: 1, Errors: 0, Skipped: 0 > Enable Ratis to run on high-performance RDMA networks > - > > Key: RATIS-72 > URL: https://issues.apache.org/jira/browse/RATIS-72 > Project: Ratis > Issue Type: New Feature >Reporter: Animesh Trivedi >Assignee: Animesh Trivedi >Priority: Minor > > We want to enable Ratis to run on high-performance RDMA networks. There has > already been quite a bit of activity in leveraging RDMA networks in modern > data processing stacks. Distributed consensus is one of the problems that can > be accelerated using high-performance/RDMA messaging. > We start by implementing a Ratis RPC implementation using DiSNI and DaPRC > modules (https://github.com/zrlio/) for RDMA networks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (RATIS-72) Enable Ratis to run on high-performance RDMA networks
[ https://issues.apache.org/jira/browse/RATIS-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967378#comment-15967378 ] Tsz Wo Nicholas Sze commented on RATIS-72: -- Thanks [~atrivedi]. You may use this as an umbrella JIRA and file subtasks for the works. > Enable Ratis to run on high-performance RDMA networks > - > > Key: RATIS-72 > URL: https://issues.apache.org/jira/browse/RATIS-72 > Project: Ratis > Issue Type: New Feature >Reporter: Animesh Trivedi >Assignee: Animesh Trivedi >Priority: Minor > > We want to enable Ratis to run on high-performance RDMA networks. There has > already been quite a bit of activity in leveraging RDMA networks in modern > data processing stacks. Distributed consensus is one of the problems that can > be accelerated using high-performance/RDMA messaging. > We start by implementing a Ratis RPC implementation using DiSNI and DaPRC > modules (https://github.com/zrlio/) for RDMA networks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
