[
https://issues.apache.org/jira/browse/DRILL-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382997#comment-16382997
]
ASF GitHub Bot commented on DRILL-6187:
---------------------------------------
Github user sohami commented on the issue:
https://github.com/apache/drill/pull/1145
@vrozov - Please help to review this PR.
It address the concurrency issue during authentication of control/data
client to server side. Rather than adding the connection into connection holder
right after TCP connection is available, the listener for connection success is
called after successful authentication (if needed).
> Exception in RPC communication between DataClient/ControlClient and
> respective servers when bit-to-bit security is on
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: DRILL-6187
> URL: https://issues.apache.org/jira/browse/DRILL-6187
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC, Security
> Reporter: Sorabh Hamirwasia
> Assignee: Sorabh Hamirwasia
> Priority: Major
> Fix For: 1.13.0
>
>
>
> {color:#000000}Below is the summary of issue: {color}
>
> {color:#000000}*Scenario:*{color}
> {color:#000000}It seems like first sendRecordBatch was sent to Foreman which
> initiated the Authentication handshake. But before initiating handshake for
> auth we establish a connection and store that in a registry. Now if in
> parallel there is another recordBatch (by a different minor fragment running
> on same Drillbit) to be sent then that will see the connection available in
> registry and will initiate the send. Before the authentication is completed
> this second request reached foreman and it throws below exception saying RPC
> type 3 message is not allowed and closes the connection. This also fails the
> authentication handshake which was in progress.{color}{color:#000000} Here
> the logs with details:{color}
> {color:#000000} {color}
> {color:#000000}*Forman received the SASL_START message from another
> node:*{color}
> {color:#000000}*_2018-02-21 18:43:30,759
> [_*{color}{color:#000000}_BitServer-4] TRACE
> o.a.d.e.r.s.ServerAuthenticationHandler - Received SASL message SASL_START
> from /10.10.100.161:35482_{color}
> {color:#000000} {color}
> {color:#000000}*Then around same time it received another message from client
> of Rpc Type 3 which is for SendRecordBatch and fails since handshake is not
> completed yet.*{color}
> {color:#000000} {color}
> {color:#000000}*_2018-02-21 18:43:30,762_*{color}{color:#000000}
> _[BitServer-4] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC
> communication. Connection: /10.10.100.162:31012 <--> /__10.10.100.161:35482_
> _(data server). Closing connection._{color}
> {color:#000000}_io.netty.handler.codec.DecoderException:
> org.apache.drill.exec.rpc.RpcException: Request of type 3 is not allowed
> without authentication. Client on /__10.10.100.161:35482_ _must authenticate
> before making requests. Connection dropped. [Details: Encryption: enabled ,
> MaxWrappedSize: 65536 , WrapSizeLimit: 0]_{color}
> {color:#000000} {color}
> {color:#000000}*Then client receives an channel closed exception:*{color}
> {color:#000000} {color}
> {color:#000000}*2018-02-21 18:43:30,764 [*{color}{color:#000000}BitClient-4]
> WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed
> channel. Connection: /_10.10.100.161:35482_ <--> _10.10.100.162:31012_ (data
> client){color}
> {color:#000000} {color}
> {color:#000000}*and due to this it's initial command for authentication also
> fails. Since there is channel closed exception above I will think that
> triggered the failure of authentication request as well.*{color}
> {color:#000000} {color}
> {color:#000000}_Caused by: org.apache.drill.exec.rpc.RpcException: Command
> failed while establishing connection. Failure type AUTHENTICATION._{color}
> {color:#000000} _at
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} _at
> org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} _at
> org.apache.drill.exec.rpc.data.DataTunnel$SendBatchAsyncListen.connectionFailed(DataTunnel.java:166)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} _at
> org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:203)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} _at
> org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:147)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} _at
> org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.waitAndRun(ReconnectingConnection.java:122)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} _at
> org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:83)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} _at
> org.apache.drill.exec.rpc.data.DataTunnel._{color}{color:#000000}*_sendRecordBatch_*{color}{color:#000000}_(DataTunnel.java:84)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} _at
> org.apache.drill.exec.ops.AccountingDataTunnel.sendRecordBatch(AccountingDataTunnel.java:45)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} _at
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:127)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} {color}
> {color:#000000}So I think there is a concurrency issue where even though the
> authentication is not completed the other requests are send to remote node as
> soon as TCP connection is available. Instead it should wait until
> authentication is completed. Something like TCP connection should be made
> available from registry only if authentication is completed.{color}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
