infearOnTheWay commented on a change in pull request #2603: Add nio support for mysql protocol implementation URL: https://github.com/apache/incubator-doris/pull/2603#discussion_r362385781
########## File path: fe/src/main/java/org/apache/doris/mysql/nio/NMysqlChannel.java ########## @@ -0,0 +1,111 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +package org.apache.doris.mysql.nio; + +import org.apache.doris.mysql.MysqlChannel; +import org.apache.doris.qe.ConnectProcessor; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; +import org.xnio.StreamConnection; +import org.xnio.channels.Channels; + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.nio.ByteBuffer; + +/** + * mysql Channel based on nio. + */ +public class NMysqlChannel extends MysqlChannel { + protected final Logger LOG = LogManager.getLogger(this.getClass()); + private StreamConnection conn; + + public NMysqlChannel(StreamConnection connection) { + super(); + this.conn = connection; + if (connection.getPeerAddress() instanceof InetSocketAddress) { + InetSocketAddress address = (InetSocketAddress) connection.getPeerAddress(); + remoteHostPortString = address.getHostString() + ":" + address.getPort(); + remoteIp = address.getAddress().getHostAddress(); + } else { + // Reach here, what's it? + remoteHostPortString = connection.getPeerAddress().toString(); + remoteIp = connection.getPeerAddress().toString(); + } + } + + /** + * read packet until all data is ready, unless block. + * + * @param dstBuf + * @return + * @throws IOException + */ + @Override + protected int readAll(ByteBuffer dstBuf) throws IOException { + int ret = Channels.readBlocking(conn.getSourceChannel(), dstBuf); + // return -1 when remote peer close the channel + if (ret == -1) { + return 0; + } + return ret; Review comment: @lingbin > A simple solution to achieve this is to add a loop logic(like the existing implementation), to make sure this function will not return until enough data is read,but this may take a long time for the IO thread because it is not sure what time the remaining packets are will arrive. The loop logic is currentlt retained in `MysqlChannel.fetchOnePacket`, and only the method `readAll` is overwritten based on nio. So if one logic packets is split to multiple TCP packets, all packets would be read and compose. On the other hand, this read process is handled by **task thread** instead of **io thread**(io thread just notify readlistenter while read event arrived). So it would not matter. > When the MySQL request message is large, it will be split into multiple TCP packets, but it is only read once here, and readBlocking() does not guarantee that the content of the parameter dstBuf is filled The source code of `readBlocking` is as belows: ``` public static <C extends ReadableByteChannel & SuspendableReadChannel> int readBlocking(C channel, ByteBuffer buffer) throws IOException { int res; while ((res = channel.read(buffer)) == 0 && buffer.hasRemaining()) { channel.awaitReadable(); } return res; } ``` It seems that `dstbuffer` would be filled, unless one TCP packet is split to multiple TCP packets. In other words, **different parts of one TCP packets** do not arrived at same time. Then just part is read and cause error in upper layer. Is it this the problem you concerned about? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
