Author: elecharny Date: Wed Apr 3 10:33:31 2019 New Revision: 1856869 URL: http://svn.apache.org/viewvc?rev=1856869&view=rev Log: Added some techincal documentation
Added: mina/site/trunk/content/mina-project/technical-documentation/ mina/site/trunk/content/mina-project/technical-documentation/message-flow.mdtext mina/site/trunk/content/mina-project/technical-documentation/ssl-tls-internal.mdtext Removed: mina/site/trunk/content/mina-project/documentation/ Modified: mina/site/trunk/content/mina-project/documentation.mdtext mina/site/trunk/content/mina-project/userguide/ch11-ssl-filter/ch11-ssl-filter.mdtext Modified: mina/site/trunk/content/mina-project/documentation.mdtext URL: http://svn.apache.org/viewvc/mina/site/trunk/content/mina-project/documentation.mdtext?rev=1856869&r1=1856868&r2=1856869&view=diff ============================================================================== --- mina/site/trunk/content/mina-project/documentation.mdtext (original) +++ mina/site/trunk/content/mina-project/documentation.mdtext Wed Apr 3 10:33:31 2019 @@ -51,9 +51,9 @@ You might also want to read the [frequen * [MINA v2.0 Quick Start Guide](quick-start-guide.html) - Create your first MINA based program using MINA version 2.0 * [Logging Configuration](userguide/ch12-logging-filter/ch12-logging-filter.html) - Configuring your MINA-based application for logging * Transport-specific Configuration - * [Serial Tutorial](userguide/ch6-transports/serial-transport.html) - Serial communications with MINA trunk + * [Serial Tutorial](userguide/ch6-transports/ch6.2-serial-transport.html) - Serial communications with MINA trunk * [UDP Tutorial](userguide/ch6-transports/ch6-transports) - Writing a User Datagram Protocol (UDP) client and server using MINA - * [APR Transport](userguide/ch6-transports/apr-transport.html) - Describes use of APR Transport with MINA + * [APR Transport](userguide/ch6-transports/ch6.1-apr-transport.html) - Describes use of APR Transport with MINA * [Integrating with Spring](userguide/ch17-spring-integration/ch17-spring-integration.html) - Demonstrates how to integrate MINA application with Spring * [Codec Repository](codec-repo.html) - Links to available codec implementations for MINA * Advanced Topic Added: mina/site/trunk/content/mina-project/technical-documentation/message-flow.mdtext URL: http://svn.apache.org/viewvc/mina/site/trunk/content/mina-project/technical-documentation/message-flow.mdtext?rev=1856869&view=auto ============================================================================== --- mina/site/trunk/content/mina-project/technical-documentation/message-flow.mdtext (added) +++ mina/site/trunk/content/mina-project/technical-documentation/message-flow.mdtext Wed Apr 3 10:33:31 2019 @@ -0,0 +1,206 @@ +Title: Message flow +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +# Message flow + +When a selector receives some _OP_READ_ event, the associated _IoProcessor_ is awaken and starts to process this event. It all starts with reading the channel, then pushing the read data through the filter chain down to the _IoHandler_ (aka, the application). If the application decides to write back a response, it goes through the chain again, andultimately the data to be pushed into the channel is pushed into a queue (_writeRequestQueue), then the stack is unfolled, down to the _IoProcessor_, which pops the data from the queue and writes it into the channel. + +Here is the representation of such a flow, where the _SslFilter_ has been included (we will come back to it later) + + ::: text + +-------------------+ + | IoAcceptor | + | +---------+ | + | | Select | | + | +----+----+ | + | | | + +---------+---------+ + . flush/ + +---------+ . +---------+ wakeup + | Channel <-----read------+ . +-----write-----> Channel | +----+ + +---------+ | | | +---------+ | | + +---+--v--+---+ +----v----+---+ + | | writeRequestQueue | | + | IoProcessor | +-+-+-+-+-+-+-+-+ | IoProcessor | + | <- pop -+ | | | | | | | <- put- | | + +---+-----^---+ +-+-+-+-+-+-+-+-+ +---------^---+ + | . . | + fireMessageReceived | . . | write + | . . | + +---v---------+ +---v-----+---+ + | HeadFilter | | HeadFilter <---------------filterWrite---------------+ + +---+-----^---+ +-------------+ | + | . . +-----+------+ + MessageReceived | . . +----------flushScheduledEvents--------> SslHandler | + | . . | +-----+------+ + +---v---------+ +---v------+--+ | + | SslFilter | | SslFilter +----put----+ | + +---+-----^---+ +---------^---+ | | + | . . | | filterWriteQueue | + MessageReceived | . . | filterWrite | +-+-+-+-+-+-+-+-+ | + | . . | +--> | | | | | | | <----pop---+ + +---v---------+ +---v---------+ +-+-+-+-+-+-+-+-+ + | <filters> | | <filters> | + +---+-----v---+ +-------------+ + | . . | + MessageReceived | . . | filterWrite + | . . | + +---v---------+ +---v-----+---+ + | TailFilter | | TailFilter | + +---+-----^---+ +---------^---+ + | . . | + MessageReceived | . . | fireFilterWrite + | . . | + +---v---------+ . | + | IoHandler | . | + +---+-----^---+ . | + | . . | + | ......................................... | + | | + +-------------------sessionWrite--------------------+ + + +For instance, if we consider an application which filter chain contains the _MdcInjectionFilter_, _SslFilter_, _CompressionFilter_, _ProtocolCodecFilter_, _LoggingFilter_, we end uo with a stack of 77 calls, which needs to be emptied before the response can eventually be written. Here is the stack (in reverse order, the latest call on top of it): + + ::: text + NioProcessor(AbstractPollingIoProcessor<S>).flush(S) line: 449 + NioProcessor(AbstractPollingIoProcessor<S>).write(S, WriteRequest) line: 436 + NioProcessor(AbstractPollingIoProcessor<S>).write(IoSession, WriteRequest) line: 68 + SimpleIoProcessorPool<S>.write(S, WriteRequest) line: 278 + SimpleIoProcessorPool<S>.write(IoSession, WriteRequest) line: 80 + DefaultIoFilterChain$HeadFilter.filterWrite(IoFilter$NextFilter, IoSession, WriteRequest) line: 914 + DefaultIoFilterChain.callPreviousFilterWrite(IoFilterChain$Entry, IoSession, WriteRequest) line: 753 + DefaultIoFilterChain.access$1500(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, WriteRequest) line: 49 + DefaultIoFilterChain$EntryImpl$1.filterWrite(IoSession, WriteRequest) line: 1146 + IoFilterEvent.fire() line: 131 + MdcInjectionFilter.filter(IoFilterEvent) line: 162 + MdcInjectionFilter(CommonEventFilter).filterWrite(IoFilter$NextFilter, IoSession, WriteRequest) line: 68 + DefaultIoFilterChain.callPreviousFilterWrite(IoFilterChain$Entry, IoSession, WriteRequest) line: 753 + DefaultIoFilterChain.access$1500(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, WriteRequest) line: 49 + DefaultIoFilterChain$EntryImpl$1.filterWrite(IoSession, WriteRequest) line: 1146 + SslHandler.flushScheduledEvents() line: 330 + SslFilter.filterWrite(IoFilter$NextFilter, IoSession, WriteRequest) line: 671 + DefaultIoFilterChain.callPreviousFilterWrite(IoFilterChain$Entry, IoSession, WriteRequest) line: 753 + DefaultIoFilterChain.access$1500(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, WriteRequest) line: 49 + DefaultIoFilterChain$EntryImpl$1.filterWrite(IoSession, WriteRequest) line: 1146 + CompressionFilter.filterWrite(IoFilter$NextFilter, IoSession, WriteRequest) line: 152 + DefaultIoFilterChain.callPreviousFilterWrite(IoFilterChain$Entry, IoSession, WriteRequest) line: 753 + DefaultIoFilterChain.access$1500(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, WriteRequest) line: 49 + DefaultIoFilterChain$EntryImpl$1.filterWrite(IoSession, WriteRequest) line: 1146 + ProtocolCodecFilter.filterWrite(IoFilter$NextFilter, IoSession, WriteRequest) line: 340 + DefaultIoFilterChain.callPreviousFilterWrite(IoFilterChain$Entry, IoSession, WriteRequest) line: 753 + DefaultIoFilterChain.access$1500(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, WriteRequest) line: 49 + DefaultIoFilterChain$EntryImpl$1.filterWrite(IoSession, WriteRequest) line: 1146 + LoggingFilter(IoFilterAdapter).filterWrite(IoFilter$NextFilter, IoSession, WriteRequest) line: 138 + DefaultIoFilterChain.callPreviousFilterWrite(IoFilterChain$Entry, IoSession, WriteRequest) line: 753 + DefaultIoFilterChain.access$1500(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, WriteRequest) line: 49 + DefaultIoFilterChain$EntryImpl$1.filterWrite(IoSession, WriteRequest) line: 1146 + DefaultIoFilterChain$TailFilter(IoFilterAdapter).filterWrite(IoFilter$NextFilter, IoSession, WriteRequest) line: 138 + DefaultIoFilterChain.callPreviousFilterWrite(IoFilterChain$Entry, IoSession, WriteRequest) line: 753 + DefaultIoFilterChain.fireFilterWrite(WriteRequest) line: 746 + NioSocketSession(AbstractIoSession).write(Object, SocketAddress) line: 570 + NioSocketSession(AbstractIoSession).write(Object) line: 515 + ChatProtocolHandler.messageReceived(IoSession, Object) line: 106 + DefaultIoFilterChain$TailFilter.messageReceived(IoFilter$NextFilter, IoSession, Object) line: 1015 + DefaultIoFilterChain.callNextMessageReceived(IoFilterChain$Entry, IoSession, Object) line: 650 + DefaultIoFilterChain.access$1300(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, Object) line: 49 + DefaultIoFilterChain$EntryImpl$1.messageReceived(IoSession, Object) line: 1128 + LoggingFilter.messageReceived(IoFilter$NextFilter, IoSession, Object) line: 208 + DefaultIoFilterChain.callNextMessageReceived(IoFilterChain$Entry, IoSession, Object) line: 650 + DefaultIoFilterChain.access$1300(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, Object) line: 49 + DefaultIoFilterChain$EntryImpl$1.messageReceived(IoSession, Object) line: 1128 + ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(IoFilter$NextFilter, IoSession) line: 413 + ProtocolCodecFilter.messageReceived(IoFilter$NextFilter, IoSession, Object) line: 257 + DefaultIoFilterChain.callNextMessageReceived(IoFilterChain$Entry, IoSession, Object) line: 650 + DefaultIoFilterChain.access$1300(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, Object) line: 49 + DefaultIoFilterChain$EntryImpl$1.messageReceived(IoSession, Object) line: 1128 + CompressionFilter.messageReceived(IoFilter$NextFilter, IoSession, Object) line: 169 + DefaultIoFilterChain.callNextMessageReceived(IoFilterChain$Entry, IoSession, Object) line: 650 + DefaultIoFilterChain.access$1300(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, Object) line: 49 + DefaultIoFilterChain$EntryImpl$1.messageReceived(IoSession, Object) line: 1128 + SslHandler.flushScheduledEvents() line: 335 + SslFilter.messageReceived(IoFilter$NextFilter, IoSession, Object) line: 553 + DefaultIoFilterChain.callNextMessageReceived(IoFilterChain$Entry, IoSession, Object) line: 650 + DefaultIoFilterChain.access$1300(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, Object) line: 49 + DefaultIoFilterChain$EntryImpl$1.messageReceived(IoSession, Object) line: 1128 + IoFilterEvent.fire() line: 105 + MdcInjectionFilter.filter(IoFilterEvent) line: 162 + MdcInjectionFilter(CommonEventFilter).messageReceived(IoFilter$NextFilter, IoSession, Object) line: 84 + DefaultIoFilterChain.callNextMessageReceived(IoFilterChain$Entry, IoSession, Object) line: 650 + DefaultIoFilterChain.access$1300(DefaultIoFilterChain, IoFilterChain$Entry, IoSession, Object) line: 49 + DefaultIoFilterChain$EntryImpl$1.messageReceived(IoSession, Object) line: 1128 + DefaultIoFilterChain$HeadFilter(IoFilterAdapter).messageReceived(IoFilter$NextFilter, IoSession, Object) line: 122 + DefaultIoFilterChain.callNextMessageReceived(IoFilterChain$Entry, IoSession, Object) line: 650 + DefaultIoFilterChain.fireMessageReceived(Object) line: 643 + NioProcessor(AbstractPollingIoProcessor<S>).read(S) line: 539 + AbstractPollingIoProcessor<S>.access$1200(AbstractPollingIoProcessor, AbstractIoSession) line: 68 + AbstractPollingIoProcessor$Processor.process(S) line: 1223 + AbstractPollingIoProcessor$Processor.process() line: 1212 + AbstractPollingIoProcessor$Processor.run() line: 683 + NamePreservingRunnable.run() line: 64 + ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1149 + ThreadPoolExecutor$Worker.run() line: 624 + Thread.run() line: 748 + +Writing the data is done when we are back in the AbstractPollingIoProcessor$Processor.run() call. Here is the stack for that action : + + ::: text + NioProcessor.write(NioSession, IoBuffer, int) line: 384 + NioProcessor.write(AbstractIoSession, IoBuffer, int) line: 47 + AbstractPollingIoProcessor$Processor.writeBuffer(S, WriteRequest, boolean, int, long) line: 1107 + AbstractPollingIoProcessor$Processor.flushNow(S, long) line: 994 + AbstractPollingIoProcessor$Processor.flush(long) line: 921 + AbstractPollingIoProcessor$Processor.run() line: 688 + NamePreservingRunnable.run() line: 64 + ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1149 + ThreadPoolExecutor$Worker.run() line: 624 + Thread.run() line: 748 + + +## The SslFilter implementation + +As we can see,the _SslFilter_ never calls the next filter. It delegates that to the _SslHandler_ class. The trick is that it uses a queue to store encrypted messages, which will wait until the message is fully encrypted before flushing them. The rational is that a message can be pretty big, and teh **SSL/TLS** protocol just allows limited encrypted data blocks (around 16k), so it keeps each block of encrypted data into the queue temporarily. + + +# Room for improvements + + +## Reducing the stack size + +This is clearly sub-optimal. First of all, the stack is crazy big, there is no reason for that. The idea was intially to allow a session to dynamically update the chain, by adding or removing a filter while a session is active. This is a fine idea, as some protocols need to do that: typically, LDAP allows a user to send a _startTls_ extended request which tells the server to establish a secured communication. In this very case, we add the _SslFilter_ in the session's filter chain. + +Now, we can do that more efficiently. A session's filter chain is created when the session itself is created. Adding a new filter in this chain is done by the application, and at this point, no other thread uses this session (each session is associated with one single _IoProcessor_ instance's thread), unless an _ExecutorFilter_ has been added to the chain. In this case, things get complicated... Having an _ExecutorFilter_ in the chain means any event can be processed concurrently, in different threads. Although they are sharing the same filter chain. We have to protect this chain against concurrent modifications, and we must also guarantee that any change made in the chain will not be an issue for the application. +Typically, adding a _SslFilter_ in a chain while a message is being written is problematic: +* we have to wait for any pending messages to be sent to the remote peer, which may have already initiated a handshake (although it's the client responsability to handle this situation), or we decide to dicth any of those messages (the right thing to do, IMHO) +* we have to deal with any message that are somewhere in the chain before the _SslFilter_. Again, the strategy would be to discard them +* and we have to deal with any message somwhere in the chain after the _SslFilter_ and before they are pushed into the session write queue, for the exact same reason. + +For messages received *before* the addition of the _SslFilter_, but still in the chain, we should ditch them for the exact same reason. + +Bottom line, this is a application issue, and should be handled by the application, because **MINA** can't deal with all the possible. starties in place of the application. + +That being said, we can use the _IoProcessor_ as a controller. It would b eits responsability to propagate events from one filter to the other, using such a loop: + + ::: text + for each filter in the chain + do + propagate the event to the filter with the previous result + store the result + done + +What we call the 'result' is just the transformed data. Typically, when calling the _ProtocolCodecFilter.messageReceived_ event, the result will be the decoded message, not the _IoBuffer_ (or whatever) input. This will be passed to the next filter (remember that the _IoFilter.messageReceived_ method takes an _Object_ as a parameter). \ No newline at end of file Added: mina/site/trunk/content/mina-project/technical-documentation/ssl-tls-internal.mdtext URL: http://svn.apache.org/viewvc/mina/site/trunk/content/mina-project/technical-documentation/ssl-tls-internal.mdtext?rev=1856869&view=auto ============================================================================== --- mina/site/trunk/content/mina-project/technical-documentation/ssl-tls-internal.mdtext (added) +++ mina/site/trunk/content/mina-project/technical-documentation/ssl-tls-internal.mdtext Wed Apr 3 10:33:31 2019 @@ -0,0 +1,66 @@ +Title: SSL/TLS internals +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +# SSL/TLS internals + +This is a technical description of how SSL/TLS are handled in MINA. You don't need to read this to have it working, better have a look at the [SslFilter user guide](../userguide/ch11-ss-filter/ch11-ssl-filter.html) page. However, if you want to get a deeper understanding on how it's built, this is the place ! + +## Components + +The **SslFilter** is the filter that command everything related to **SSL/TLS**. It works hands in hands with the **SSLHandler** class, which handles the logic. + +With **NIO**, Java provides a special class that handles the **SSL/TLS** protocol, the **SSLEngine** class. It's quite a complex piece of code, and it requires some time to tame it... We will have a look into it during this tutotial. + +## Creation + +In order to inject the filter in the chain, it must first be created. This filter takes one or two parameters: +* **SSLContext**: the Java class that contains all the information related to the **SSL/TLS** establishment +* __autoStart__: tells if the handshake should be started immediately. This is an optional parameter, defaulting to **TRUE**. In some cases, like when using __startTLS__, it's critical to not start the handshake immediately, as it will be established on demand. + +## Initialization + +When injecting the **SslFilter** into your chain, either before starting your service, or while running it, it has to be initialized in some way. Actually, this is a two phases process : +- pre-initialization, during the filter injection +- post-initialization, by starting the Handshake dialog + +Most of the time, both phases will be merged, as one can tell the first phase to immediately starts the second. + +In any case, it's handled by the __onPreAdd__ and __onPostAdd_ events, which means it's automatically processed when you push the **SslFilter** into the chain. + +### onPreAdd + +The **SslHandler** class is created and initialized, and the instance is stored into the session attributes. That means each session has its own instance of **SslHandler**. This initialization will create a **SSLEngine** instance based on the provided **SSLContext** instance. The initialization will differ based on the 'side' you are on: server or client. Basically, the server side will wait for the client to initiate the handshake, while the client side will intitiate it. + +It's also responsible to set the enabled ciphers and protocols, if one wants to use a restricted set, or an extended set (newer versions of Java have disabled old protocols and insecured ciphers). + +Last, not least, it sets a list of status flags : +* writingEncryptedData: false. This flag is used during the handshake +* handshakeStatus: the HandShake status, which is originally set to __NOT_HANDSHAKING__ +* firstSSLNegociation: true. This flag is used to tell the **sslHandler** to send or not an event to the application (MINA 2.1 only) +* handshakeComplete: false. It will be set to true when teh handshake has been completed. + +Side note: those flags are probably spurious. Some cleanup might be done to get rid of the useless ones. + + +## onPostAdd + +This event will initiate an immediate handshake if required. Depending on the perr side, the action will be different. + +- if we are on the server peer + + Modified: mina/site/trunk/content/mina-project/userguide/ch11-ssl-filter/ch11-ssl-filter.mdtext URL: http://svn.apache.org/viewvc/mina/site/trunk/content/mina-project/userguide/ch11-ssl-filter/ch11-ssl-filter.mdtext?rev=1856869&r1=1856868&r2=1856869&view=diff ============================================================================== --- mina/site/trunk/content/mina-project/userguide/ch11-ssl-filter/ch11-ssl-filter.mdtext (original) +++ mina/site/trunk/content/mina-project/userguide/ch11-ssl-filter/ch11-ssl-filter.mdtext Wed Apr 3 10:33:31 2019 @@ -61,7 +61,7 @@ You obviously need a **SslContext** inst This is up to you to provide the **KeyManager**, **TrustManager** and **SecureRandom** instances. -be sure to inject the **SslFilter** on the first position in your chain ! +Be sure to inject the **SslFilter** on the first position in your chain ! We will see later a detailed exemple on how to create a **SSLContext**. @@ -73,7 +73,7 @@ If you want to get a deeper understandin We are not going to explain how **SSL** works, there are very [good books](http://www.amazon.com/SSL-TLS-Designing-Building-Systems/dp/0201615983) outhere. We will just give a quick introduction on how it works and how it is implemented in MINA. -First of all, you have to understand that **SSL/TLS** is a protocol defined in **RFCs** : [TLS 1.0](https://www.rfc-editor.org/rfc/rfc2246.txt), [TLS 1.1](https://www.rfc-editor.org/rfc/rfc4346.txt) and [TLS 1.2](https://www.rfc-editor.org/rfc/rfc5246.txt). There is a [TLS 1.3](https://tlswg.github.io/tls13-spec/) draft being worked on... +First of all, you have to understand that **SSL/TLS** is a protocol defined in **RFCs** : [TLS 1.0](https://www.rfc-editor.org/rfc/rfc2246.txt), [TLS 1.1](https://www.rfc-editor.org/rfc/rfc4346.txt), [TLS 1.2](https://www.rfc-editor.org/rfc/rfc5246.txt) and [RLS 1.3](https://tools.ietf.org/html/rfc8446). It was initially developed by **Netscape**, and named **SSL** (from 1.0 to 3.0), before becoming **TLS**. Nowadays, ***SSL 2.0** and **SSL 3.0** have been deprecated and should not be used. @@ -90,9 +90,11 @@ It's enough to know that any secured exc As you can see in this picture, it's a 2 phases protocol : first the handshake, then when completed the client and the server will be able to exchange data that will be encrypted. +There are also other phases, like the **SSL/TLS** closure, or a renegociation phase. + #### The Handshake -Basically, it's all about negociating many elements that are to be used to encrypt the data. The details are not so interesting in the context of this document, enough said that many messages are going to be exchanged between the client and the server, and no data can be sent during this phase. +Basically, it's all about negociating many elements that are to be used to encrypt/decrypt the data. The details are not so interesting in the context of this document, enough said that many messages are going to be exchanged between the client and the server, and no data can be sent during this phase. Actually, there are two conditions for the handshake to start : * The server must be waiting for some handshake message to arrive