Hi,
more logs
2020-10-14 12:25:05,106 - ERROR
[nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler$CertificateVerifier@257]
- Unsuccessful handshake with session 0x0
2020-10-14 12:25:05,107 - WARN
[nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler@138] -
Exception caught
io.netty.handler.codec.DecoderException:
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
737276720a
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS
record: 737276720a
at
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
... 17 more
Thanks,
---------------------
Kuldeep Singh Budania
Software Architect
On Wed, Oct 14, 2020 at 3:51 PM kuldeep singh <kuldeep.sing...@gmail.com>
wrote:
> Hi,
>
>
> below error is coming on zookeeper logs on SSL.
>
> 2020-10-14 12:18:27,410 - WARN [nioEventLoopGroup-4-16:NettyServerCnxn@540]
> - Closing connection to /127.0.0.1:49470
> java.io.IOException: Len error 369296128
> at
> org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:533)
> at
> org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360)
> at
> org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:184)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
>
> Thanks,
> ---------------------
> Kuldeep Singh Budania
>
>
>
> On Thu, Oct 8, 2020 at 6:33 PM Szalay-Bekő Máté <
> szalay.beko.m...@gmail.com> wrote:
>
>> Sounds like a bug or a configuration issue...
>> can you share the configs (before and after the scale-up) and the logs?
>> also: does the truststores recognise all the keys used on all the 5
>> nodes? (e.g. the truststores on the old nodes accept the new keys?)
>>
>> Best Regards,
>> Mate
>>
>> On Thu, Oct 8, 2020 at 2:31 PM kuldeep singh <kuldeep.sing...@gmail.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > Yes, My client and server both are using certificate and have added in
>> ZK
>> > and client as well.
>> >
>> > Thanks,
>> > ---------------------
>> > Kuldeep Singh Budania
>> >
>> >
>> >
>> > On Thu, Oct 8, 2020 at 5:56 PM Enrico Olivelli <eolive...@gmail.com>
>> wrote:
>> >
>> > > Il giorno gio 8 ott 2020 alle ore 14:17 kuldeep singh <
>> > > kuldeep.sing...@gmail.com> ha scritto:
>> > >
>> > > > Hi Team,
>> > > >
>> > > > I am facing one issue in SSL communication between client and
>> zookeeper
>> > > > server.
>> > > >
>> > > > ZK 3.5.6 version
>> > > >
>> > > > 1. Mi on 3 node
>> > > > 2. Applying SSL and 3 nodes cluster is working fine
>> > > > 3. Scaled my cluster with 2 nodes and now my cluster have 5 nodes
>> over
>> > > SSL
>> > > >
>> > > > but after scaling my SSL is not working between client and ZK
>> server and
>> > > > even not able to login using zkCli as well.
>> > > >
>> > > > Can someone provide the details please why it is happening?
>> > > >
>> > >
>> > > Is your client configured to use SSL ?
>> > >
>> > > Enrico
>> > >
>> > >
>> > >
>> > > >
>> > > > Thanks,
>> > > > ---------------------
>> > > > Kuldeep Singh Budania
>> > > > Software Architect
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Jul 13, 2020 at 2:19 PM Enrico Olivelli - Diennea
>> > > > <enrico.olive...@diennea.com.invalid> wrote:
>> > > >
>> > > > > It looks like we ported it to 3.5.
>> > > > >
>> > > > > See the subtask
>> > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2792
>> > > > >
>> > > > > Enrico
>> > > > >
>> > > > > Il giorno 13/07/20, 10:37 "kuldeep singh" <
>> kuldeep.sing...@gmail.com>
>> > > > ha
>> > > > > scritto:
>> > > > >
>> > > > > Hi Team,
>> > > > >
>> > > > > I appreciate it if I will get a response as soon as possible,
>> as I
>> > > am
>> > > > > stuck
>> > > > > at this point.
>> > > > >
>> > > > > Thanks,
>> > > > > ---------------------
>> > > > > Kuldeep Singh Budania
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Jul 13, 2020 at 11:10 AM kuldeep singh <
>> > > > > kuldeep.sing...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Team,
>> > > > > >
>> > > > > > Server to Server communication is not supported in 3.5.6
>> version
>> > > > as
>> > > > > per
>> > > > > > below JIRA issue?
>> > > > > >
>> > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2639
>> > > > > >
>> > > > > > Thanks,
>> > > > > > ---------------------
>> > > > > > Kuldeep Singh Budania
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Jul 2, 2020 at 4:24 PM kuldeep singh <
>> > > > > kuldeep.sing...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > >> Thanks for the reply.
>> > > > > >>
>> > > > > >> Now my ZKCli cmd is working fine as we use some our
>> customized
>> > > > > >> authentication and we resolve the issue.
>> > > > > >>
>> > > > > >> Now I am going to implement Server to Server communication.
>> > > > > >>
>> > > > > >> Thanks,
>> > > > > >> ---------------------
>> > > > > >> Kuldeep Singh Budania
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >> On Thu, Jul 2, 2020 at 3:53 PM Szalay-Bekő Máté <
>> > > > > >> szalay.beko.m...@gmail.com> wrote:
>> > > > > >>
>> > > > > >>> I think SSL is working for you already... If you managed
>> to
>> > > start
>> > > > > the
>> > > > > >>> zkCli.sh and be able to connect to ZooKeeper on the
>> secure port
>> > > > > and issue
>> > > > > >>> any kind of command (like: " ls / "), then the wire
>> encryption
>> > > is
>> > > > > working
>> > > > > >>> and your server/client communication is secured by
>> ZooKeeper.
>> > > > > >>>
>> > > > > >>> Why you want to run the following command?
>> > > > > >>> addauth ztpasswd zooadmin:<password>
>> > > > > >>>
>> > > > > >>> Do you also want to configure a superDigest user in
>> ZooKeeper?
>> > > > > Please
>> > > > > >>> note
>> > > > > >>> that this command is independent from SSL. If you need to
>> > > create
>> > > > a
>> > > > > >>> username-password pair for digest authentication then
>> please
>> > > use
>> > > > > the
>> > > > > >>> command in the following way:
>> > > > > >>> addauth digest zooadmin:yourSuperSecretPassword
>> > > > > >>>
>> > > > > >>> Kind regards,
>> > > > > >>> Mate
>> > > > > >>>
>> > > > > >>> On Thu, Jul 2, 2020 at 6:59 AM kuldeep singh <
>> > > > > kuldeep.sing...@gmail.com>
>> > > > > >>> wrote:
>> > > > > >>>
>> > > > > >>> > 1. sh zkCli.sh --config /etc/zookeeper -server
>> localhost:2281
>> > > > > >>> >
>> > > > > >>> > 2. addauth ztpasswd zooadmin:<password>
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> > Thanks,
>> > > > > >>> > ---------------------
>> > > > > >>> > Kuldeep Singh Budania
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> > On Thu, Jul 2, 2020 at 9:56 AM kuldeep singh <
>> > > > > >>> kuldeep.sing...@gmail.com>
>> > > > > >>> > wrote:
>> > > > > >>> >
>> > > > > >>> > > Hi Team,
>> > > > > >>> > >
>> > > > > >>> > > Any update on this?
>> > > > > >>> > >
>> > > > > >>> > > Thanks,
>> > > > > >>> > > ---------------------
>> > > > > >>> > > Kuldeep Singh Budania
>> > > > > >>> > >
>> > > > > >>> > >
>> > > > > >>> > >
>> > > > > >>> > > On Wed, Jul 1, 2020 at 6:43 PM kuldeep singh <
>> > > > > >>> kuldeep.sing...@gmail.com>
>> > > > > >>> > > wrote:
>> > > > > >>> > >
>> > > > > >>> > >> Sorry this is my bad, there were server setting like
>> below
>> > > > > >>> > >>
>> > > > > >>> > >> export SERVER_JVMFLAGS="
>> > > > > >>> > >>
>> > > > > >>> > >>
>> > > > > >>> > >>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>> > > > > >>> > >>
>> > > > > >>> > >>
>> > > > > >>>
>> > > > >
>> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks
>> > > > > >>> > >> -Dzookeeper.ssl.keyStore.password=testpass
>> > > > > >>> > >>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks
>> > > > > >>> > >> -Dzookeeper.ssl.trustStore.password=testpass"
>> > > > > >>> > >>
>> > > > > >>> > >>
>> > > > > >>> > >>
>> > > > > >>> > >> export CLIENT_JVMFLAGS="
>> > > > > >>> > >>
>> > > > > >>> > >>
>> > > > > >>>
>> > > > >
>> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
>> > > > > >>> > >> -Dzookeeper.client.secure=true
>> > > > > >>> > >>
>> > > > > >>>
>> > > > >
>> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks
>> > > > > >>> > >> -Dzookeeper.ssl.keyStore.password=testpass
>> > > > > >>> > >>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks
>> > > > > >>> > >> -Dzookeeper.ssl.trustStore.password=testpass"
>> > > > > >>> > >>
>> > > > > >>> > >> I want to have SSL between client to server
>> communication
>> > > > > >>> > >>
>> > > > > >>> > >> I am already following the same link which you have
>> shared
>> > > > > with me
>> > > > > >>> but
>> > > > > >>> > >> that is not working.
>> > > > > >>> > >>
>> > > > > >>> > >> Zoo.cfg
>> > > > > >>> > >>
>> > > > > >>> > >> secureClientPort=2281
>> > > > > >>> > >> initLimit=10
>> > > > > >>> > >> syncLimit=5
>> > > > > >>> > >> dataDir=/var/lib/zookeeper/data
>> > > > > >>> > >> tickTime=2000
>> > > > > >>> > >> autopurge.snapRetainCount=3
>> > > > > >>> > >> autopurge.purgeInterval=1
>> > > > > >>> > >> admin.enableServer=false
>> > > > > >>> > >> standaloneEnabled=false
>> > > > > >>> > >> jute.maxbuffer=2147483648
>> > > > > >>> > >>
>> > > > >
>> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>> > > > > >>> > >> server.1=host1_priv:10288:10388
>> > > > > >>> > >> server.2=host2_priv:10288:10388
>> > > > > >>> > >> server.3=host3_priv:10288:10388
>> > > > > >>> > >>
>> > > > > >>> > >>
>> > > > > >>> > >> command to connect using zkcli
>> > > > > >>> > >>
>> > > > > >>> > >> 1. zkcli zoo.cfg localhost:2281
>> > > > > >>> > >> 2. addauth ztpasswd usernaem:password
>> > > > > >>> > >>
>> > > > > >>> > >> after second step we are getting below error
>> > > > > >>> > >>
>> > > > > >>> > >> WatchedEvent state:AuthFailed type:None path:null
>> > > > > >>> > >>
>> > > > > >>> > >>
>> > > > > >>> > >> Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN
>> > > > > >>> > >> [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No
>> > > > > authentication
>> > > > > >>> > provider
>> > > > > >>> > >> for scheme: ztpasswd has x509 ip digest*
>> > > > > >>> > >>
>> > > > > >>> > >> Thanks,
>> > > > > >>> > >> ---------------------
>> > > > > >>> > >> Kuldeep Singh Budania
>> > > > > >>> > >>
>> > > > > >>> > >>
>> > > > > >>> > >>
>> > > > > >>> > >> On Wed, Jul 1, 2020 at 6:25 PM Szalay-Bekő Máté <
>> > > > > >>> > >> szalay.beko.m...@gmail.com> wrote:
>> > > > > >>> > >>
>> > > > > >>> > >>> > No authentication provider for scheme: ztpasswd
>> has
>> > > x509
>> > > > > ip
>> > > > > >>> digest*
>> > > > > >>> > >>>
>> > > > > >>> > >>> This suggest you have some configuration error...
>> Where
>> > > did
>> > > > > you
>> > > > > >>> use the
>> > > > > >>> > >>> "ztpasswd" string in your configs / commands?
>> > > > > >>> > >>>
>> > > > > >>> > >>> On Wed, Jul 1, 2020 at 2:53 PM Szalay-Bekő Máté <
>> > > > > >>> > >>> szalay.beko.m...@gmail.com>
>> > > > > >>> > >>> wrote:
>> > > > > >>> > >>>
>> > > > > >>> > >>> > > My ZK server is up and running in secure mode
>> > > > > >>> > >>> >
>> > > > > >>> > >>> > What is your goal? You want to setup
>> client-server SSL
>> > > > > >>> connection?
>> > > > > >>> > >>> >
>> > > > > >>> > >>> > see:
>> > > > > >>> > >>> >
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide
>> > > > > >>> > >>> >
>> > > > > >>> > >>> > (or you want to have both SSL and SASL enabled?)
>> > > > > >>> > >>> >
>> > > > > >>> > >>> > Anyway, please remove the following line from the
>> > > > > >>> SERVER_JVMFLAGS:
>> > > > > >>> > >>> >
>> > > > > >>> >
>> > > > >
>> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
>> > > > > >>> > >>> > This is a configuration that makes sense only for
>> the
>> > > > > ZooKeeper
>> > > > > >>> > client,
>> > > > > >>> > >>> > not for the server. For the server, use the
>> following:
>> > > > > >>> > >>> >
>> > > > > >>> > >>> >
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>> > > > > >>> > >>> >
>> > > > > >>> > >>> > If it doesn't solve the issue, then can you
>> please send
>> > > > > your
>> > > > > >>> zoo.cfg
>> > > > > >>> > >>> file?
>> > > > > >>> > >>> > Also can you please send the zkCli command you
>> execute?
>> > > > > (you
>> > > > > >>> need to
>> > > > > >>> > >>> > connect to the secure ZooKeeper port, unless
>> > > > > portUnification is
>> > > > > >>> > >>> enabled)
>> > > > > >>> > >>> >
>> > > > > >>> > >>> > Kind regards,
>> > > > > >>> > >>> > Mate
>> > > > > >>> > >>> >
>> > > > > >>> > >>> > On Wed, Jul 1, 2020 at 9:48 AM kuldeep singh <
>> > > > > >>> > >>> kuldeep.sing...@gmail.com>
>> > > > > >>> > >>> > wrote:
>> > > > > >>> > >>> >
>> > > > > >>> > >>> >> Hi,
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> we have done below changes in java.env file
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> export SERVER_JVMFLAGS="
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
>> > > > > >>> > >>> >>
>> > > > > >>> >
>> > > > >
>> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks
>> > > > > >>> > >>> >> -Dzookeeper.ssl.keyStore.password=testpass
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks
>> > > > > >>> > >>> >> -Dzookeeper.ssl.trustStore.password=testpass"
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> export CLIENT_JVMFLAGS="
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
>> > > > > >>> > >>> >> -Dzookeeper.client.secure=true
>> > > > > >>> > >>> >>
>> > > > > >>> >
>> > > > >
>> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks
>> > > > > >>> > >>> >> -Dzookeeper.ssl.keyStore.password=testpass
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks
>> > > > > >>> > >>> >> -Dzookeeper.ssl.trustStore.password=testpass"
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> I have started the ZK server and it is up
>> without any
>> > > > > issue.
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> But now when I login to ZkCli then it gives the
>> below
>> > > > > error.
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> WatchedEvent state:AuthFailed type:None path:null
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN
>> > > > > >>> > >>> >> [nioEventLoopGroup-4-2:ZooKeeperServer@1119] -
>> No
>> > > > > >>> authentication
>> > > > > >>> > >>> provider
>> > > > > >>> > >>> >> for scheme: ztpasswd has x509 ip digest*
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> Please help me on this issue
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> Thanks,
>> > > > > >>> > >>> >> ---------------------
>> > > > > >>> > >>> >> Kuldeep Singh Budania
>> > > > > >>> > >>> >> Software Architect
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> On Wed, Jul 1, 2020 at 12:05 PM kuldeep singh <
>> > > > > >>> > >>> kuldeep.sing...@gmail.com>
>> > > > > >>> > >>> >> wrote:
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >> > Hi,
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> > My ZK server is up and running in secure
>> mode, But
>> > > > > When I am
>> > > > > >>> > >>> trying to
>> > > > > >>> > >>> >> > connect to the ZK server using ZKCli, it gives
>> the
>> > > > below
>> > > > > >>> error.
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> > WatchedEvent state:AuthFailed type:None
>> path:null
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> > Zookeeper logs :- *2020-07-01 07:38:09,342 -
>> WARN
>> > > > > >>> > >>> >> > [nioEventLoopGroup-4-2:ZooKeeperServer@1119]
>> - No
>> > > > > >>> authentication
>> > > > > >>> > >>> >> provider
>> > > > > >>> > >>> >> > for scheme: ztpasswd has x509 ip digest*
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> > Can someone please help me on this issue. we
>> are
>> > > using
>> > > > > the
>> > > > > >>> 3.5.6
>> > > > > >>> > >>> >> version.
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> > I appreciate if I will get a response as soon
>> as
>> > > > > possible, as
>> > > > > >>> I am
>> > > > > >>> > >>> stuck
>> > > > > >>> > >>> >> > at this point.
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> > Thanks,
>> > > > > >>> > >>> >> > ---------------------
>> > > > > >>> > >>> >> > Kuldeep Singh Budania
>> > > > > >>> > >>> >> > Software Architect
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> > On Thu, Jun 25, 2020 at 11:54 AM Enrico
>> Olivelli -
>> > > > > Diennea
>> > > > > >>> > >>> >> > <enrico.olive...@diennea.com.invalid> wrote:
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >> >> I mean in zoo.cfg
>> > > > > >>> > >>> >> >> Not as a system property
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> Enrico
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> Il giorno 25/06/20, 08:19 "Enrico Olivelli -
>> > > > Diennea" <
>> > > > > >>> > >>> >> >> enrico.olive...@diennea.com.INVALID> ha
>> scritto:
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> Hi
>> > > > > >>> > >>> >> >> You have to enable Netty on the server
>> side
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> Something like:
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> >
>> > > > >
>> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> Hope that helps
>> > > > > >>> > >>> >> >> Enrico
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> Il giorno 24/06/20, 19:17 "kuldeep singh"
>> <
>> > > > > >>> > >>> >> kuldeep.sing...@gmail.com>
>> > > > > >>> > >>> >> >> ha scritto:
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> Hi,
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> I got below error while setting SSL
>> > > > properties
>> > > > > in
>> > > > > >>> > zkEnv.sh
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> ==========================================================================================
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,864 - INFO
>> > > > > >>> > [main:QuorumPeerConfig@133]
>> > > > > >>> > >>> -
>> > > > > >>> > >>> >> >> Reading
>> > > > > >>> > >>> >> >> configuration from:
>> /etc/zookeeper/zoo.cfg
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,874 - INFO
>> > > > > >>> > [main:QuorumPeerConfig@385]
>> > > > > >>> > >>> -
>> > > > > >>> > >>> >> >> clientPortAddress is
>> 0.0.0.0/0.0.0.0:10181
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,874 - INFO
>> > > > > >>> > [main:QuorumPeerConfig@399]
>> > > > > >>> > >>> -
>> > > > > >>> > >>> >> >> secureClientPortAddress is
>> > > > > 0.0.0.0/0.0.0.0:2281
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,878 - INFO
>> > > > > [main:X509Util@79] -
>> > > > > >>> > >>> Setting
>> > > > > >>> > >>> >> -D
>> > > > > >>> > >>> >> >>
>> > > > > jdk.tls.rejectClientInitiatedRenegotiation=true to
>> > > > > >>> > disable
>> > > > > >>> > >>> >> >> client-initiated
>> > > > > >>> > >>> >> >> TLS renegotiation
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,897 - INFO
>> > > > > >>> > >>> [main:DatadirCleanupManager@78]
>> > > > > >>> > >>> >> -
>> > > > > >>> > >>> >> >> autopurge.snapRetainCount set to 3
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,897 - INFO
>> > > > > >>> > >>> [main:DatadirCleanupManager@79]
>> > > > > >>> > >>> >> -
>> > > > > >>> > >>> >> >> autopurge.purgeInterval set to 1
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,898 - INFO [
>> > > > > >>> > >>> >> >>
>> > > PurgeTask:DatadirCleanupManager$PurgeTask@138
>> > > > ]
>> > > > > -
>> > > > > >>> Purge
>> > > > > >>> > >>> task
>> > > > > >>> > >>> >> >> started.
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,899 - INFO
>> > > > > [main:ManagedUtil@46]
>> > > > > >>> -
>> > > > > >>> > >>> Log4j
>> > > > > >>> > >>> >> >> found with
>> > > > > >>> > >>> >> >> jmx enabled.
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,903 - INFO
>> > > > > >>> > >>> [PurgeTask:FileTxnSnapLog@103]
>> > > > > >>> > >>> >> -
>> > > > > >>> > >>> >> >> zookeeper.snapshot.trust.empty : false
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,910 - INFO [
>> > > > > >>> > >>> >> >>
>> > > PurgeTask:DatadirCleanupManager$PurgeTask@144
>> > > > ]
>> > > > > -
>> > > > > >>> Purge
>> > > > > >>> > >>> task
>> > > > > >>> > >>> >> >> completed.
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,975 - INFO
>> > > > > >>> [main:QuorumPeerMain@141
>> > > > > >>> > ]
>> > > > > >>> > >>> -
>> > > > > >>> > >>> >> >> Starting quorum
>> > > > > >>> > >>> >> >> peer
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,983 - INFO
>> > > > > >>> > [main:ServerCnxnFactory@135]
>> > > > > >>> > >>> -
>> > > > > >>> > >>> >> >> Using
>> > > > > >>> > >>> >> >>
>> > > > > org.apache.zookeeper.server.NIOServerCnxnFactory as
>> > > > > >>> > server
>> > > > > >>> > >>> >> >> connection
>> > > > > >>> > >>> >> >> factory
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,986 - INFO
>> > > > > >>> > >>> [main:NIOServerCnxnFactory@673]
>> > > > > >>> > >>> >> -
>> > > > > >>> > >>> >> >> Configuring NIO connection handler
>> with 10s
>> > > > > >>> sessionless
>> > > > > >>> > >>> >> >> connection timeout,
>> > > > > >>> > >>> >> >> 2 selector thread(s), 16 worker
>> threads,
>> > > and
>> > > > > 64 kB
>> > > > > >>> direct
>> > > > > >>> > >>> >> buffers.
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,992 - INFO
>> > > > > >>> > >>> [main:NIOServerCnxnFactory@686
>> > > > > >>> > >>> >> ]
>> > > > > >>> > >>> >> >> - binding
>> > > > > >>> > >>> >> >> to port 0.0.0.0/0.0.0.0:10181
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,994 - INFO
>> > > > > >>> > [main:ServerCnxnFactory@135]
>> > > > > >>> > >>> -
>> > > > > >>> > >>> >> >> Using
>> > > > > >>> > >>> >> >>
>> > > > > org.apache.zookeeper.server.NIOServerCnxnFactory as
>> > > > > >>> > server
>> > > > > >>> > >>> >> >> connection
>> > > > > >>> > >>> >> >> factory
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> 2020-06-24 15:49:35,995 - ERROR
>> > > > > >>> [main:QuorumPeerMain@101
>> > > > > >>> > ]
>> > > > > >>> > >>> -
>> > > > > >>> > >>> >> >> Unexpected
>> > > > > >>> > >>> >> >> exception, exiting abnormally
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> java.lang.UnsupportedOperationException:
>> > > SSL
>> > > > > isn't
>> > > > > >>> > >>> supported in
>> > > > > >>> > >>> >> >> NIOServerCnxn
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> at
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:644)
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> at
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:155)
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> at
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> at
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> I have set the following properties in
>> > > > > >>> SERVER_JVMFLAGS in
>> > > > > >>> > >>> >> >> zkEnv.sh file :
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> "-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> -Dzookeeper.ssl.keyStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkkeystore.p12
>> > > > > >>> > >>> >> >>
>> > > > > -Dzookeeper.ssl.keyStore.password=EvaiKiO1@123456
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> -Dzookeeper.ssl.trustStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkTrustStore.jks
>> > > > > >>> > >>> >> >>
>> > > > > -Dzookeeper.ssl.trustStore.password=EvaiKiO1@123456"
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> Thanks,
>> > > > > >>> > >>> >> >> ---------------------
>> > > > > >>> > >>> >> >> Kuldeep Singh Budania
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> On Mon, Jun 22, 2020 at 8:08 PM Jordan
>> > > > > Zimmerman <
>> > > > > >>> > >>> >> >> jor...@jordanzimmerman.com>
>> > > > > >>> > >>> >> >> wrote:
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> > It's the same as the normal
>> ZooKeeper
>> > > > client:
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions
>> > > > > >>> > >>> >> >> > <
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions
>> > > > > >>> > >>> >> >> > >
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >> > -Jordan
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >> > > On Jun 22, 2020, at 5:50 AM,
>> kuldeep
>> > > > singh
>> > > > > <
>> > > > > >>> > >>> >> >> kuldeep.sing...@gmail.com>
>> > > > > >>> > >>> >> >> > wrote:
>> > > > > >>> > >>> >> >> > >
>> > > > > >>> > >>> >> >> > > Hi Team,
>> > > > > >>> > >>> >> >> > > How we will do secure
>> communication
>> > > > > between the
>> > > > > >>> > Curator
>> > > > > >>> > >>> >> >> framework and
>> > > > > >>> > >>> >> >> > > zookeeper 3.5.6 ?
>> > > > > >>> > >>> >> >> > > I didn't get any solution right
>> now.
>> > > > > >>> > >>> >> >> > > I appreciate it if someone could
>> help
>> > > me
>> > > > > with the
>> > > > > >>> > same.
>> > > > > >>> > >>> >> >> > >
>> > > > > >>> > >>> >> >> > > Thanks,
>> > > > > >>> > >>> >> >> > > ---------------------
>> > > > > >>> > >>> >> >> > > Kuldeep Singh Budania
>> > > > > >>> > >>> >> >> > > Software Architect
>> > > > > >>> > >>> >> >> > >
>> > > > > >>> > >>> >> >> > >
>> > > > > >>> > >>> >> >> > > On Fri, Apr 17, 2020 at 4:53 PM
>> > > > > Szalay-Bekő Máté
>> > > > > >>> <
>> > > > > >>> > >>> >> >> > szalay.beko.m...@gmail.com>
>> > > > > >>> > >>> >> >> > > wrote:
>> > > > > >>> > >>> >> >> > >
>> > > > > >>> > >>> >> >> > >> Hello Kuldeep,
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> > >> did you download the source from
>> the
>> > > > > ZooKeeper
>> > > > > >>> > >>> webpage, or
>> > > > > >>> > >>> >> >> checked out
>> > > > > >>> > >>> >> >> > from
>> > > > > >>> > >>> >> >> > >> git?
>> > > > > >>> > >>> >> >> > >> Anyway, the following commands
>> should
>> > > > > work:
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> > >> wget
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> https://downloads.apache.org/zookeeper/zookeeper-3.5.7/apache-zookeeper-3.5.7.tar.gz
>> > > > > >>> > >>> >> >> > >> tar xzvf
>> > > ./apache-zookeeper-3.5.7.tar.gz
>> > > > > >>> > >>> >> >> > >> cd apache-zookeeper-3.5.7
>> > > > > >>> > >>> >> >> > >> mvn clean install -DskipTests
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> > >> I tested it with OpenJDK 8u424
>> and
>> > > maven
>> > > > > 3.6.0.
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> > >> Kind regards,
>> > > > > >>> > >>> >> >> > >> Mate
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> > >> On Fri, Apr 17, 2020 at 12:51 PM
>> > > kuldeep
>> > > > > singh <
>> > > > > >>> > >>> >> >> > kuldeep.sing...@gmail.com>
>> > > > > >>> > >>> >> >> > >> wrote:
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> > >>> Hi All,
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>> I have download the zookeeper
>> source
>> > > > > code and
>> > > > > >>> run
>> > > > > >>> > >>> the mvn
>> > > > > >>> > >>> >> >> command for
>> > > > > >>> > >>> >> >> > >>> packaging but get problem on
>> > > > > "zookeeper-server"
>> > > > > >>> > >>> >> directory.
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>> *command :- mvn clean package*
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>> [ERROR] Failed to execute goal
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >>
>> > > > > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
>> > > > > >>> > >>> >> >> > >>> (default-compile) on project
>> > > zookeeper:
>> > > > > Fatal
>> > > > > >>> error
>> > > > > >>> > >>> >> >> compiling:
>> > > > > >>> > >>> >> >> > >>> java.lang.NullPointerException
>> ->
>> > > [Help
>> > > > > 1]
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>>
>> org.apache.maven.lifecycle.LifecycleExecutionException:
>> > > > > >>> > >>> >> >> Failed to
>> > > > > >>> > >>> >> >> > execute
>> > > > > >>> > >>> >> >> > >>> goal
>> > > > > >>> > >>> >> >>
>> > > > > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
>> > > > > >>> > >>> >> >> > >>> (default-compile) on project
>> > > zookeeper:
>> > > > > Fatal
>> > > > > >>> error
>> > > > > >>> > >>> >> >> compiling
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >>
>> > > > > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > > > org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > > > org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > > org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > > > > Method)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> java.lang.reflect.Method.invoke(Method.java:498)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > >
>> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
>> > > > > >>> > >>> >> >> > >>> Caused by:
>> > > > > >>> > >>> >> org.apache.maven.plugin.MojoExecutionException:
>> > > > > >>> > >>> >> >> Fatal error
>> > > > > >>> > >>> >> >> > >>> compiling
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:1140)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:188)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>> > > > > >>> > >>> >> >> > >>> ... 19 more
>> > > > > >>> > >>> >> >> > >>> Caused by:
>> > > > > >>> > >>> >> org.codehaus.plexus.compiler.CompilerException:
>> > > > > >>> > >>> >> >> > >>> java.lang.NullPointerException
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:173)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile(JavacCompiler.java:174)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:1129)
>> > > > > >>> > >>> >> >> > >>> ... 22 more
>> > > > > >>> > >>> >> >> > >>> Caused by:
>> > > java.lang.RuntimeException:
>> > > > > >>> > >>> >> >> java.lang.NullPointerException
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > com.sun.tools.javac.main.Main.compile(Main.java:559)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > > com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:126)
>> > > > > >>> > >>> >> >> > >>> ... 24 more
>> > > > > >>> > >>> >> >> > >>> Caused by:
>> > > > java.lang.NullPointerException
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> com.sun.tools.javac.model.JavacTypes.getOverriddenMethods(JavacTypes.java:306)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >>
>> com.sun.tools.doclint.Env.setCurrent(Env.java:155)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > com.sun.tools.doclint.Checker.scan(Checker.java:144)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > > > com.sun.tools.doclint.DocLint$2.visitDecl(DocLint.java:285)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:359)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:346)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > >
>> com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:720)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>>
>> > > com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > > com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:81)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > >
>> com.sun.source.util.TreeScanner.visitNewClass(TreeScanner.java:280)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > >
>> com.sun.tools.javac.tree.JCTree$JCNewClass.accept(JCTree.java:1532)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>>
>> > > com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > > com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:81)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > >
>> com.sun.source.util.TreeScanner.visitVariable(TreeScanner.java:153)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitVariable(DocLint.java:373)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitVariable(DocLint.java:346)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> com.sun.tools.javac.tree.JCTree$JCVariableDecl.accept(JCTree.java:864)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>>
>> > > com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > > com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:81)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > > > com.sun.source.util.TreeScanner.scan(TreeScanner.java:91)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > > com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:99)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >>
>> > > > > >>>
>> > > com.sun.source.util.TreeScanner.visitClass(TreeScanner.java:133)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:360)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:346)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > >
>> com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:720)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>>
>> > > com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > > > com.sun.source.util.TreeScanner.scan(TreeScanner.java:91)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > > com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:99)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> com.sun.source.util.TreeScanner.visitCompilationUnit(TreeScanner.java:120)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitCompilationUnit(DocLint.java:354)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitCompilationUnit(DocLint.java:346)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > >
>> com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:550)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>>
>> > > com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > > > com.sun.tools.doclint.DocLint$3.started(DocLint.java:296)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> com.sun.tools.javac.api.ClientCodeWrapper$WrappedTaskListener.started(ClientCodeWrapper.java:668)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> com.sun.tools.javac.api.MultiTaskListener.started(MultiTaskListener.java:103)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> com.sun.tools.javac.main.JavaCompiler.attribute(JavaCompiler.java:1240)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> com.sun.tools.javac.main.JavaCompiler.compile2(JavaCompiler.java:901)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:860)
>> > > > > >>> > >>> >> >> > >>> at
>> > > > > >>> > >>> >> >>
>> > > com.sun.tools.javac.main.Main.compile(Main.java:523)
>> > > > > >>> > >>> >> >> > >>> ... 27 more
>> > > > > >>> > >>> >> >> > >>> [ERROR]
>> > > > > >>> > >>> >> >> > >>> [ERROR] Re-run Maven using the
>> -X
>> > > > switch
>> > > > > to
>> > > > > >>> enable
>> > > > > >>> > >>> full
>> > > > > >>> > >>> >> >> debug logging.
>> > > > > >>> > >>> >> >> > >>> [ERROR]
>> > > > > >>> > >>> >> >> > >>> [ERROR] For more information
>> about
>> > > the
>> > > > > errors
>> > > > > >>> and
>> > > > > >>> > >>> >> possible
>> > > > > >>> > >>> >> >> solutions,
>> > > > > >>> > >>> >> >> > >>> please read the following
>> articles:
>> > > > > >>> > >>> >> >> > >>> [ERROR] [Help 1]
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > >
>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>> Can someone help me on this
>> issue?
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>> Thanks,
>> > > > > >>> > >>> >> >> > >>> ---------------------
>> > > > > >>> > >>> >> >> > >>> Kuldeep Singh Budania
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>> On Sat, Apr 4, 2020 at 5:57 PM
>> > > > > Szalay-Bekő
>> > > > > >>> Máté <
>> > > > > >>> > >>> >> >> > >>> szalay.beko.m...@gmail.com>
>> > > > > >>> > >>> >> >> > >>> wrote:
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>>> these exceptions can mean many
>> > > > > things... I
>> > > > > >>> think
>> > > > > >>> > >>> this
>> > > > > >>> > >>> >> can
>> > > > > >>> > >>> >> >> be even
>> > > > > >>> > >>> >> >> > >> normal
>> > > > > >>> > >>> >> >> > >>>> duding rolling restart (as some
>> > > > > connections
>> > > > > >>> get
>> > > > > >>> > >>> broken
>> > > > > >>> > >>> >> in
>> > > > > >>> > >>> >> >> this case)
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>> However, I saw cases already
>> when
>> > > > > exceptions
>> > > > > >>> like
>> > > > > >>> > >>> these
>> > > > > >>> > >>> >> >> killed
>> > > > > >>> > >>> >> >> > receiver
>> > > > > >>> > >>> >> >> > >>> or
>> > > > > >>> > >>> >> >> > >>>> sender threads in
>> QuorumCnxManager /
>> > > > > Leader
>> > > > > >>> > >>> Election in
>> > > > > >>> > >>> >> >> such a way
>> > > > > >>> > >>> >> >> > that
>> > > > > >>> > >>> >> >> > >>>> they were not able to recover,
>> so
>> > > the
>> > > > > node was
>> > > > > >>> > >>> unable to
>> > > > > >>> > >>> >> >> connect to
>> > > > > >>> > >>> >> >> > any
>> > > > > >>> > >>> >> >> > >>>> quorum until restart. I
>> remember
>> > > > seeing
>> > > > > this
>> > > > > >>> in
>> > > > > >>> > 3.4
>> > > > > >>> > >>> too.
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>> Do you see these exceptions in
>> the
>> > > > > second
>> > > > > >>> server
>> > > > > >>> > >>> (the
>> > > > > >>> > >>> >> one
>> > > > > >>> > >>> >> >> which you
>> > > > > >>> > >>> >> >> > >> just
>> > > > > >>> > >>> >> >> > >>>> upgraded in step 3)?
>> > > > > >>> > >>> >> >> > >>>> Is this issue reproducible?
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>> What is the tickTime and
>> initLimit
>> > > you
>> > > > > use?
>> > > > > >>> Maybe
>> > > > > >>> > >>> the
>> > > > > >>> > >>> >> >> server just
>> > > > > >>> > >>> >> >> > >> require
>> > > > > >>> > >>> >> >> > >>>> more time to sync?
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>> I would need more logs to
>> really see
>> > > > > what
>> > > > > >>> > happened.
>> > > > > >>> > >>> Can
>> > > > > >>> > >>> >> >> you create a
>> > > > > >>> > >>> >> >> > >> Jira
>> > > > > >>> > >>> >> >> > >>>> issue and upload the logs and
>> also
>> > > the
>> > > > > >>> ZooKeeper
>> > > > > >>> > >>> >> configs?
>> > > > > >>> > >>> >> >> I am happy
>> > > > > >>> > >>> >> >> > to
>> > > > > >>> > >>> >> >> > >>>> take a closer look.
>> > > > > >>> > >>> >> >> > >>>> (if you need to re-run the
>> test to
>> > > > > collect the
>> > > > > >>> > logs,
>> > > > > >>> > >>> >> then
>> > > > > >>> > >>> >> >> enabling
>> > > > > >>> > >>> >> >> > >> DEBUG
>> > > > > >>> > >>> >> >> > >>>> logs would be great. The INFO
>> level
>> > > > > logs are
>> > > > > >>> > usually
>> > > > > >>> > >>> >> >> enough for these
>> > > > > >>> > >>> >> >> > >>>> problems, but one can never
>> know...)
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>> Kind regards,
>> > > > > >>> > >>> >> >> > >>>> Mate
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>> On Fri, Apr 3, 2020 at 10:05 AM
>> > > > kuldeep
>> > > > > singh
>> > > > > >>> <
>> > > > > >>> > >>> >> >> > >> kuldeep.sing...@gmail.com
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>> wrote:
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>>> Hi Team,
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> I have done some POC on
>> rolling
>> > > > > upgrade and
>> > > > > >>> found
>> > > > > >>> > >>> below
>> > > > > >>> > >>> >> >> result.
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> 1. On 1st node upgrade
>> zookeeper
>> > > .
>> > > > > Traffic
>> > > > > >>> was
>> > > > > >>> > >>> >> running
>> > > > > >>> > >>> >> >> fine
>> > > > > >>> > >>> >> >> > >> because
>> > > > > >>> > >>> >> >> > >>> 2
>> > > > > >>> > >>> >> >> > >>>>> nodes are already on old
>> > > zookeeper.
>> > > > > >>> > >>> >> >> > >>>>> 2. On 1st node upgrade our
>> > > > > application and
>> > > > > >>> > didn’t
>> > > > > >>> > >>> >> find
>> > > > > >>> > >>> >> >> any issue
>> > > > > >>> > >>> >> >> > >>>>> 3. On 2nd node upgrade
>> zookeeper
>> > > > but
>> > > > > got
>> > > > > >>> below
>> > > > > >>> > >>> error
>> > > > > >>> > >>> >> >> and zookeeper
>> > > > > >>> > >>> >> >> > >>> is
>> > > > > >>> > >>> >> >> > >>>>> not taking any requests
>> > > > > >>> > >>> >> >> > >>>>> 4.
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> java.io.EOFException
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> at
>> > > > > >>> > >>> >> >>
>> > > > > java.io.DataInputStream.readInt(DataInputStream.java:392)
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> at
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> 2020-03-30 14:19:55,587 - WARN
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > [RecvWorker:1:QuorumCnxManager$RecvWorker@765]
>> > > > > >>> -
>> > > > > >>> > >>> >> >> Interrupting
>> > > > > >>> > >>> >> >> > >>> SendWorker
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> 2020-03-30 14:19:55,588 -
>> ERROR
>> > > > > >>> [LearnerHandler-/
>> > > > > >>> > >>> >> >> 192.168.44.73:33754
>> > > > > >>> > >>> >> >> > >>>>> :LearnerHandler@562] -
>> Unexpected
>> > > > > exception
>> > > > > >>> > >>> causing
>> > > > > >>> > >>> >> >> shutdown while
>> > > > > >>> > >>> >> >> > >>> sock
>> > > > > >>> > >>> >> >> > >>>>> still open
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> java.io.EOFException
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> at
>> > > > > >>> > >>> >> >>
>> > > > > java.io.DataInputStream.readInt(DataInputStream.java:392)
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> at
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>>
>> > > > > >>>
>> > > > >
>> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> at
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> at
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> at
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >>
>> > > > > >>> > >>>
>> > > > > >>> >
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:476)
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> 2020-03-30 14:19:55,588 - WARN
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > [SendWorker:1:QuorumCnxManager$SendWorker@679]
>> > > > > >>> -
>> > > > > >>> > >>> >> >> Interrupted while
>> > > > > >>> > >>> >> >> > >>>> waiting
>> > > > > >>> > >>> >> >> > >>>>> for message on queue
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> Please let me know is this the
>> > > known
>> > > > > issue or
>> > > > > >>> > this
>> > > > > >>> > >>> is
>> > > > > >>> > >>> >> >> different issue
>> > > > > >>> > >>> >> >> > >>>> which
>> > > > > >>> > >>> >> >> > >>>>> is mention in Apache zookeeper
>> > > > > documentation
>> > > > > >>> when
>> > > > > >>> > >>> >> >> upgrading from
>> > > > > >>> > >>> >> >> > >> 3.4.5
>> > > > > >>> > >>> >> >> > >>> to
>> > > > > >>> > >>> >> >> > >>>>> 3.5.6
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> Thanks,
>> > > > > >>> > >>> >> >> > >>>>> ---------------------
>> > > > > >>> > >>> >> >> > >>>>> Kuldeep Singh Budania
>> > > > > >>> > >>> >> >> > >>>>> Software Architect
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>> On Sun, Mar 29, 2020 at 9:06
>> AM
>> > > > > Alexander
>> > > > > >>> Shraer
>> > > > > >>> > <
>> > > > > >>> > >>> >> >> shra...@gmail.com>
>> > > > > >>> > >>> >> >> > >>>>> wrote:
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>>> +1 to what Mate said (I
>> wrote the
>> > > > > quoted
>> > > > > >>> > >>> >> instructions).
>> > > > > >>> > >>> >> >> > >>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>
>> > > > > >>> > >>> >> >> > >>>>>> On Tue, Mar 24, 2020 at 7:03
>> AM
>> > > > > Szalay-Bekő
>> > > > > >>> > Máté <
>> > > > > >>> > >>> >> >> > >>>>>> szalay.beko.m...@gmail.com>
>> > > > > >>> > >>> >> >> > >>>>>> wrote:
>> > > > > >>> > >>> >> >> > >>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>> Hi Kuldeep,
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>> I just want to provide you
>> some
>> > > > > background
>> > > > > >>> info
>> > > > > >>> > >>> about
>> > > > > >>> > >>> >> >> our
>> > > > > >>> > >>> >> >> > >>>>> documentation.
>> > > > > >>> > >>> >> >> > >>>>>>> The reason to upgrade to
>> 3.4.6
>> > > > first
>> > > > > is to
>> > > > > >>> > avoid
>> > > > > >>> > >>> the
>> > > > > >>> > >>> >> >> following
>> > > > > >>> > >>> >> >> > >>> error:
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>> 2013-01-30 11:32:10,663
>> > > [myid:2] -
>> > > > > WARN
>> > > > > >>> > >>> [localhost/
>> > > > > >>> > >>> >> >> > >>> 127.0.0.1:2784
>> > > > > >>> > >>> >> >> > >>>>>>> :QuorumCnxManager@349] -
>> Invalid
>> > > > > server
>> > > > > >>> id:
>> > > > > >>> > >>> -65536
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>> This error comes because of
>> the
>> > > > > protocol
>> > > > > >>> > changes
>> > > > > >>> > >>> >> between
>> > > > > >>> > >>> >> >> > >> ZooKeeper
>> > > > > >>> > >>> >> >> > >>>>> server
>> > > > > >>> > >>> >> >> > >>>>>>> nodes during connection
>> > > initiation
>> > > > > for
>> > > > > >>> leader
>> > > > > >>> > >>> >> election.
>> > > > > >>> > >>> >> >> In
>> > > > > >>> > >>> >> >> > >>> ZooKeeper
>> > > > > >>> > >>> >> >> > >>>>> 3.5
>> > > > > >>> > >>> >> >> > >>>>>> a
>> > > > > >>> > >>> >> >> > >>>>>>> protocol version was
>> introduced
>> > > > (see
>> > > > > >>> > >>> ZOOKEEPER-107)
>> > > > > >>> > >>> >> and
>> > > > > >>> > >>> >> >> since
>> > > > > >>> > >>> >> >> > >> that
>> > > > > >>> > >>> >> >> > >>>> time
>> > > > > >>> > >>> >> >> > >>>>>> the
>> > > > > >>> > >>> >> >> > >>>>>>> fist long value sent in the
>> > > initial
>> > > > > >>> message is
>> > > > > >>> > >>> not
>> > > > > >>> > >>> >> the
>> > > > > >>> > >>> >> >> server ID
>> > > > > >>> > >>> >> >> > >>> but
>> > > > > >>> > >>> >> >> > >>>>> the
>> > > > > >>> > >>> >> >> > >>>>>>> protocol version (-65536).
>> In
>> > > > > ZooKeeper
>> > > > > >>> 3.4.6
>> > > > > >>> > we
>> > > > > >>> > >>> made
>> > > > > >>> > >>> >> >> the old 3.4
>> > > > > >>> > >>> >> >> > >>>>>>> ZooKeepers backward
>> compatible,
>> > > so
>> > > > > they are
>> > > > > >>> > able
>> > > > > >>> > >>> to
>> > > > > >>> > >>> >> >> parse both
>> > > > > >>> > >>> >> >> > >> the
>> > > > > >>> > >>> >> >> > >>>> old
>> > > > > >>> > >>> >> >> > >>>>>> and
>> > > > > >>> > >>> >> >> > >>>>>>> the new protocol format (see
>> > > > > >>> ZOOKEEPER-1633).
>> > > > > >>> > >>> This
>> > > > > >>> > >>> >> >> issue happens
>> > > > > >>> > >>> >> >> > >>> only
>> > > > > >>> > >>> >> >> > >>>>>> when
>> > > > > >>> > >>> >> >> > >>>>>>> you need to use old (3.4.0 -
>> > > 3.4.5)
>> > > > > and new
>> > > > > >>> > >>> (3.5.0+)
>> > > > > >>> > >>> >> >> ZooKeeper
>> > > > > >>> > >>> >> >> > >>>> servers
>> > > > > >>> > >>> >> >> > >>>>>>> together in the same
>> cluster.
>> > > > During
>> > > > > a
>> > > > > >>> rolling
>> > > > > >>> > >>> >> upgrade,
>> > > > > >>> > >>> >> >> this is
>> > > > > >>> > >>> >> >> > >>>> usually
>> > > > > >>> > >>> >> >> > >>>>>> the
>> > > > > >>> > >>> >> >> > >>>>>>> case to have old and new
>> > > ZooKeepers
>> > > > > present
>> > > > > >>> > >>> together.
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>> The fact that you haven't
>> seen
>> > > any
>> > > > > issues
>> > > > > >>> might
>> > > > > >>> > >>> be
>> > > > > >>> > >>> >> >> caused by the
>> > > > > >>> > >>> >> >> > >>>> order
>> > > > > >>> > >>> >> >> > >>>>> of
>> > > > > >>> > >>> >> >> > >>>>>>> the servers. In ZooKeeper
>> the
>> > > > > connection
>> > > > > >>> > >>> initiation
>> > > > > >>> > >>> >> >> between the
>> > > > > >>> > >>> >> >> > >>>> servers
>> > > > > >>> > >>> >> >> > >>>>>>> during the leader election
>> > > follows
>> > > > a
>> > > > > >>> specific
>> > > > > >>> > >>> rule.
>> > > > > >>> > >>> >> As
>> > > > > >>> > >>> >> >> far as I
>> > > > > >>> > >>> >> >> > >>>>> remember
>> > > > > >>> > >>> >> >> > >>>>>>> always the server with the
>> larger
>> > > > ID
>> > > > > 'wins
>> > > > > >>> the
>> > > > > >>> > >>> >> >> challenge', so it
>> > > > > >>> > >>> >> >> > >> is
>> > > > > >>> > >>> >> >> > >>>>>>> possible, that the old
>> server
>> > > > didn't
>> > > > > need
>> > > > > >>> to
>> > > > > >>> > >>> parse
>> > > > > >>> > >>> >> any
>> > > > > >>> > >>> >> >> initial
>> > > > > >>> > >>> >> >> > >>>> message
>> > > > > >>> > >>> >> >> > >>>>>> (if
>> > > > > >>> > >>> >> >> > >>>>>>> it had the largest ID) and
>> this
>> > > is
>> > > > > why you
>> > > > > >>> > >>> haven't
>> > > > > >>> > >>> >> seen
>> > > > > >>> > >>> >> >> the
>> > > > > >>> > >>> >> >> > >> issue.
>> > > > > >>> > >>> >> >> > >>>> Also
>> > > > > >>> > >>> >> >> > >>>>>>> having 2 nodes up from the 3
>> > > nodes
>> > > > > cluster
>> > > > > >>> > still
>> > > > > >>> > >>> >> makes
>> > > > > >>> > >>> >> >> the
>> > > > > >>> > >>> >> >> > >> cluster
>> > > > > >>> > >>> >> >> > >>>> work
>> > > > > >>> > >>> >> >> > >>>>>> (so
>> > > > > >>> > >>> >> >> > >>>>>>> you should also check if
>> all the
>> > > > > servers
>> > > > > >>> are
>> > > > > >>> > >>> part of
>> > > > > >>> > >>> >> >> the quorum).
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>> I agree with Enrico and
>> Norbert,
>> > > > the
>> > > > > >>> safest and
>> > > > > >>> > >>> most
>> > > > > >>> > >>> >> >> stable way
>> > > > > >>> > >>> >> >> > >> is
>> > > > > >>> > >>> >> >> > >>>>>> upgrade
>> > > > > >>> > >>> >> >> > >>>>>>> first to 3.4.latest, then
>> go to
>> > > > > 3.5.latest.
>> > > > > >>> > >>> Still, if
>> > > > > >>> > >>> >> >> you don't
>> > > > > >>> > >>> >> >> > >> see
>> > > > > >>> > >>> >> >> > >>>>> that
>> > > > > >>> > >>> >> >> > >>>>>>> you would hit this specific
>> issue
>> > > > > (e.g. no
>> > > > > >>> > >>> "Invalid
>> > > > > >>> > >>> >> >> server id" in
>> > > > > >>> > >>> >> >> > >>> the
>> > > > > >>> > >>> >> >> > >>>>> log
>> > > > > >>> > >>> >> >> > >>>>>>> files), and all the three
>> servers
>> > > > can
>> > > > > >>> handle
>> > > > > >>> > >>> traffic,
>> > > > > >>> > >>> >> >> then maybe
>> > > > > >>> > >>> >> >> > >>> you
>> > > > > >>> > >>> >> >> > >>>>>> don't
>> > > > > >>> > >>> >> >> > >>>>>>> need to upgrade first to
>> > > > 3.4.latest,
>> > > > > it is
>> > > > > >>> your
>> > > > > >>> > >>> >> >> decision.
>> > > > > >>> > >>> >> >> > >>> Definitely
>> > > > > >>> > >>> >> >> > >>>>> you
>> > > > > >>> > >>> >> >> > >>>>>>> should test it first, as
>> > > suggested
>> > > > > by the
>> > > > > >>> > others.
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>> Kind regards,
>> > > > > >>> > >>> >> >> > >>>>>>> Mate
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>> On Tue, Mar 24, 2020 at
>> 12:29 PM
>> > > > > Norbert
>> > > > > >>> Kalmar
>> > > > > >>> > >>> >> >> > >>>>>>>
>> <nkal...@cloudera.com.invalid>
>> > > > > wrote:
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>> Hi,
>> > > > > >>> > >>> >> >> > >>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>> That guide is to upgrade to
>> > > 3.5.0,
>> > > > > which
>> > > > > >>> was
>> > > > > >>> > an
>> > > > > >>> > >>> >> alpha
>> > > > > >>> > >>> >> >> version.
>> > > > > >>> > >>> >> >> > >> A
>> > > > > >>> > >>> >> >> > >>>> lot
>> > > > > >>> > >>> >> >> > >>>>>> has
>> > > > > >>> > >>> >> >> > >>>>>>>> changed for the first
>> stable
>> > > > > release of
>> > > > > >>> 3.5.5
>> > > > > >>> > >>> and
>> > > > > >>> > >>> >> then
>> > > > > >>> > >>> >> >> a few
>> > > > > >>> > >>> >> >> > >>> more,
>> > > > > >>> > >>> >> >> > >>>>> even
>> > > > > >>> > >>> >> >> > >>>>>>>> rolling upgrade issues
>> have been
>> > > > > fixed for
>> > > > > >>> > >>> 3.5.6.
>> > > > > >>> > >>> >> >> > >>>>>>>> This is a more up-to-date
>> guide:
>> > > > > >>> > >>> >> >> > >>>>>>>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >>
>> > > > > >>> >
>> > > > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Upgrade+FAQ
>> > > > > >>> > >>> >> >> > >>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>> If you have done your
>> testing
>> > > > (with
>> > > > > prod
>> > > > > >>> > >>> snapshot!),
>> > > > > >>> > >>> >> >> then you
>> > > > > >>> > >>> >> >> > >> can
>> > > > > >>> > >>> >> >> > >>>>> skip
>> > > > > >>> > >>> >> >> > >>>>>>> 3.4
>> > > > > >>> > >>> >> >> > >>>>>>>> latest upgrade, but keep
>> in mind
>> > > > we
>> > > > > do our
>> > > > > >>> > >>> >> >> recommendations for
>> > > > > >>> > >>> >> >> > >> a
>> > > > > >>> > >>> >> >> > >>>>>> reason.
>> > > > > >>> > >>> >> >> > >>>>>>>> There were issues reported
>> > > and/or
>> > > > > found
>> > > > > >>> during
>> > > > > >>> > >>> >> >> testing. Some
>> > > > > >>> > >>> >> >> > >> are
>> > > > > >>> > >>> >> >> > >>>>> fixed
>> > > > > >>> > >>> >> >> > >>>>>>> with
>> > > > > >>> > >>> >> >> > >>>>>>>> 3.5.6, some only happens if
>> > > > certain
>> > > > > >>> conditions
>> > > > > >>> > >>> stand
>> > > > > >>> > >>> >> >> > >>> (IOException:
>> > > > > >>> > >>> >> >> > >>>> No
>> > > > > >>> > >>> >> >> > >>>>>>>> snapshot found - mentioned
>> in
>> > > the
>> > > > > guide,
>> > > > > >>> fixed
>> > > > > >>> > >>> in
>> > > > > >>> > >>> >> >> 3.5.6).
>> > > > > >>> > >>> >> >> > >>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>> So it is up to you, I would
>> > > still
>> > > > > >>> recommend to
>> > > > > >>> > >>> do an
>> > > > > >>> > >>> >> >> 3.4
>> > > > > >>> > >>> >> >> > >> upgrade
>> > > > > >>> > >>> >> >> > >>>>> first,
>> > > > > >>> > >>> >> >> > >>>>>>> if
>> > > > > >>> > >>> >> >> > >>>>>>>> it's feasible.
>> > > > > >>> > >>> >> >> > >>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>> Regards,
>> > > > > >>> > >>> >> >> > >>>>>>>> Norbert
>> > > > > >>> > >>> >> >> > >>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>> On Tue, Mar 24, 2020 at
>> 11:45 AM
>> > > > > kuldeep
>> > > > > >>> > singh <
>> > > > > >>> > >>> >> >> > >>>>>>> kuldeep.sing...@gmail.com>
>> > > > > >>> > >>> >> >> > >>>>>>>> wrote:
>> > > > > >>> > >>> >> >> > >>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>> Hi,
>> > > > > >>> > >>> >> >> > >>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>> Current Zookeeper version
>> :-
>> > > > 3.4.5
>> > > > > >>> > >>> >> >> > >>>>>>>>> Upgraded version
>> > > > :-
>> > > > > 3.5.6
>> > > > > >>> > >>> >> >> > >>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>> We are not going with
>> 3.5.7.
>> > > Our
>> > > > > final
>> > > > > >>> > >>> decision is
>> > > > > >>> > >>> >> >> zookeeper
>> > > > > >>> > >>> >> >> > >>>>> version
>> > > > > >>> > >>> >> >> > >>>>>> is
>> > > > > >>> > >>> >> >> > >>>>>>>>> 3.5.6
>> > > > > >>> > >>> >> >> > >>>>>>>>> as per your reply first
>> we need
>> > > > to
>> > > > > move
>> > > > > >>> > latest
>> > > > > >>> > >>> >> >> version of
>> > > > > >>> > >>> >> >> > >>> 3.4.x,
>> > > > > >>> > >>> >> >> > >>>>> like
>> > > > > >>> > >>> >> >> > >>>>>>>> below
>> > > > > >>> > >>> >> >> > >>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>> 3.4.5 -> 3.4.14 -> 3.5.6
>> > > (Correct
>> > > > > me if
>> > > > > >>> I am
>> > > > > >>> > >>> wrong
>> > > > > >>> > >>> >> >> here)
>> > > > > >>> > >>> >> >> > >>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>> But if We are not facing
>> any
>> > > > > problem
>> > > > > >>> that i
>> > > > > >>> > >>> have
>> > > > > >>> > >>> >> >> shared you
>> > > > > >>> > >>> >> >> > >>> that
>> > > > > >>> > >>> >> >> > >>>> we
>> > > > > >>> > >>> >> >> > >>>>>>> have
>> > > > > >>> > >>> >> >> > >>>>>>>>> set up of 3 node cluster
>> where
>> > > 2
>> > > > > node
>> > > > > >>> are on
>> > > > > >>> > >>> 3.5.6
>> > > > > >>> > >>> >> >> version
>> > > > > >>> > >>> >> >> > >> and
>> > > > > >>> > >>> >> >> > >>> 1
>> > > > > >>> > >>> >> >> > >>>>> node
>> > > > > >>> > >>> >> >> > >>>>>>> on
>> > > > > >>> > >>> >> >> > >>>>>>>>> 3.4.5, Everything is
>> running
>> > > fine
>> > > > > and
>> > > > > >>> didn't
>> > > > > >>> > >>> get
>> > > > > >>> > >>> >> any
>> > > > > >>> > >>> >> >> issue,
>> > > > > >>> > >>> >> >> > >> So
>> > > > > >>> > >>> >> >> > >>>> what
>> > > > > >>> > >>> >> >> > >>>>>>> other
>> > > > > >>> > >>> >> >> > >>>>>>>>> problem we can face if we
>> > > > directly
>> > > > > move
>> > > > > >>> to
>> > > > > >>> > >>> 3.5.6
>> > > > > >>> > >>> >> >> > >>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>> Thanks,
>> > > > > >>> > >>> >> >> > >>>>>>>>> ---------------------
>> > > > > >>> > >>> >> >> > >>>>>>>>> Kuldeep Singh Budania
>> > > > > >>> > >>> >> >> > >>>>>>>>> Software Architect
>> > > > > >>> > >>> >> >> > >>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>> On Tue, Mar 24, 2020 at
>> 3:58 PM
>> > > > > Enrico
>> > > > > >>> > >>> Olivelli <
>> > > > > >>> > >>> >> >> > >>>>> eolive...@gmail.com
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>> wrote:
>> > > > > >>> > >>> >> >> > >>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>> Hi
>> > > > > >>> > >>> >> >> > >>>>>>>>>> You have to upgrade to
>> latest
>> > > > > 3.4.x
>> > > > > >>> > Zookeeper
>> > > > > >>> > >>> then
>> > > > > >>> > >>> >> >> you will
>> > > > > >>> > >>> >> >> > >>>>> upgrade
>> > > > > >>> > >>> >> >> > >>>>>>> to
>> > > > > >>> > >>> >> >> > >>>>>>>>>> 3.5.7.
>> > > > > >>> > >>> >> >> > >>>>>>>>>> All should run well
>> without
>> > > > issues
>> > > > > >>> > >>> >> >> > >>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>> Enrico
>> > > > > >>> > >>> >> >> > >>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>> Il Mar 24 Mar 2020, 10:18
>> > > > kuldeep
>> > > > > singh
>> > > > > >>> <
>> > > > > >>> > >>> >> >> > >>>>> kuldeep.sing...@gmail.com
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>> ha
>> > > > > >>> > >>> >> >> > >>>>>>>>>> scritto:
>> > > > > >>> > >>> >> >> > >>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> Hi Team,
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> We are upgrading
>> zookeeper
>> > > from
>> > > > > 3.4.5
>> > > > > >>> to
>> > > > > >>> > >>> 3.5.6. I
>> > > > > >>> > >>> >> >> have
>> > > > > >>> > >>> >> >> > >> set
>> > > > > >>> > >>> >> >> > >>>> up 3
>> > > > > >>> > >>> >> >> > >>>>>>> node
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> cluster where 2 node
>> are on
>> > > > 3.5.6
>> > > > > >>> version
>> > > > > >>> > >>> and 1
>> > > > > >>> > >>> >> >> node on
>> > > > > >>> > >>> >> >> > >>>> 3.4.5.
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> Everything is running
>> fine
>> > > and
>> > > > > didn't
>> > > > > >>> get
>> > > > > >>> > any
>> > > > > >>> > >>> >> issue
>> > > > > >>> > >>> >> >> on my
>> > > > > >>> > >>> >> >> > >>>>> system.
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> but I found something on
>> > > apache
>> > > > > site
>> > > > > >>> that
>> > > > > >>> > >>> first
>> > > > > >>> > >>> >> we
>> > > > > >>> > >>> >> >> need
>> > > > > >>> > >>> >> >> > >> to
>> > > > > >>> > >>> >> >> > >>>>>> upgrade
>> > > > > >>> > >>> >> >> > >>>>>>>> on
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> 3.4.6 than we can
>> upgrade to
>> > > > > 3.5.6. So
>> > > > > >>> is
>> > > > > >>> > it
>> > > > > >>> > >>> >> >> mandatory
>> > > > > >>> > >>> >> >> > >> to
>> > > > > >>> > >>> >> >> > >>> go
>> > > > > >>> > >>> >> >> > >>>>> on
>> > > > > >>> > >>> >> >> > >>>>>>>> 3.4.6
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> first.
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> *Upgrading to 3.5.0*
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> Upgrading a running
>> ZooKeeper
>> > > > > ensemble
>> > > > > >>> to
>> > > > > >>> > >>> 3.5.0
>> > > > > >>> > >>> >> >> should be
>> > > > > >>> > >>> >> >> > >>>> done
>> > > > > >>> > >>> >> >> > >>>>>> only
>> > > > > >>> > >>> >> >> > >>>>>>>>> after
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> upgrading your ensemble
>> to
>> > > the
>> > > > > 3.4.6
>> > > > > >>> > release.
>> > > > > >>> > >>> >> Note
>> > > > > >>> > >>> >> >> that
>> > > > > >>> > >>> >> >> > >>> this
>> > > > > >>> > >>> >> >> > >>>> is
>> > > > > >>> > >>> >> >> > >>>>>>> only
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> necessary for rolling
>> > > upgrades
>> > > > > (if
>> > > > > >>> you're
>> > > > > >>> > >>> fine
>> > > > > >>> > >>> >> with
>> > > > > >>> > >>> >> >> > >>> shutting
>> > > > > >>> > >>> >> >> > >>>>> down
>> > > > > >>> > >>> >> >> > >>>>>>> the
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> system completely, you
>> don't
>> > > > > have to go
>> > > > > >>> > >>> through
>> > > > > >>> > >>> >> >> 3.4.6).
>> > > > > >>> > >>> >> >> > >> If
>> > > > > >>> > >>> >> >> > >>>> you
>> > > > > >>> > >>> >> >> > >>>>>>>> attempt
>> > > > > >>> > >>> >> >> > >>>>>>>>> a
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> rolling upgrade without
>> going
>> > > > > through
>> > > > > >>> 3.4.6
>> > > > > >>> > >>> (for
>> > > > > >>> > >>> >> >> example
>> > > > > >>> > >>> >> >> > >>> from
>> > > > > >>> > >>> >> >> > >>>>>>> 3.4.5),
>> > > > > >>> > >>> >> >> > >>>>>>>>> you
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> may get the following
>> error:
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> 2013-01-30 11:32:10,663
>> > > > [myid:2]
>> > > > > - INFO
>> > > > > >>> > >>> >> [localhost/
>> > > > > >>> > >>> >> >> > >>>>>> 127.0.0.1:2784
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > :QuorumCnxManager$Listener@498
>> > > > ]
>> > > > > -
>> > > > > >>> Received
>> > > > > >>> > >>> >> >> connection
>> > > > > >>> > >>> >> >> > >>>> request
>> > > > > >>> > >>> >> >> > >>>>> /
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> 127.0.0.1:60876
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> 2013-01-30 11:32:10,663
>> > > > [myid:2]
>> > > > > - WARN
>> > > > > >>> > >>> >> [localhost/
>> > > > > >>> > >>> >> >> > >>>>>> 127.0.0.1:2784
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> :QuorumCnxManager@349]
>> -
>> > > > Invalid
>> > > > > >>> server
>> > > > > >>> > id:
>> > > > > >>> > >>> >> -65536
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> During a rolling
>> upgrade,
>> > > each
>> > > > > server
>> > > > > >>> is
>> > > > > >>> > >>> taken
>> > > > > >>> > >>> >> down
>> > > > > >>> > >>> >> >> in
>> > > > > >>> > >>> >> >> > >> turn
>> > > > > >>> > >>> >> >> > >>>> and
>> > > > > >>> > >>> >> >> > >>>>>>>>> rebooted
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> with the new 3.5.0
>> binaries.
>> > > > > Before
>> > > > > >>> > starting
>> > > > > >>> > >>> the
>> > > > > >>> > >>> >> >> server
>> > > > > >>> > >>> >> >> > >>> with
>> > > > > >>> > >>> >> >> > >>>>>> 3.5.0
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> binaries, we highly
>> recommend
>> > > > > updating
>> > > > > >>> the
>> > > > > >>> > >>> >> >> configuration
>> > > > > >>> > >>> >> >> > >>> file
>> > > > > >>> > >>> >> >> > >>>>> so
>> > > > > >>> > >>> >> >> > >>>>>>> that
>> > > > > >>> > >>> >> >> > >>>>>>>>> all
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> server statements
>> > > > "server.x=..."
>> > > > > >>> contain
>> > > > > >>> > >>> client
>> > > > > >>> > >>> >> >> ports
>> > > > > >>> > >>> >> >> > >> (see
>> > > > > >>> > >>> >> >> > >>>> the
>> > > > > >>> > >>> >> >> > >>>>>>>> section
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> Specifying
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> the client port). As
>> > > explained
>> > > > > earlier
>> > > > > >>> you
>> > > > > >>> > >>> may
>> > > > > >>> > >>> >> >> leave the
>> > > > > >>> > >>> >> >> > >>>>>>>> configuration
>> > > > > >>> > >>> >> >> > >>>>>>>>>> in a
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> single file, as well as
>> leave
>> > > > the
>> > > > > >>> > >>> >> >> > >>>> clientPort/clientPortAddress
>> > > > > >>> > >>> >> >> > >>>>>>>>> statements
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> (although if you specify
>> > > client
>> > > > > ports
>> > > > > >>> in
>> > > > > >>> > the
>> > > > > >>> > >>> new
>> > > > > >>> > >>> >> >> format,
>> > > > > >>> > >>> >> >> > >>>> these
>> > > > > >>> > >>> >> >> > >>>>>>>>> statements
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> are now redundant).
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> Could you please let me
>> know
>> > > > > about this
>> > > > > >>> > case.
>> > > > > >>> > >>> >> >> Appreciate
>> > > > > >>> > >>> >> >> > >> if
>> > > > > >>> > >>> >> >> > >>>>>> respond
>> > > > > >>> > >>> >> >> > >>>>>>>>> soon.
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> Thanks,
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> ---------------------
>> > > > > >>> > >>> >> >> > >>>>>>>>>>> Kuldeep Singh Budania
>> > > > > >>> > >>> >> >> > >>>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>>
>> > > > > >>> > >>> >> >> > >>>>>>
>> > > > > >>> > >>> >> >> > >>>>>
>> > > > > >>> > >>> >> >> > >>>>
>> > > > > >>> > >>> >> >> > >>>
>> > > > > >>> > >>> >> >> > >>
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >> >
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> ________________________________
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> CONFIDENTIALITY & PRIVACY NOTICE
>> > > > > >>> > >>> >> >> This e-mail (including any attachments) is
>> > > > strictly
>> > > > > >>> > >>> confidential
>> > > > > >>> > >>> >> and
>> > > > > >>> > >>> >> >> may also contain privileged information. If
>> you are
>> > > > > not the
>> > > > > >>> > >>> intended
>> > > > > >>> > >>> >> >> recipient you are not authorised to read,
>> print,
>> > > > save,
>> > > > > >>> process or
>> > > > > >>> > >>> >> disclose
>> > > > > >>> > >>> >> >> this message. If you have received this
>> message by
>> > > > > mistake,
>> > > > > >>> > please
>> > > > > >>> > >>> >> inform
>> > > > > >>> > >>> >> >> the sender immediately and destroy this
>> e-mail, its
>> > > > > >>> attachments
>> > > > > >>> > >>> and any
>> > > > > >>> > >>> >> >> copies. Any use, distribution, reproduction or
>> > > > > disclosure by
>> > > > > >>> any
>> > > > > >>> > >>> person
>> > > > > >>> > >>> >> >> other than the intended recipient is strictly
>> > > > > prohibited and
>> > > > > >>> the
>> > > > > >>> > >>> person
>> > > > > >>> > >>> >> >> responsible may incur in penalties.
>> > > > > >>> > >>> >> >> The use of this e-mail is only for
>> professional
>> > > > > purposes;
>> > > > > >>> > >>> there is
>> > > > > >>> > >>> >> no
>> > > > > >>> > >>> >> >> guarantee that the correspondence towards this
>> > > e-mail
>> > > > > will be
>> > > > > >>> > read
>> > > > > >>> > >>> >> only by
>> > > > > >>> > >>> >> >> the recipient, because, under certain
>> > > circumstances,
>> > > > > there
>> > > > > >>> may
>> > > > > >>> > be a
>> > > > > >>> > >>> >> need to
>> > > > > >>> > >>> >> >> access this email by third subjects belonging
>> to
>> > > the
>> > > > > Company.
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> ________________________________
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >> CONFIDENTIALITY & PRIVACY NOTICE
>> > > > > >>> > >>> >> >> This e-mail (including any attachments) is
>> strictly
>> > > > > >>> confidential
>> > > > > >>> > >>> and
>> > > > > >>> > >>> >> may
>> > > > > >>> > >>> >> >> also contain privileged information. If you
>> are not
>> > > > the
>> > > > > >>> intended
>> > > > > >>> > >>> >> recipient
>> > > > > >>> > >>> >> >> you are not authorised to read, print, save,
>> > > process
>> > > > or
>> > > > > >>> disclose
>> > > > > >>> > >>> this
>> > > > > >>> > >>> >> >> message. If you have received this message by
>> > > > mistake,
>> > > > > please
>> > > > > >>> > >>> inform
>> > > > > >>> > >>> >> the
>> > > > > >>> > >>> >> >> sender immediately and destroy this e-mail,
>> its
>> > > > > attachments
>> > > > > >>> and
>> > > > > >>> > any
>> > > > > >>> > >>> >> copies.
>> > > > > >>> > >>> >> >> Any use, distribution, reproduction or
>> disclosure
>> > > by
>> > > > > any
>> > > > > >>> person
>> > > > > >>> > >>> other
>> > > > > >>> > >>> >> than
>> > > > > >>> > >>> >> >> the intended recipient is strictly prohibited
>> and
>> > > the
>> > > > > person
>> > > > > >>> > >>> >> responsible
>> > > > > >>> > >>> >> >> may incur in penalties.
>> > > > > >>> > >>> >> >> The use of this e-mail is only for
>> professional
>> > > > > purposes;
>> > > > > >>> there
>> > > > > >>> > is
>> > > > > >>> > >>> no
>> > > > > >>> > >>> >> >> guarantee that the correspondence towards this
>> > > e-mail
>> > > > > will be
>> > > > > >>> > read
>> > > > > >>> > >>> >> only by
>> > > > > >>> > >>> >> >> the recipient, because, under certain
>> > > circumstances,
>> > > > > there
>> > > > > >>> may
>> > > > > >>> > be a
>> > > > > >>> > >>> >> need to
>> > > > > >>> > >>> >> >> access this email by third subjects belonging
>> to
>> > > the
>> > > > > Company.
>> > > > > >>> > >>> >> >>
>> > > > > >>> > >>> >> >
>> > > > > >>> > >>> >>
>> > > > > >>> > >>> >
>> > > > > >>> > >>>
>> > > > > >>> > >>
>> > > > > >>> >
>> > > > > >>>
>> > > > > >>
>> > > > >
>> > > > >
>> > > > > ________________________________
>> > > > >
>> > > > > CONFIDENTIALITY & PRIVACY NOTICE
>> > > > > This e-mail (including any attachments) is strictly confidential
>> and
>> > > may
>> > > > > also contain privileged information. If you are not the intended
>> > > > recipient
>> > > > > you are not authorised to read, print, save, process or disclose
>> this
>> > > > > message. If you have received this message by mistake, please
>> inform
>> > > the
>> > > > > sender immediately and destroy this e-mail, its attachments and
>> any
>> > > > copies.
>> > > > > Any use, distribution, reproduction or disclosure by any person
>> other
>> > > > than
>> > > > > the intended recipient is strictly prohibited and the person
>> > > responsible
>> > > > > may incur in penalties.
>> > > > > The use of this e-mail is only for professional purposes; there
>> is no
>> > > > > guarantee that the correspondence towards this e-mail will be
>> read only
>> > > > by
>> > > > > the recipient, because, under certain circumstances, there may be
>> a
>> > > need
>> > > > to
>> > > > > access this email by third subjects belonging to the Company.
>> > > > >
>> > > >
>> > >
>>
>>
