[
https://issues.apache.org/jira/browse/ZOOKEEPER-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015550#comment-13015550
]
César Álvarez Núñez commented on ZOOKEEPER-1043:
------------------------------------------------
Hi Mahadev,
I apologize for my poor description. I will try to do it better this time.
The bug has two parts: Server-side and Client-side.
Few days ago (Mon, Mar 14, 2011) one of our production "Zookeeper server" was
restarted because it was consuming too much CPU and flooding log files with
millions of lines like the following.
{noformat}
2011-04-04 12:13:10,841 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn$Factory@272][] -
Ignoring unexpected runtime exception
java.lang.NullPointerException
at
org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:244)
{noformat}
Unfortunately I was not able to recover the log content previous to the NPE.
But it responds fine to an "ruok" and "stat".
A few hours later, the content of one ZNode was modified resulting on the
behavior described in the email.
I've reviewed the code that makes use of the affected ZNodes and I'm sure that
it does not invoke any "exists" method.
Today the "Zookeeper server" on the same physical server than before start
again to flooding log files.
Fortunately this time I have recovered the log entries (which are included on
bug description) previous to the NPE loop.
The "Zookeeper server" was shutdown at 16:19+- and restarted at 16:23:50
>From the "zookeeper client" side:
- Only one client shows a reconnection following by unexpected NodeCreated.
{noformat}
2011-04-04 16:19:58,307 I [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:Disconnected type:None path:null)
2011-04-04 16:19:58,864 I [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:None path:null)
2011-04-04 16:19:58,870 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/AP/subscriptionLegalConditions)
2011-04-04 16:19:58,871 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/Certification/fileTemp)
2011-04-04 16:19:58,871 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/Pdf/urlFont)
2011-04-04 16:19:58,872 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/ActiveX/classid)
2011-04-04 16:19:58,872 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/Signature/timeOut)
2011-04-04 16:19:58,873 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/Locale/locales)
2011-04-04 16:19:58,873 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/pc/locales)
2011-04-04 16:19:58,874 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/Deh/aceptarNotComLegalConditions)
2011-04-04 16:19:58,874 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/pc/pathImagenIssuingEntity)
2011-04-04 16:19:58,875 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/pc/context)
2011-04-04 16:19:58,875 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/ActiveX/xadesActivex)
2011-04-04 16:19:58,876 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/pc/maxMostReadedFaqs)
2011-04-04 16:19:58,876 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/Deh/createUserLegalConditions)
2011-04-04 16:19:58,877 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/pc/hostAlternative)
2011-04-04 16:19:58,877 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/Deh/modificationUserLegalConditions)
2011-04-04 16:19:58,878 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/Pdf/imageLogo)
2011-04-04 16:19:58,879 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/Pdf/font)
2011-04-04 16:19:58,879 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/pc/host)
2011-04-04 16:19:58,880 D [main-EventThread] e.z.ReadOnlyConfiguration@bc794cc
- process(WatchedEvent state:SyncConnected type:NodeCreated
path:/path/Configuration/Avisos/fechaFin)
{noformat}
- The above log belongs to a client which was restarted at 13:06.
- The "/path/Configuration/Avisos/fechaFin" has as ctime "10-nov-2010 18:00:53"
and as mtime "11-mar-2011 14:07:14"
- I've modified one ZNode which should have triggered a NodeDataChanged on the
tree "zookeeper client" instances; but no one has received any WatchedEvent.
> Looped NPE at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:244)
> -------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1043
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1043
> Project: ZooKeeper
> Issue Type: Bug
> Affects Versions: 3.3.3
> Environment: Sparc Solaris 10
> Java 6u17 64 bits
> 5 nodes ensemble
> Reporter: César Álvarez Núñez
>
> I'm sorry but I only have this log (which belongs to a "follower" node) and a
> previous message [Unexpected NodeCreated event after a
> reconnection.|http://mail-archives.apache.org/mod_mbox/zookeeper-user/201103.mbox/%[email protected]%3E]
> where I describe a potential side-effect at client side.
> {noformat}
> 2011-04-04 09:31:09,608 - INFO [Snapshot Thread:FileTxnSnapLog@208][] -
> Snapshotting: 1700527e36
> 2011-04-04 09:31:09,653 - INFO [SyncThread:1:FileTxnLog@197][] - Creating
> new log file: log.1700527e38
> 2011-04-04 10:13:39,287 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn$Factory@251][] -
> Accepted socket connection from /XXX.XXX.XXX.69:1093
> 2011-04-04 10:13:39,371 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn@777][] - Client
> attempting to establish new session at /XXX.XXX.XXX.69:1093
> 2011-04-04 10:13:39,376 - INFO [CommitProcessor:1:NIOServerCnxn@1580][] -
> Established session 0x12ee79c4a720022 with negotiated timeout 20000 for
> client /XXX.XXX.XXX.69:1093
> 2011-04-04 12:04:11,131 - INFO [SyncThread:1:FileTxnLog@197][] - Creating
> new log file: log.170053bf15
> 2011-04-04 12:04:11,131 - INFO [Snapshot Thread:FileTxnSnapLog@208][] -
> Snapshotting: 170053bf17
> 2011-04-04 12:13:10,779 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn$Factory@251][] -
> Accepted socket connection from /XXX.XXX.XXX.63:1817
> 2011-04-04 12:13:10,790 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn@777][] - Client
> attempting to establish new session at /XXX.XXX.XXX.63:1817
> 2011-04-04 12:13:10,794 - INFO [CommitProcessor:1:NIOServerCnxn@1580][] -
> Established session 0x12ee79c4a720023 with negotiated timeout 20000 for
> client /XXX.XXX.XXX.63:1817
> 2011-04-04 12:13:10,814 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn@634][] -
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x12ee79c4a720023, likely client has closed socket
> 2011-04-04 12:13:10,816 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn@1435][] - Closed
> socket connection for client /XXX.XXX.XXX.63:1817 which had sessionid
> 0x12ee79c4a720023
> 2011-04-04 12:13:10,839 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn$Factory@251][] -
> Accepted socket connection from /XXX.XXX.XXX.63:1814
> 2011-04-04 12:13:10,840 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn$Factory@274][] -
> Ignoring exception
> java.net.SocketException: Invalid argument
> at sun.nio.ch.Net.setIntOption0(Native Method)
> at sun.nio.ch.Net.setIntOption(Unknown Source)
> at sun.nio.ch.SocketChannelImpl$1.setInt(Unknown Source)
> at sun.nio.ch.SocketOptsImpl.setBoolean(Unknown Source)
> at sun.nio.ch.SocketOptsImpl$IP$TCP.noDelay(Unknown Source)
> at sun.nio.ch.OptionAdaptor.setTcpNoDelay(Unknown Source)
> at sun.nio.ch.SocketAdaptor.setTcpNoDelay(Unknown Source)
> at
> org.apache.zookeeper.server.NIOServerCnxn.<init>(NIOServerCnxn.java:1367)
> at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.createConnection(NIOServerCnxn.java:215)
> at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:256)
> 2011-04-04 12:13:10,841 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn$Factory@272][] -
> Ignoring unexpected runtime exception
> java.lang.NullPointerException
> at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:244)
> 2011-04-04 12:13:10,841 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn$Factory@272][] -
> Ignoring unexpected runtime exception
> java.lang.NullPointerException
> at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:244)
> 2011-04-04 12:13:10,842 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn$Factory@272][] -
> Ignoring unexpected runtime exception
> java.lang.NullPointerException
> at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:244)
> ...
> ...
> ...
> 2011-04-04 16:49:23,101 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2301:NIOServerCnxn$Factory@272][] -
> Ignoring unexpected runtime exception
> java.lang.NullPointerException
> at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:244)
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira