Exception "too many open files" in ThriftEventSource (with infinite loop as
result)
-----------------------------------------------------------------------------------
Key: FLUME-929
URL: https://issues.apache.org/jira/browse/FLUME-929
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v0.9.4
Environment: Flume 0.9.4-cdh3u2 on Linux Squeeze, System with one
master and two nodes
Reporter: bert Passek
Priority: Blocker
We use ThriftEventSource for logging huge amount of data. However, an exception
is thrown randomly and not reproducable which results in an infinite loop in
one of the worker threads. We had to stop all flume nodes because the log file
was dramatically increasing due to logged exceptions.
This is just a small part of the log file:
2012-01-11 14:57:30,487 ERROR com.cloudera.flume.core.connector.DirectDriver:
Exiting driver logicalNode newsletterImpressionLog01-21 in error state
ThriftEventSource | CassandraSink because sleep interrupted
2012-01-11 17:18:14,808 WARN org.apache.thrift.server.TSaneThreadPoolServer:
Transport error occurred during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
Caused by: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
... 2 more
2012-01-11 17:18:14,809 WARN org.apache.thrift.server.TSaneThreadPoolServer:
Transport error occurred during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
Caused by: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
... 2 more
2012-01-11 17:18:14,809 WARN org.apache.thrift.server.TSaneThreadPoolServer:
Transport error occurred during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
Caused by: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
... 2 more
2012-01-11 17:18:14,809 WARN org.apache.thrift.server.TSaneThreadPoolServer:
Transport error occurred during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
Caused by: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
... 2 more
2012-01-11 17:18:14,810 WARN org.apache.thrift.server.TSaneThreadPoolServer:
Transport error occurred during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
Caused by: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
... 2 more
2012-01-11 17:18:14,810 WARN org.apache.thrift.server.TSaneThreadPoolServer:
Transport error occurred during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
Caused by: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
... 2 more
2012-01-11 17:18:14,810 WARN org.apache.thrift.server.TSaneThreadPoolServer:
Transport error occurred during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
Caused by: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
... 2 more
So there are actually 2 issues to get fixed:
1. Why are there "too many open files"?
2. The worker thread must shutdown correctly and not ending in an infinite loop.
This is a blocker issue for us because with this kind of error we are not able
to use flume furthermore.
Best Regards
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
