[
https://issues.apache.org/jira/browse/FLUME-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marimuthu Ponnambalam updated FLUME-2405:
-----------------------------------------
Attachment: ScribeSource.java
{code}
//Added to accept custom frame size
private int thriftFrameSize = DEFAULT_THRIFT_FRAME_SIZE;
private SourceCounter sourceCounter;
@Override
public void configure(Context context) {
port = context.getInteger("port", port);
workers = context.getInteger("workerThreads", DEFAULT_WORKERS);
if (workers <= 0) {
workers = DEFAULT_WORKERS;
}
if (sourceCounter == null) {
sourceCounter = new SourceCounter(getName());
}
thriftFrameSize = context.getInteger("thriftFrameSize",
DEFAULT_THRIFT_FRAME_SIZE);
if (thriftFrameSize <= 0) {
thriftFrameSize = DEFAULT_THRIFT_FRAME_SIZE;
}
}
private class Startup extends Thread {
public void run() {
try {
Scribe.Processor processor = new Scribe.Processor(new Receiver());
TNonblockingServerTransport transport = new
TNonblockingServerSocket(port);
THsHaServer.Args args = new THsHaServer.Args(transport);
args.workerThreads(workers);
args.processor(processor);
args.transportFactory(new TFramedTransport.Factory(thriftFrameSize));
{code}
> Flume stops working on arbitrary Frame Size more than 16 MB on ScribeSource
> ---------------------------------------------------------------------------
>
> Key: FLUME-2405
> URL: https://issues.apache.org/jira/browse/FLUME-2405
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.4.0, v1.5.0
> Reporter: Marimuthu Ponnambalam
> Priority: Critical
> Labels: FrameSize, Scribe, Source
> Fix For: v1.4.0, v1.5.0, v1.5.0.1, v1.6.0
>
> Attachments: ScribeSource.java
>
>
> Flume stops receiving data on scribe source if there is an arbitrary size of
> data send from scribe, beyond 16MB.
> TFrameTransport throws exception, due to frame size is more than 16 MB, which
> is default size on TFrameTransport.
> This can be resolved by adding one more property "thriftFrameSize", which
> will allow custom frame size be configured, which will be used to create
> TFrameTransport.
> This happens while scribe lag is being cleared, scribe sends Frame size
> larger than 16MB.
> http://stackoverflow.com/questions/24150439/flume-scribesource-how-to-set-thrift-frame-size
> 2014-06-10 19:40:40,405 WARN org.apache.thrift.server.THsHaServer: Exception
> while invoking!
> org.apache.thrift.transport.TTransportException: Frame size (23757404) larger
> than max length (16384000)!
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
