[
https://issues.apache.org/jira/browse/FLUME-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240857#comment-13240857
]
[email protected] commented on FLUME-1037:
------------------------------------------------------
bq. On 2012-03-28 23:58:37, Hari Shreedharan wrote:
bq. > flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java,
line 438
bq. > <https://reviews.apache.org/r/4497/diff/2/?file=96931#file96931line438>
bq. >
bq. > Don't we run the risk of buffer overflow here? I think reader would
try to dump data using a put operation. If the buffer is full, there could be a
BufferOverflowException here, right?
bq.
bq. Mike Percy wrote:
bq. Nope... see
http://docs.oracle.com/javase/6/docs/api/java/io/Reader.html#read%28java.nio.CharBuffer%29
It does say it is a put operation.A put operation can cause an overflow
exception, maybe the reader somehow handles it. Otherwise looks good!
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4497/#review6498
-----------------------------------------------------------
On 2012-03-28 23:16:36, Mike Percy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4497/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-28 23:16:36)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The NetcatSource does not create one event per newline. Instead, it
creates one event per TCP session.
bq.
bq. I have addressed multiple issues with NetcatSource in this patch:
bq.
bq. 1. Properly process a new event per newline.
bq. 2. Enforce a maximum length per line, to ensure clients cannot run the
server out of memory. (This is now configurable; the default is 512 characters.)
bq. 3. Properly flush responses to the client.
bq. 4. Increment counters for successful processing and failure.
bq. 5. Use shutdownNow() when shutting down the server, otherwise an open
client connection will cause the server to hang on shutdown.
bq. 6. Made the inner classes of NetcatSource private; I believe making them
public was unintentional.
bq. 7. Corrected unit test so it now sends a newline with each request. Also
now checks for "OK" response from server.
bq. 8. Attempting to sneak in a minor fix for the EventHelper.dumpEvent()
method into this patch, since I'm testing with LoggerSink and it's bugging me
when it throws an exception on a zero-length body.
bq.
bq.
bq. This addresses bug FLUME-1037.
bq. https://issues.apache.org/jira/browse/FLUME-1037
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/src/main/java/org/apache/flume/event/EventHelper.java
a326a70
bq. flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java
a841b0e
bq.
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java
fb2a960
bq.
bq. Diff: https://reviews.apache.org/r/4497/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Unit tests pass. Did a bunch of manual testing using the nc tool with a
Netcat source and a Logger sink.
bq.
bq.
bq. Thanks,
bq.
bq. Mike
bq.
bq.
> NETCAT handler theads terminate under stress test
> -------------------------------------------------
>
> Key: FLUME-1037
> URL: https://issues.apache.org/jira/browse/FLUME-1037
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Environment: [CentOS 6.2 64-bit]
> Reporter: Will McQueen
> Assignee: Mike Percy
>
> Steps:
> 1. Use a props file such as the following:
> \# a = agent
> \# r = source
> \# c = channel
> \# k = sink
> a1.sources = r1
> a1.channels = c1
> a1.sinks = k1
> \# ===SOURCES===
> a1.sources.r1.type = NETCAT
> a1.sources.r1.channels = c1
> a1.sources.r1.bind = localhost
> a1.sources.r1.port = 1473
> \# ===CHANNELS===
> a1.channels.c1.type = MEMORY
> \# ===SINKS===
> a1.sinks.k1.type = NULL
> a1.sinks.k1.channel = c1
> 2. Set the FLUME_CONF_DIR to point to your conf dir
> [will@localhost flume-1.2.0-incubating-SNAPSHOT]$ export
> FLUME_CONF_DIR=/home/will/git/apache/flume/flume-1.2.0-incubating-SNAPSHOT/conf
> 3. Create a flume-env.sh file
> [will@localhost flume-1.2.0-incubating-SNAPSHOT]$ cp
> conf/flume-env.sh.template conf/flume-env.sh
> 4. Adjust the memory size within flume-env.sh (this file will be
> automatically sourced when calling bin/flume-ng, but only if you've specified
> the FLUME_CONF_DIR env var)
> (here, I went to the extreme and I set the min and max heap to 1GB. I also
> specified a YourKit profiler agent)
> Sample contents of flume-env.sh:
> export JAVA_OPTS="-Xms1024m -Xmx1024m
> -agentpath:/home/will/tools/yjp-10.0.6/bin/linux-x86-64/libyjpagent.so=tracing,noj2ee"
> 5. Run the flume NG agent:
> bin/flume-ng node --conf conf --conf-file conf/a1.properties --name a1
> 6. Open-up 10 terminal windows (on the same host) to connect to the netcat
> server port. Sent continuous output in each terminal. I chose to use the
> command:
> yes | nc localhost 1473
> The "yes" unix cmd will continuously output 'y' char, followed by newline
> char. If you use YourKit and go into the Threads view, you'll see that after
> a while (possibly need to wait up to 10 mins) after a netcat handler thread
> has continuously been alternating between Runnable and Blocked states
> (blocking due to org.apache.klog4j.Category.log(..), but that's beside the
> point), the netcat handler thread enters a continuous wait state for exactly
> 1 minute, and then terminates (while its associated 'yes | nc localhost 1473'
> command is still running).
> I haven't done further analysis. My first thought was a thread safety issue.
> Note that there are no property file reconfigurations done during this test
> -- I leave the props file alone.
> I welcome your ideas/comments. I initially ran this test with the default
> -Xmx20m but it ran out of memory. For a future test I might lower the Xmx/Xms
> from 1GB to maybe 128MB.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
