[jira] [Commented] (FLUME-2883) LoadBalancingRpcClient returns wrong value for batchSize
[ https://issues.apache.org/jira/browse/FLUME-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535071#comment-15535071 ] Umesh Chaudhary commented on FLUME-2883: Hi [~agnek], Apologies as I assign this to me without asking to you. Would you want to work on this ? > LoadBalancingRpcClient returns wrong value for batchSize > > > Key: FLUME-2883 > URL: https://issues.apache.org/jira/browse/FLUME-2883 > Project: Flume > Issue Type: Bug > Components: Client SDK >Reporter: Alexey Churkin >Assignee: Umesh Chaudhary >Priority: Minor > > LoadBalancingRpcClient doesn't read batchSize property from configuration and > returns default value in getBatchSize method. > But underlying clients creates with right batchSize value read from > configuration. So LoadBalancingRpcClient should also read batch-size > configuration param and set it field batchSize according to it value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2811) Taildir source doesn't call stop() on graceful shutdown
[
https://issues.apache.org/jira/browse/FLUME-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535050#comment-15535050
]
Umesh Chaudhary commented on FLUME-2811:
Hello [~siefried12], [~liorze] IMHO InterruptedException would be thrown when
the thread is interrupted in method tailFileProcess and ChannelException is
thrown when channel is misbehaving. So InterruptedException should not cause
infinite loop.
> Taildir source doesn't call stop() on graceful shutdown
> ---
>
> Key: FLUME-2811
> URL: https://issues.apache.org/jira/browse/FLUME-2811
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Jun Seok Hong
>Assignee: Umesh Chaudhary
>Priority: Critical
> Labels: newbie
> Fix For: v1.7.0
>
>
> Taildir source doesn't call stop() on graceful shutdown.
> Test configuration.
> source - taildir
> channel - PseudoTxnMemoryChannel / flume-kafka-channel
> sink - none
> I found that flume sometimes doesn't terminate with Taildir source.
> I had to kill the process to terminate it.
> tailFileProcess() function in TaildirSource.java has a infinite loop.
> When the process interrupted, ChannelException will happen, but it can't
> breaks the infinite loop.
> I think that's the reason why Taildir can't call stop() function.
> {code:title=TaildirSource.java|borderStyle=solid}
> private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
> throws IOException, InterruptedException {
> while (true) {
> reader.setCurrentFile(tf);
> List events = reader.readEvents(batchSize, backoffWithoutNL);
> if (events.isEmpty()) {
> break;
> }
> sourceCounter.addToEventReceivedCount(events.size());
> sourceCounter.incrementAppendBatchReceivedCount();
> try {
> getChannelProcessor().processEventBatch(events);
> reader.commit();
> } catch (ChannelException ex) {
> logger.warn("The channel is full or unexpected failure. " +
> "The source will try again after " + retryInterval + " ms");
> TimeUnit.MILLISECONDS.sleep(retryInterval);
> retryInterval = retryInterval << 1;
> retryInterval = Math.min(retryInterval, maxRetryInterval);
> continue;
> }
> retryInterval = 1000;
> sourceCounter.addToEventAcceptedCount(events.size());
> sourceCounter.incrementAppendBatchAcceptedCount();
> if (events.size() < batchSize) {
> break;
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2583) Flafka unit tests should randomize ports
[ https://issues.apache.org/jira/browse/FLUME-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535035#comment-15535035 ] Umesh Chaudhary commented on FLUME-2583: Hi [~gwenshap] , would it be fine if I take this one? Will need your inputs though. > Flafka unit tests should randomize ports > > > Key: FLUME-2583 > URL: https://issues.apache.org/jira/browse/FLUME-2583 > Project: Flume > Issue Type: Improvement >Reporter: Gwen Shapira >Assignee: Gwen Shapira > > Flafka (i.e. Kafka source, sink and channel) unit tests don't randomize > ports, therefore they can fail (after creating a small mess) when running on > machines that already have Kafka or Zookeeper running. > Lets fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2623) Test testRemoveFields(org.apache.flume.source.TestSyslogUdpSource) fails randomly
[
https://issues.apache.org/jira/browse/FLUME-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535027#comment-15535027
]
Umesh Chaudhary commented on FLUME-2623:
Hi [~hshreedharan] [~jholoman] [~bessbd] Would it be fine if I take this one?
> Test testRemoveFields(org.apache.flume.source.TestSyslogUdpSource) fails
> randomly
> -
>
> Key: FLUME-2623
> URL: https://issues.apache.org/jira/browse/FLUME-2623
> Project: Flume
> Issue Type: Bug
> Components: Test
>Affects Versions: v1.5.0
> Environment: RHEL 7.1 on PPC64 LE
>Reporter: Tony Reix
>Priority: Minor
>
> The test org.apache.flume.source.TestSyslogUdpSource is not 100% reliable. It
> fails sometimes randomly.
> Source code dealing with the issue is:
> flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java
> (about line 101)
> for (int i = 0; i < 100 ; i++) {
> syslogSocket = new DatagramSocket();
> syslogSocket.send(datagramPacket);
> syslogSocket.close();
> }
> List channelEvents = new ArrayList();
> Transaction txn = channel.getTransaction();
> txn.begin();
> for (int i = 0; i < 100; i++) {
> Event e = channel.take();
> Assert.assertNotNull(e);
> channelEvents.add(e);
> }
> Sometimes... "e" is null.
> Failure deals with:
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertNotNull(Assert.java:526) at
> org.junit.Assert.assertNotNull(Assert.java:537)
> at
> org.apache.flume.source.TestSyslogUdpSource.runKeepFieldsTest(TestSyslogUdpSource.java:101)
> at
> org.apache.flume.source.TestSyslogUdpSource.testRemoveFields(TestSyslogUdpSource.java:177)
> With OpenJDK, I got it failing once out of 30 tries.
> However, with IBM JVM, I got if failing 6 times out of 10.
> After I added a Thread.sleep(2000) in the middle, with IBM JVM, I've reduced
> the probability of the failure from 6/10 to 2/10 . So, that helps, but that
> is not enough. A better solution must be found.
> The issue appears more often with IBM JVM probably because things are handled
> differently, or quicker, by IBM JVM. Anyway, the issue also appears with
> OpenJDK.
> I guess that the issue is still there with master version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2988) Kafka Sink metrics missing eventDrainAttemptCount
[
https://issues.apache.org/jira/browse/FLUME-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534702#comment-15534702
]
Jeff Holoman commented on FLUME-2988:
-
Go for it.
> Kafka Sink metrics missing eventDrainAttemptCount
> -
>
> Key: FLUME-2988
> URL: https://issues.apache.org/jira/browse/FLUME-2988
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Reporter: Denes Arvay
>Assignee: Udai Kiran Potluri
>Priority: Minor
>
> {{eventDrainAttemptCount}} doesn't get incremented in Kafka Sink, only the
> {{eventDrainSuccessCount}}
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L210)
> resulting in misleading statistics (e.g. when calculating {{attemptCount -
> successCount}})
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2988) Kafka Sink metrics missing eventDrainAttemptCount
[
https://issues.apache.org/jira/browse/FLUME-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534380#comment-15534380
]
Udai Kiran Potluri commented on FLUME-2988:
---
[~jholoman] - Would it be ok if I take this one?
> Kafka Sink metrics missing eventDrainAttemptCount
> -
>
> Key: FLUME-2988
> URL: https://issues.apache.org/jira/browse/FLUME-2988
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Reporter: Denes Arvay
>Assignee: Udai Kiran Potluri
>Priority: Minor
>
> {{eventDrainAttemptCount}} doesn't get incremented in Kafka Sink, only the
> {{eventDrainSuccessCount}}
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L210)
> resulting in misleading statistics (e.g. when calculating {{attemptCount -
> successCount}})
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLUME-2988) Kafka Sink metrics missing eventDrainAttemptCount
[
https://issues.apache.org/jira/browse/FLUME-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udai Kiran Potluri reassigned FLUME-2988:
-
Assignee: Udai Kiran Potluri (was: Jeff Holoman)
> Kafka Sink metrics missing eventDrainAttemptCount
> -
>
> Key: FLUME-2988
> URL: https://issues.apache.org/jira/browse/FLUME-2988
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Reporter: Denes Arvay
>Assignee: Udai Kiran Potluri
>Priority: Minor
>
> {{eventDrainAttemptCount}} doesn't get incremented in Kafka Sink, only the
> {{eventDrainSuccessCount}}
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L210)
> resulting in misleading statistics (e.g. when calculating {{attemptCount -
> successCount}})
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[ https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532922#comment-15532922 ] Attila Simon commented on FLUME-2998: - Long.MAX_VALUE is 9223372036854775807. Although Long.MAX_VALUE is not a valid expression in the config file I would doubt that the exact number is useful in the docs. What I saw in other places is "see description". If it really needed to be changed from Long.MAX_VALUE in the "default value" column then we may use that trick and put it into the description column. > Add missing configuration parameter to SequenceSource docs > -- > > Key: FLUME-2998 > URL: https://issues.apache.org/jira/browse/FLUME-2998 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2998.patch > > > totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[ https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2998: Attachment: (was: FLUME-2998.patch) > Add missing configuration parameter to SequenceSource docs > -- > > Key: FLUME-2998 > URL: https://issues.apache.org/jira/browse/FLUME-2998 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2998.patch > > > totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[ https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2998: Attachment: FLUME-2998.patch > Add missing configuration parameter to SequenceSource docs > -- > > Key: FLUME-2998 > URL: https://issues.apache.org/jira/browse/FLUME-2998 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2998.patch > > > totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[
https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532585#comment-15532585
]
Bessenyei Balázs Donát commented on FLUME-2998:
---
[~sati]: thank you for the patch!
* What do you mean be "practically" in the first line of the change?
* nit: "Retry if put failed." -> "Retries when it can't send events to the
channel."
Side note: this would be the first parameter in the documentation with a Java
constant ("Long.MAX_VALUE") for default. Do we want to maybe put numbers there
instead?
> Add missing configuration parameter to SequenceSource docs
> --
>
> Key: FLUME-2998
> URL: https://issues.apache.org/jira/browse/FLUME-2998
> Project: Flume
> Issue Type: Documentation
> Components: Docs
>Affects Versions: v1.7.0
>Reporter: Attila Simon
>Assignee: Attila Simon
> Fix For: v1.7.0
>
> Attachments: FLUME-2998.patch
>
>
> totalEvents can be specified
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
