[jira] [Commented] (KAFKA-3993) Console producer drops data
[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552622#comment-15552622 ] ASF GitHub Bot commented on KAFKA-3993: --- Github user theduderog closed the pull request at: https://github.com/apache/kafka/pull/1667 > Console producer drops data > --- > > Key: KAFKA-3993 > URL: https://issues.apache.org/jira/browse/KAFKA-3993 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.10.0.0 >Reporter: Roger Hoover > > The console producer drops data when if the process exits too quickly. I > suspect that the shutdown hook does not call close() or something goes wrong > during that close(). > Here's a simple to illustrate the issue: > {noformat} > export BOOTSTRAP_SERVERS=localhost:9092 > export TOPIC=bar > export MESSAGES=1 > ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --partitions 1 > --replication-factor 1 --topic "$TOPIC" \ > && echo "acks=all" > /tmp/producer.config \ > && echo "linger.ms=0" >> /tmp/producer.config \ > && seq "$MESSAGES" | ./bin/kafka-console-producer.sh --broker-list > "$BOOTSTRAP_SERVERS" --topic "$TOPIC" --producer-config /tmp/producer.config \ > && ./bin/kafka-console-consumer.sh --bootstrap-server "$BOOTSTRAP_SERVERS" > --new-consumer --from-beginning --max-messages "${MESSAGES}" --topic "$TOPIC" > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3993) Console producer drops data
[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450054#comment-15450054 ] Roger Hoover commented on KAFKA-3993: - Thanks, [~cotedm]. I tried to set acks=all but apparently that doesn't work. > Console producer drops data > --- > > Key: KAFKA-3993 > URL: https://issues.apache.org/jira/browse/KAFKA-3993 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.10.0.0 >Reporter: Roger Hoover > > The console producer drops data when if the process exits too quickly. I > suspect that the shutdown hook does not call close() or something goes wrong > during that close(). > Here's a simple to illustrate the issue: > {noformat} > export BOOTSTRAP_SERVERS=localhost:9092 > export TOPIC=bar > export MESSAGES=1 > ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --partitions 1 > --replication-factor 1 --topic "$TOPIC" \ > && echo "acks=all" > /tmp/producer.config \ > && echo "linger.ms=0" >> /tmp/producer.config \ > && seq "$MESSAGES" | ./bin/kafka-console-producer.sh --broker-list > "$BOOTSTRAP_SERVERS" --topic "$TOPIC" --producer-config /tmp/producer.config \ > && ./bin/kafka-console-consumer.sh --bootstrap-server "$BOOTSTRAP_SERVERS" > --new-consumer --from-beginning --max-messages "${MESSAGES}" --topic "$TOPIC" > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3993) Console producer drops data
[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415811#comment-15415811 ] Roger Hoover commented on KAFKA-3993: - Thanks, [~vahid]. Yeah, it looks the same. > Console producer drops data > --- > > Key: KAFKA-3993 > URL: https://issues.apache.org/jira/browse/KAFKA-3993 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.10.0.0 >Reporter: Roger Hoover > > The console producer drops data when if the process exits too quickly. I > suspect that the shutdown hook does not call close() or something goes wrong > during that close(). > Here's a simple to illustrate the issue: > {noformat} > export BOOTSTRAP_SERVERS=localhost:9092 > export TOPIC=bar > export MESSAGES=1 > ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --partitions 1 > --replication-factor 1 --topic "$TOPIC" \ > && echo "acks=all" > /tmp/producer.config \ > && echo "linger.ms=0" >> /tmp/producer.config \ > && seq "$MESSAGES" | ./bin/kafka-console-producer.sh --broker-list > "$BOOTSTRAP_SERVERS" --topic "$TOPIC" --producer-config /tmp/producer.config \ > && ./bin/kafka-console-consumer.sh --bootstrap-server "$BOOTSTRAP_SERVERS" > --new-consumer --from-beginning --max-messages "${MESSAGES}" --topic "$TOPIC" > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3993) Console producer drops data
[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415793#comment-15415793 ] Vahid Hashemian commented on KAFKA-3993: [~theduderog] I wonder if this is the same issue as the one reported in [this JIRA|https://issues.apache.org/jira/browse/KAFKA-3129]. > Console producer drops data > --- > > Key: KAFKA-3993 > URL: https://issues.apache.org/jira/browse/KAFKA-3993 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.10.0.0 >Reporter: Roger Hoover > > The console producer drops data when if the process exits too quickly. I > suspect that the shutdown hook does not call close() or something goes wrong > during that close(). > Here's a simple to illustrate the issue: > {noformat} > export BOOTSTRAP_SERVERS=localhost:9092 > export TOPIC=bar > export MESSAGES=1 > ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --partitions 1 > --replication-factor 1 --topic "$TOPIC" \ > && echo "acks=all" > /tmp/producer.config \ > && echo "linger.ms=0" >> /tmp/producer.config \ > && seq "$MESSAGES" | ./bin/kafka-console-producer.sh --broker-list > "$BOOTSTRAP_SERVERS" --topic "$TOPIC" --producer-config /tmp/producer.config \ > && ./bin/kafka-console-consumer.sh --bootstrap-server "$BOOTSTRAP_SERVERS" > --new-consumer --from-beginning --max-messages "${MESSAGES}" --topic "$TOPIC" > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3993) Console producer drops data
[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394244#comment-15394244 ] ASF GitHub Bot commented on KAFKA-3993: --- GitHub user theduderog opened a pull request: https://github.com/apache/kafka/pull/1667 KAFKA-3993 Console Producer Drops Data @ijuma Would you mind taking a look? This fixes the problem for me using this test script. ``` export BOOTSTRAP_SERVERS=localhost:9092 export TOPIC=bar export MESSAGES=1 ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --partitions 1 --replication-factor 1 --topic "$TOPIC" \ && echo "acks=all" > /tmp/producer.config \ && echo "linger.ms=0" >> /tmp/producer.config \ && seq "$MESSAGES" | ./bin/kafka-console-producer.sh --broker-list "$BOOTSTRAP_SERVERS" --topic "$TOPIC" \ && ./bin/kafka-console-consumer.sh --bootstrap-server "$BOOTSTRAP_SERVERS" --new-consumer --from-beginning --max-messages "${MESSAGES}" --topic "$TOPIC" ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/theduderog/kafka KAFKA-3993-close-producer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1667.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1667 commit 093d2aaaf85a49dbfe1a07a8ea47739909a321b7 Author: Roger HooverDate: 2016-07-26T18:08:47Z Added close to producer > Console producer drops data > --- > > Key: KAFKA-3993 > URL: https://issues.apache.org/jira/browse/KAFKA-3993 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.10.0.0 >Reporter: Roger Hoover > > The console producer drops data when if the process exits too quickly. I > suspect that the shutdown hook does not call close() or something goes wrong > during that close(). > Here's a simple to illustrate the issue: > {noformat} > export BOOTSTRAP_SERVERS=localhost:9092 > export TOPIC=bar > export MESSAGES=1 > ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --partitions 1 > --replication-factor 1 --topic "$TOPIC" \ > && echo "acks=all" > /tmp/producer.config \ > && echo "linger.ms=0" >> /tmp/producer.config \ > && seq "$MESSAGES" | ./bin/kafka-console-producer.sh --broker-list > "$BOOTSTRAP_SERVERS" --topic "$TOPIC" \ > && ./bin/kafka-console-consumer.sh --bootstrap-server "$BOOTSTRAP_SERVERS" > --new-consumer --from-beginning --max-messages "${MESSAGES}" --topic "$TOPIC" > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)