[GitHub] storm issue #1276: STORM-1664: Allow Java users to start a local cluster wit...

2016-06-14 Thread srdo
Github user srdo commented on the issue: https://github.com/apache/storm/pull/1276 @knusbaum Mind merging if you're still +1 on this change? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] storm issue #1487: off-by-one error for UNCOMMITTED_LATEST and UNCOMMITTED_E...

2016-06-14 Thread srdo
Github user srdo commented on the issue: https://github.com/apache/storm/pull/1487 Are you sure this is the case? It looks to me like the spout is committing the offset of the latest acked tuple that doesn't have uncommitted predecessors. It seems to make sense that it seeks

[GitHub] storm issue #1487: off-by-one error for UNCOMMITTED_LATEST and UNCOMMITTED_E...

2016-06-16 Thread srdo
Github user srdo commented on the issue: https://github.com/apache/storm/pull/1487 Could you check also if you see the issue when auto commit is set to false? Are you running your topology in local mode or deploying it with storm jar? --- If your project is set up

[GitHub] storm issue #1487: off-by-one error for UNCOMMITTED_LATEST and UNCOMMITTED_E...

2016-06-16 Thread srdo
Github user srdo commented on the issue: https://github.com/apache/storm/pull/1487 You probably shouldn't enable auto commit if you can't accept some message loss. As I understand it, it makes the Kafka consumer commit the messages it has received from poll() every so often

[GitHub] storm pull request: STORM-1549: Add support for resetting tuple ti...

2016-02-14 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1107 STORM-1549: Add support for resetting tuple timeout from bolts via the OutputCollector See https://issues.apache.org/jira/browse/STORM-1549. Other than implementing tuple timeout reset, I

[GitHub] storm pull request: STORM-1549: Add support for resetting tuple ti...

2016-03-01 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1107#issuecomment-190881185 Great. I'll make another PR when I've made these changes on master. Should I close this one? --- If your project is set up for it, you can reply to this email and have

[GitHub] storm pull request: STORM-1549: Add support for resetting tuple ti...

2016-03-01 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1107#issuecomment-190949609 @revans2 Sure, I'll leave this open then. Here's the master version of this PR https://github.com/apache/storm/pull/1174 --- If your project is set up for it, you can

[GitHub] storm pull request: STORM-1549: [master] Add support for resetting...

2016-03-01 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1174 STORM-1549: [master] Add support for resetting tuple timeout from bolts via the OutputCollector This is a port of https://github.com/apache/storm/pull/1107 to master. You can merge this pull request

[GitHub] storm pull request: STORM-1549: [master] Add support for resetting...

2016-03-15 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1174#issuecomment-197008299 @revans2 @abhishekagarwal87 Sorry, didn't notice this had merge conflicts. It should be good to go now though :) --- If your project is set up for it, you can reply

[GitHub] storm pull request: STORM-1549: Add support for resetting tuple ti...

2016-03-15 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1107#issuecomment-197016989 The test failure looks unrelated to this PR, was also happening on 1.x-branch https://travis-ci.org/apache/storm/jobs/115948898 --- If your project is set up for it, you

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-03-15 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1209#issuecomment-196988005 I'll add some tests that errors are going into Zookeeper when executors hang soon/that workers are getting killed only if the option is enabled, and will take a look

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-03-13 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1209#issuecomment-196028832 I don't want this merged yet, just posted to get feedback :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-03-13 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1209 STORM-956: When the execute() or nextTuple() hang on external resources, stop the Worker's heartbeat The previous PR at https://github.com/apache/storm/pull/647 doesn't look active anymore. Having

[GitHub] storm pull request: STORM-1620: Update curator to fix CURATOR-209

2016-03-11 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1205 STORM-1620: Update curator to fix CURATOR-209 See https://issues.apache.org/jira/browse/STORM-1620 You can merge this pull request into a Git repository by running: $ git pull https://github.com

[GitHub] storm pull request: Shade Objenesis in storm-core

2016-03-29 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1274#issuecomment-202955987 @hustfxj Do you mean we should shade it? I don't understand why copying the Objenesis code into Storm would improve things. If we put it in the original package it's

[GitHub] storm pull request: Shade Objenesis in storm-core

2016-03-30 Thread srdo
Github user srdo closed the pull request at: https://github.com/apache/storm/pull/1274 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm pull request: Shade Objenesis in storm-core

2016-03-30 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1274#issuecomment-203626582 Seems like Objenesis is being reincluded here https://github.com/apache/storm/pull/1279. Closing this. --- If your project is set up for it, you can reply to this email

[GitHub] storm pull request: STORM-1664: Allow Java users to start a local ...

2016-03-30 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1276#issuecomment-203608785 Hi, yes it's useful for running https://github.com/roshannaik/storm-benchmark-sol in local mode, since that code uses NimbusClient for pulling throughput metrics

[GitHub] storm pull request: STORM-1664: Allow Java users to start a local ...

2016-03-30 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1276#discussion_r57956028 --- Diff: storm-core/src/clj/org/apache/storm/testing4j.clj --- @@ -79,10 +79,12 @@ [cluster-type mkClusterParam code] `(let [supervisors

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-04-08 Thread srdo
GitHub user srdo reopened a pull request: https://github.com/apache/storm/pull/1209 STORM-956: When the execute() or nextTuple() hang on external resources, stop the Worker's heartbeat The previous PR at https://github.com/apache/storm/pull/647 doesn't look active anymore. Having

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-04-08 Thread srdo
Github user srdo closed the pull request at: https://github.com/apache/storm/pull/1209 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-04-08 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1209#issuecomment-207432109 @hustfxj Please take another look when you have time, I think this feature is done :) The test failure seems to occur intermittently in storm-kafka, looks

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-03-20 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1209#issuecomment-198878063 Will reopen this when the PR is closer to done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-03-20 Thread srdo
Github user srdo closed the pull request at: https://github.com/apache/storm/pull/1209 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm pull request: Made StormTimer join task thread on close

2016-03-21 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1244 Made StormTimer join task thread on close Most tests creating a cluster can be flaky because the supervisor threads can terminate out of order. The StormTimer's close should block until the thread

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-03-21 Thread srdo
GitHub user srdo reopened a pull request: https://github.com/apache/storm/pull/1209 STORM-956: When the execute() or nextTuple() hang on external resources, stop the Worker's heartbeat The previous PR at https://github.com/apache/storm/pull/647 doesn't look active anymore. Having

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-03-21 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1209#issuecomment-199348274 The test failure looks to be the same as on master https://travis-ci.org/apache/storm/jobs/117200843. It ran locally for me with all-tests. --- If your project is set up

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-03-21 Thread srdo
Github user srdo closed the pull request at: https://github.com/apache/storm/pull/1209 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-03-21 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1209#issuecomment-199348649 Sorry, pressed the close button by accident. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] storm pull request: STORM-956: When the execute() or nextTuple() h...

2016-03-21 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1209#issuecomment-199321366 The hang checks should now support writing errors to Zookeeper, extending the timeout by interacting with an OutputCollector, setting different time limits per component

[GitHub] storm pull request: Fix wait-for-desired-code-replication always c...

2016-03-25 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1260 Fix wait-for-desired-code-replication always choosing distributed mod… …e regardless of actual cluster mode Since total-storm-conf is the merge of daemon conf and topology conf, it more

[GitHub] storm pull request: STORM-1549: [master] Add support for resetting...

2016-03-02 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1174#issuecomment-191326068 The NPE should be fixed, I've added the missing space to the if, and added a comment to the javadoc about reset timeout being an expensive operation. The comment has also

[GitHub] storm pull request: STORM-1549: Add support for resetting tuple ti...

2016-03-01 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1107#issuecomment-190826744 Anyone have any feedback on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] storm pull request: STORM-1664: Allow Java users to start a local ...

2016-03-29 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1276 STORM-1664: Allow Java users to start a local cluster with a Nimbus T… …hrift server. You can merge this pull request into a Git repository by running: $ git pull https://github.com/srdo

[GitHub] storm pull request: Shade Objenesis in storm-core

2016-03-29 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1274 Shade Objenesis in storm-core Versions of Kryo past 2.23 have Objenesis as a regular dependency instead of included in the Kryo jar. https://github.com/EsotericSoftware/kryo/commit

[GitHub] storm pull request: Shade Objenesis in storm-core

2016-03-29 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1274#issuecomment-202802884 Test failure seems unrelated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] storm pull request: Shade Objenesis in storm-core

2016-03-29 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1274#issuecomment-202845577 This doesn't really work. Shading Objenesis means that code with a dependency on storm-core won't get a functional Kryo, since the transitive dependency on Kryo refers

[GitHub] storm pull request: STORM-1745: Add partition to log output in Par...

2016-04-29 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1380 STORM-1745: Add partition to log output in PartitionManager Adding topic name and partition id to the log output makes it easier to debug when an issue arises. You can merge this pull request

[GitHub] storm pull request: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION...

2016-04-29 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1381 STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log to warn from error This error is transient most of the time and will occur during normal operation. Downgrading it to warning makes the error log

[GitHub] storm pull request: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION...

2016-04-29 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1381#discussion_r61555375 --- Diff: external/storm-kafka/src/jvm/org/apache/storm/kafka/KafkaUtils.java --- @@ -213,7 +213,11 @@ public static ByteBufferMessageSet fetchMessages

[GitHub] storm pull request: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION...

2016-04-29 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1381#issuecomment-215683426 This has the effect of downgrading all errors from fetch to warning level. Maybe the error log in KafkaUtils should be added back with a check for whether it's

[GitHub] storm pull request: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION...

2016-04-29 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1381#issuecomment-215688879 Looks like a flaky test. Going to try again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] storm pull request: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION...

2016-04-29 Thread srdo
GitHub user srdo reopened a pull request: https://github.com/apache/storm/pull/1381 STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log to warn from error This error is transient most of the time and will occur during normal operation. Downgrading it to warning makes the error log

[GitHub] storm pull request: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION...

2016-04-29 Thread srdo
Github user srdo closed the pull request at: https://github.com/apache/storm/pull/1381 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm pull request: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION...

2016-04-29 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1381#discussion_r61565774 --- Diff: external/storm-kafka/src/jvm/org/apache/storm/kafka/KafkaUtils.java --- @@ -213,7 +213,9 @@ public static ByteBufferMessageSet fetchMessages

[GitHub] storm pull request: STORM-1735: Nimbus should log that replication...

2016-04-27 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1369 STORM-1735: Nimbus should log that replication succeeded when min rep… …licas was reached exactly You can merge this pull request into a Git repository by running: $ git pull https

[GitHub] storm pull request: STORM-1750: Ensure worker dies when report-err...

2016-04-30 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1384 STORM-1750: Ensure worker dies when report-error-and-die is called. M… …ake ZkStateStorage set_data try setting data if node creation fails because the node exists Similar changes

[GitHub] storm pull request: STORM-1848: Make KafkaMessageId and Partition ...

2016-05-18 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1428#issuecomment-220129656 I'll leave it at adding the comments then. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] storm pull request: STORM-1844: Make netty unit test use STORM_TES...

2016-05-17 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1421 STORM-1844: Make netty unit test use STORM_TEST_TIMEOUT_MS, increase … …STORM_TEST_TIMEOUT_MS to 10 seconds, increase DisruptorQueueTest timeout to 5 seconds from 1 Link for convenience

[GitHub] storm pull request: STORM-1837: Fix complete-topology and prevent ...

2016-05-17 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1417#issuecomment-219674715 Something like this? https://github.com/apache/storm/pull/1421 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] storm pull request: STORM-1682: Refactor DynamicBrokersReader to l...

2016-05-17 Thread srdo
GitHub user srdo reopened a pull request: https://github.com/apache/storm/pull/1340 STORM-1682: Refactor DynamicBrokersReader to lookup leader informatio… …n via Kafka metadata API. Put Zookeeper metadata lookup into new class. See https://issues.apache.org/jira/browse

[GitHub] storm pull request: STORM-1682: Refactor DynamicBrokersReader to l...

2016-05-17 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1340#issuecomment-219834590 The tests pass on 0.8.2.1, which is the version targeted by https://github.com/apache/storm/pull/1386. Reopening. --- If your project is set up for it, you can reply

[GitHub] storm pull request: STORM-1837: Fix complete-topology and prevent ...

2016-05-16 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1417#issuecomment-219549248 I'd be happy to bump it. I don't think test-load-fn actually hits any of the changed code though, since it uses the netty.Context rather than local.Context. What do you

[GitHub] storm pull request: STORM-1755: Revert the kafka client version to...

2016-05-16 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1386#issuecomment-219405686 Shouldn't this also go on master? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] storm pull request: STORM-1682: Refactor DynamicBrokersReader to l...

2016-05-16 Thread srdo
Github user srdo closed the pull request at: https://github.com/apache/storm/pull/1340 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm pull request: STORM-1682: Refactor DynamicBrokersReader to l...

2016-05-16 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1340#issuecomment-219393499 Closing this temporarily. I need to check that this works on 0.8.x --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] storm pull request: STORM-1848: Make KafkaMessageId and Partition ...

2016-05-18 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1428 STORM-1848: Make KafkaMessageId and Partition serializable to support… … eventlogging You can merge this pull request into a Git repository by running: $ git pull https://github.com/srdo

[GitHub] storm pull request: STORM-1682: Refactor DynamicBrokersReader to l...

2016-05-18 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1340#issuecomment-219992956 It's not strictly necessary, no. It's just an alternate way of looking up leaders that avoids the concurrency issue when looking up the leader in Zookeeper. As a side

[GitHub] storm pull request: STORM-1848: Make KafkaMessageId and Partition ...

2016-05-18 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1428#issuecomment-220050357 Does it make sense to restrict the emit functions on SpoutOutputCollector to take only serializable messageIds? --- If your project is set up for it, you can reply

[GitHub] storm pull request: STORM-1848: Make KafkaMessageId and Partition ...

2016-05-18 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1428#issuecomment-220071724 You're right, I forgot about backwards compatibility, I mainly wanted to ensure this doesn't pop up in other spouts :) Is there any way for the Kafka spout to register

[GitHub] storm pull request: STORM-1837: Fix complete-topology and prevent ...

2016-05-15 Thread srdo
Github user srdo closed the pull request at: https://github.com/apache/storm/pull/1417 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm pull request: STORM-1837: Fix complete-topology and prevent ...

2016-05-15 Thread srdo
GitHub user srdo reopened a pull request: https://github.com/apache/storm/pull/1417 STORM-1837: Fix complete-topology and prevent message loss when runni… …ng local clusters with no time simulation Link for convenience https://issues.apache.org/jira/browse/STORM-1837

[GitHub] storm pull request: STORM-1837: Fix complete-topology and prevent ...

2016-05-15 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1417 STORM-1837: Fix complete-topology and prevent message loss when runni… …ng local clusters with no time simulation Link for convenience https://issues.apache.org/jira/browse/STORM-1837

[GitHub] storm pull request: STORM-1756: Explicitly null KafkaServer refere...

2016-05-03 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1387#issuecomment-216488648 @HeartSaVioR I saw OOME on one of my branches, because each KafkaServerStartable allocates a 130MB byte array for the log cleaner thread. A heap dump showed

[GitHub] storm pull request: STORM-1756: Explicitly null KafkaServer refere...

2016-05-03 Thread srdo
Github user srdo closed the pull request at: https://github.com/apache/storm/pull/1387 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm pull request: STORM-1750: Ensure worker dies when report-err...

2016-05-03 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1384#issuecomment-216464027 Hi @HeartSaVioR, Sure, I'll make another pair of PRs soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] storm pull request: STORM-1745: Add partition to log output in Par...

2016-05-03 Thread srdo
Github user srdo closed the pull request at: https://github.com/apache/storm/pull/1380 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm pull request: STORM-1745: Add partition to log output in Par...

2016-05-03 Thread srdo
GitHub user srdo reopened a pull request: https://github.com/apache/storm/pull/1380 STORM-1745: Add partition to log output in PartitionManager Adding topic name and partition id to the log output makes it easier to debug when an issue arises. You can merge this pull request

[GitHub] storm pull request: STORM-1756: Explicitly null KafkaServer refere...

2016-05-03 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1387#issuecomment-216490426 If this turns out to be a problem later, disabling the log cleaner thread during tests is probably a better fix. The tests don't make logs long enough to be compacted

[GitHub] storm pull request: STORM-1750 (0.10.x): Ensure worker dies when r...

2016-05-03 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1390 STORM-1750 (0.10.x): Ensure worker dies when report-error-and-die is … …called. Make cluster set-data try setting data if node creation fails because the node exists. Backport of STORM

[GitHub] storm pull request: STORM-1756: Explicitly null KafkaServer refere...

2016-05-03 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1387#issuecomment-216508276 I did see OOME due to KafkaServer resources. 8x 130MB byte arrays caused the VM to run out of memory. As I said, I can't reproduce it. It's probably not an issue on master

[GitHub] storm pull request: STORM-1756: Explicitly null KafkaServer refere...

2016-05-03 Thread srdo
GitHub user srdo reopened a pull request: https://github.com/apache/storm/pull/1387 STORM-1756: Explicitly null KafkaServer reference in KafkaTestBroker … …to prevent out of memory on large test classes. I'm not clear on whether JUnit keeps the reference

[GitHub] storm pull request: STORM-1745: Add partition to log output in Par...

2016-05-03 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1380#issuecomment-216496822 @HeartSaVioR Sure. The current logs look like this "Starting Kafka 10.4.5.1:0 from offset 15520", which doesn't tell you which topic it's affecting. The lo

[GitHub] storm pull request: STORM-1750: Ensure worker dies when report-err...

2016-05-03 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1389 STORM-1750: Ensure worker dies when report-error-and-die is called. M… …ake zookeeper_state_factory set-data try setting data if node creation fails because the node exists. Backport

[GitHub] storm pull request: STORM-1735: Nimbus should log that replication...

2016-05-03 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1369#issuecomment-216493707 @HeartSaVioR Sure, please do :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] storm pull request: [STORM-1646] Fix Kafka unit tests

2016-05-03 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1335#issuecomment-216573099 I might have been a bit quick on the trigger there. The Kafka coding guidelines target backward compatibility for one release only. So storm-kafka might not support 0.10.0

[GitHub] storm pull request: [STORM-1646] Fix Kafka unit tests

2016-05-03 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1335#issuecomment-216564619 storm-kafka will probably support 0.10.0, I believe the Kafka developers version their APIs so new clusters can talk to old clients. That shouldn't affect the choice

[GitHub] storm pull request: STORM-1756: Explicitly null KafkaServer refere...

2016-05-02 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1387 STORM-1756: Explicitly null KafkaServer reference in KafkaTestBroker … …to prevent out of memory on large test classes. I'm not clear on whether JUnit keeps the reference for the lifetime

[GitHub] storm pull request: STORM-1750: Ensure worker dies when report-err...

2016-05-05 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1384#discussion_r62155412 --- Diff: storm-core/src/jvm/org/apache/storm/cluster/ZKStateStorage.java --- @@ -189,7 +190,15 @@ public void set_data(String path, byte[] data, List acls

[GitHub] storm pull request: STORM-1755: Revert the kafka client version to...

2016-05-06 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1386#issuecomment-217418115 Going by the Kafka coding guidelines (http://kafka.apache.org/coding-guide.html), it looks like client libraries can only be assumed to support two major (0.x) releases

[GitHub] storm pull request: STORM-1713: Replace NotImplementedException wi...

2016-04-15 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1341 STORM-1713: Replace NotImplementedException with UnsupportedOperation… …Exception https://issues.apache.org/jira/browse/STORM-1713 You can merge this pull request into a Git repository

[GitHub] storm pull request: [STORM-1646] Fix Kafka unit tests

2016-04-14 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1335#issuecomment-210042576 There's also a metadata.fetch.timeout.ms here https://github.com/apache/storm/blob/master/external/storm-kafka/src/test/org/apache/storm/kafka/bolt/KafkaBoltTest.java

[GitHub] storm pull request: STORM-1682: Refactor DynamicBrokersReader to l...

2016-04-21 Thread srdo
Github user srdo commented on the pull request: https://github.com/apache/storm/pull/1340#issuecomment-212927945 This runs reliably for me locally, Travis seemed to run into a heap space limit. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] storm pull request: STORM-1682: Refactor DynamicBrokersReader to l...

2016-04-14 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1340 STORM-1682: Refactor DynamicBrokersReader to lookup leader informatio… …n via Kafka metadata API. Put Zookeeper metadata lookup into new class. See https://issues.apache.org/jira/browse

[GitHub] storm pull request #1605: STORM-2014: Put logic around dropping messages int...

2016-08-02 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/1605 STORM-2014: Put logic around dropping messages into RetryService, rem… …ove maxRetry setting from new KafkaSpout https://issues.apache.org/jira/browse/STORM-2014 This PR removes

[GitHub] storm issue #1832: STORM-2250: Kafka Spout Refactoring to Increase Modularit...

2017-02-01 Thread srdo
Github user srdo commented on the issue: https://github.com/apache/storm/pull/1832 @hmcl Updated. This should be ready for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] storm issue #1916: fixed storm-kafka-client test failed

2017-02-03 Thread srdo
Github user srdo commented on the issue: https://github.com/apache/storm/pull/1916 It looks like Fields has an equals() now to do this https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/tuple/Fields.java --- If your project is set up for it, you can

[GitHub] storm pull request #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

2017-02-07 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1919#discussion_r99882134 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -319,8 +320,10 @@ private boolean

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

2017-02-07 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99941130 --- Diff: external/storm-kafka-client/src/test/java/org/apache/storm/kafka/spout/KafkaSpoutEmitTest.java --- @@ -0,0 +1,156 @@ +/* + * Copyright 2017

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

2017-02-07 Thread srdo
Github user srdo commented on the issue: https://github.com/apache/storm/pull/1919 Please see https://github.com/apache/storm/pull/1919#discussion_r99888764 and https://storm.apache.org/releases/1.0.0/javadocs/org/apache/storm/Config.html#TOPOLOGY_ACKER_EXECUTORS. If ack=0 Storm

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

2017-02-07 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99940042 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutRetryExponentialBackoff.java --- @@ -54,59 +56,92

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

2017-02-07 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99950879 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -289,6 +293,13 @@ private void

[GitHub] storm issue #1832: STORM-2250: Kafka Spout Refactoring to Increase Modularit...

2017-02-08 Thread srdo
Github user srdo commented on the issue: https://github.com/apache/storm/pull/1832 @hmcl I gave updating Time to support nanoseconds a shot. Not really sure I'm happy with it, since it requires that we also switch the current millisecond support over to using System.nanoTime

[GitHub] storm pull request #1381: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log...

2017-02-02 Thread srdo
GitHub user srdo reopened a pull request: https://github.com/apache/storm/pull/1381 STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log to warn from error This error is transient most of the time and will occur during normal operation. Downgrading it to warning makes the error log

[GitHub] storm issue #1381: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log to war...

2017-02-02 Thread srdo
Github user srdo commented on the issue: https://github.com/apache/storm/pull/1381 Closing since it seems like this is a fairly irrelevant change to what is now legacy code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] storm pull request #1381: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log...

2017-02-02 Thread srdo
Github user srdo closed the pull request at: https://github.com/apache/storm/pull/1381 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

2017-02-06 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99639454 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java --- @@ -329,11 +328,13 @@ private Builder(Builder

[GitHub] storm issue #1924: STORM-2343: New Kafka spout can stop emitting tuples if m...

2017-02-05 Thread srdo
Github user srdo commented on the issue: https://github.com/apache/storm/pull/1924 I think it would be best if we merged maxPollOffsets and maxUncommittedOffsets, since having them be different has some undesirable side effects. @hmcl do you have an opinion. Also this drops

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

2017-02-06 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99642608 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -240,7 +240,9 @@ private boolean commit

[GitHub] storm pull request #1832: STORM-2250: Kafka Spout Refactoring to Increase Mo...

2017-02-06 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1832#discussion_r99650222 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -58,27 +56,26 @@ public class KafkaSpout&l

  1   2   3   4   5   6   7   8   9   10   >