[GitHub] storm pull request: STORM-558 change swap! to reset! to fix as...
GitHub user xiaokang opened a pull request: https://github.com/apache/storm/pull/315 STORM-558 change swap! to reset! to fix assignment-versions always empty in supervisor This PR is for JIRA STORM-558 to fix #167 . You can merge this pull request into a Git repository by running: $ git pull https://github.com/xiaokang/incubator-storm STORM-558 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/315.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #315 commit 32a7c0b161f833f0bae2fe8ee6515af8f899829b Author: Kang Xiao kxiao.ti...@gmail.com Date: 2014-11-17T07:29:15Z STORM-558 change swap! to reset! to fix assignment-versions always empty in supervisor --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-558) supervisor's assignment-versions memory cache for ZK is always empty
[ https://issues.apache.org/jira/browse/STORM-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214413#comment-14214413 ] ASF GitHub Bot commented on STORM-558: -- GitHub user xiaokang opened a pull request: https://github.com/apache/storm/pull/315 STORM-558 change swap! to reset! to fix assignment-versions always empty in supervisor This PR is for JIRA STORM-558 to fix #167 . You can merge this pull request into a Git repository by running: $ git pull https://github.com/xiaokang/incubator-storm STORM-558 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/315.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #315 commit 32a7c0b161f833f0bae2fe8ee6515af8f899829b Author: Kang Xiao kxiao.ti...@gmail.com Date: 2014-11-17T07:29:15Z STORM-558 change swap! to reset! to fix assignment-versions always empty in supervisor supervisor's assignment-versions memory cache for ZK is always empty Key: STORM-558 URL: https://issues.apache.org/jira/browse/STORM-558 Project: Apache Storm Issue Type: Bug Reporter: Kang Xiao Assignee: Kang Xiao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-375] Smarter downloading of assignments...
Github user xiaokang commented on the pull request: https://github.com/apache/storm/pull/167#issuecomment-63273071 Thanks @nathanmarz for verification. Another PR #315 is opened. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-375) Smarter downloading of assignments by supervisors and workers
[ https://issues.apache.org/jira/browse/STORM-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214414#comment-14214414 ] ASF GitHub Bot commented on STORM-375: -- Github user xiaokang commented on the pull request: https://github.com/apache/storm/pull/167#issuecomment-63273071 Thanks @nathanmarz for verification. Another PR #315 is opened. Smarter downloading of assignments by supervisors and workers - Key: STORM-375 URL: https://issues.apache.org/jira/browse/STORM-375 Project: Apache Storm Issue Type: Improvement Reporter: Kyle Nusbaum Assignee: Kyle Nusbaum Fix For: 0.9.3 Periodically the supervisors and workers each download their assignments from zookeeper, and the traffic begins to saturate the network and hammer zookeeper as we scale. Having these daemons only grab the assignments when the zookeeper path's version number changes reduces this traffic considerably, along with STORM-376. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-558 change swap! to reset! to fix as...
Github user xiaokang commented on the pull request: https://github.com/apache/storm/pull/315#issuecomment-63273340 The unit test is failed BOTH before and after this code change. I'll try to find the problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-558 change swap! to reset! to fix as...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/315#issuecomment-63318696 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-558 change swap! to reset! to fix as...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/315#issuecomment-63320364 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-558) supervisor's assignment-versions memory cache for ZK is always empty
[ https://issues.apache.org/jira/browse/STORM-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214729#comment-14214729 ] ASF GitHub Bot commented on STORM-558: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/315#issuecomment-63320364 +1 supervisor's assignment-versions memory cache for ZK is always empty Key: STORM-558 URL: https://issues.apache.org/jira/browse/STORM-558 Project: Apache Storm Issue Type: Bug Reporter: Kang Xiao Assignee: Kang Xiao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Release Apache Storm 0.9.3
Even if we cannot reproduce it in all cases, the change was rather small not totally necessary. +1 for reverting. - Bobby On Sunday, November 16, 2014 2:23 PM, Harsha st...@harsha.io wrote: I am able to reproduce this using Sean's topology. I am +1 on reverting this. Thanks, Harsha On Fri, Nov 14, 2014, at 06:43 PM, P. Taylor Goetz wrote: I agree completely. However, I've been unable to reproduce the issue. I think we need to get more eyes on this. If anyone has the resources to do so please do. I'm +1 for reverting, but I'd also like to have a root cause. -Taylor On Nov 14, 2014, at 9:10 PM, 임정택 kabh...@gmail.com wrote: Agree with Nathan. Seems like it doesn't prepared because we should take care of STORM-350. 2014-11-15 10:54 GMT+09:00 Nathan Marz nat...@nathanmarz.com: -1. Looking at https://issues.apache.org/jira/browse/STORM-350 it seems the upgrade to disruptor caused message loss issues. That upgrade should be reverted, or I'd like @clockfly to provide more insight. On Fri, Nov 14, 2014 at 5:19 PM, Harsha st...@harsha.io wrote: I am +1 releasing 0.9.3 +1 on including STORM-555. Thanks, Harsha On Fri, Nov 14, 2014, at 02:29 PM, P. Taylor Goetz wrote: I’d like to get the community’s opinion on releasing 0.9.3 with what is currently in the 0.9.3 branch. This would be the official release (skipping the unofficial rc2). The only addition I’d like to include is STORM-555, which should be eligible for merging early next week. Thoughts? -Taylor Email had 1 attachment: + signature.asc 1k (application/pgp-signature) -- Twitter: @nathanmarz http://nathanmarz.com -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
[GitHub] storm pull request: [STORM-557] Created docs directory and added i...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/314#issuecomment-63332256 I was curious if we wanted to keep a 5MB file in our github repo that everyone must download to get storm? As it stands now the git repo is about 11MB total. This could slow things down for some developers. It might be better to put it in the SVN repo with the web page source http://svn.apache.org/viewvc/storm/ But it would make it less accessible to developers who might want to reuse it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-557) High Quality Images for presentations, etc.
[ https://issues.apache.org/jira/browse/STORM-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214810#comment-14214810 ] ASF GitHub Bot commented on STORM-557: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/314#issuecomment-63332256 I was curious if we wanted to keep a 5MB file in our github repo that everyone must download to get storm? As it stands now the git repo is about 11MB total. This could slow things down for some developers. It might be better to put it in the SVN repo with the web page source http://svn.apache.org/viewvc/storm/ But it would make it less accessible to developers who might want to reuse it. High Quality Images for presentations, etc. --- Key: STORM-557 URL: https://issues.apache.org/jira/browse/STORM-557 Project: Apache Storm Issue Type: Documentation Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Recently I created a couple of svg diagrams for a poster I was doing about secure storm. I thought it was a complete waste to not release them as open source, but I wasn't totally sure where we wanted to keep them. I can check them into git, because that would make it simple for others to find, and we could link to them from the markdown documentation. But I could also put them in the svn repo for the official site documentation. I'll throw up a very basic pull request and see what people think. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-557] Created docs directory and added i...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/314#issuecomment-63336734 @revans2 There is another JIRA asking for the current storm.apache.org docs to be moved onto github. https://issues.apache.org/jira/browse/STORM-544. As @HeartSaVioR pointed out about redis-doc https://github.com/antirez/redis-doc. I am +1 on moving all storm related doc onto its own github repo. Makes it easier for community to contribute to the docs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-557) High Quality Images for presentations, etc.
[ https://issues.apache.org/jira/browse/STORM-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214838#comment-14214838 ] ASF GitHub Bot commented on STORM-557: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/314#issuecomment-63336734 @revans2 There is another JIRA asking for the current storm.apache.org docs to be moved onto github. https://issues.apache.org/jira/browse/STORM-544. As @HeartSaVioR pointed out about redis-doc https://github.com/antirez/redis-doc. I am +1 on moving all storm related doc onto its own github repo. Makes it easier for community to contribute to the docs. High Quality Images for presentations, etc. --- Key: STORM-557 URL: https://issues.apache.org/jira/browse/STORM-557 Project: Apache Storm Issue Type: Documentation Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Recently I created a couple of svg diagrams for a poster I was doing about secure storm. I thought it was a complete waste to not release them as open source, but I wasn't totally sure where we wanted to keep them. I can check them into git, because that would make it simple for others to find, and we could link to them from the markdown documentation. But I could also put them in the svn repo for the official site documentation. I'll throw up a very basic pull request and see what people think. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-557] Created docs directory and added i...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/314#issuecomment-63337668 OK so the next question I have is, should we split the one big SVG file up along side rasterized PNG equivalents for use in the docs? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-557) High Quality Images for presentations, etc.
[ https://issues.apache.org/jira/browse/STORM-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214851#comment-14214851 ] ASF GitHub Bot commented on STORM-557: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/314#issuecomment-63337668 OK so the next question I have is, should we split the one big SVG file up along side rasterized PNG equivalents for use in the docs? High Quality Images for presentations, etc. --- Key: STORM-557 URL: https://issues.apache.org/jira/browse/STORM-557 Project: Apache Storm Issue Type: Documentation Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Recently I created a couple of svg diagrams for a poster I was doing about secure storm. I thought it was a complete waste to not release them as open source, but I wasn't totally sure where we wanted to keep them. I can check them into git, because that would make it simple for others to find, and we could link to them from the markdown documentation. But I could also put them in the svn repo for the official site documentation. I'll throw up a very basic pull request and see what people think. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-557] Created docs directory and added i...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/314#issuecomment-63338507 @revans2 sounds like a good idea to provide PNG files. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-558) supervisor's assignment-versions memory cache for ZK is always empty
[ https://issues.apache.org/jira/browse/STORM-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214898#comment-14214898 ] ASF GitHub Bot commented on STORM-558: -- Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/315#issuecomment-63344283 +1 supervisor's assignment-versions memory cache for ZK is always empty Key: STORM-558 URL: https://issues.apache.org/jira/browse/STORM-558 Project: Apache Storm Issue Type: Bug Reporter: Kang Xiao Assignee: Kang Xiao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-512) KafkaBolt doesn't handle ticks properly
[ https://issues.apache.org/jira/browse/STORM-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214916#comment-14214916 ] ASF GitHub Bot commented on STORM-512: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/275#issuecomment-63347222 The changes look fine to me. +1, @d2r in this case the bolt is taking its input and sending it to a kafka queue. I don't think we want to send ticks to the kafka queue, as the ticks are internal to storm, and will not have the key/value that all other tuples have. KafkaBolt doesn't handle ticks properly --- Key: STORM-512 URL: https://issues.apache.org/jira/browse/STORM-512 Project: Apache Storm Issue Type: Bug Reporter: Niels Basjes I found that when using the KafkaBolt the tick tuples are not handled properly. They should be ignored and the reality is that they are not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-558) supervisor's assignment-versions memory cache for ZK is always empty
[ https://issues.apache.org/jira/browse/STORM-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214920#comment-14214920 ] ASF GitHub Bot commented on STORM-558: -- Github user nathanmarz commented on the pull request: https://github.com/apache/storm/pull/315#issuecomment-63348326 +1 supervisor's assignment-versions memory cache for ZK is always empty Key: STORM-558 URL: https://issues.apache.org/jira/browse/STORM-558 Project: Apache Storm Issue Type: Bug Reporter: Kang Xiao Assignee: Kang Xiao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-512) KafkaBolt doesn't handle ticks properly
[ https://issues.apache.org/jira/browse/STORM-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214936#comment-14214936 ] ASF GitHub Bot commented on STORM-512: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/275#issuecomment-63349586 OK I am wrong. When I tried to compile with this change the following tests failed. ``` Failed tests: shouldEmitSomethingIfTickTupleIsReceived(storm.starter.bolt.IntermediateRankingsBoltTest): Index: 0, Size: 0 shouldEmitNothingIfNoObjectHasBeenCountedYetAndTickTupleIsReceived(storm.starter.bolt.RollingCountBoltTest): shouldEmitSomethingIfAtLeastOneObjectWasCountedAndTickTupleIsReceived(storm.starter.bolt.RollingCountBoltTest): shouldEmitSomethingIfTickTupleIsReceived(storm.starter.bolt.TotalRankingsBoltTest) ``` It looks like to me that the tick is being used somewhere else. Why exactly do we not want the ticks? KafkaBolt doesn't handle ticks properly --- Key: STORM-512 URL: https://issues.apache.org/jira/browse/STORM-512 Project: Apache Storm Issue Type: Bug Reporter: Niels Basjes I found that when using the KafkaBolt the tick tuples are not handled properly. They should be ignored and the reality is that they are not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
acking and offsets
Hello All, I am using kafka spout that comes with storm 0.9.3 ( https://github.com/apache/storm). I am having several different bolts consuming the same tuples from the spout (in the same topology). These bolts process the tuples and send the output to different destinations. I have a couple of basic questions: 1) What happens if the destination for some of the bolts is not accessible? I can have a mechanism in each bolt that returns without consuming any tuple if the destination is not available. But what happens to the tuples at hand and when is the next execute is invoked. - Do I need to have the destination availability check in a loop? - Do I need to keep track of the offset? 2) Is it possible to ack in the bolts. thanks, Clay
[GitHub] storm pull request: [STORM-557] Created docs directory and added i...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/314#issuecomment-63354593 OK I split them apart, cleaned up one of the SVG images to make it smaller, and added in some PNG versions, along with updating the security document to use one of the PNG images. So now the total new size is 3.6 MB. Hopefully a lot more people will find these useful. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-512) KafkaBolt doesn't handle ticks properly
[ https://issues.apache.org/jira/browse/STORM-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214986#comment-14214986 ] ASF GitHub Bot commented on STORM-512: -- Github user nielsbasjes commented on the pull request: https://github.com/apache/storm/pull/275#issuecomment-63355100 I'll have a look into why these tests failed. Regarding the 'why': A Tick is a message between the Storm framework and a bolt that runs within the framework. Kafka bolt should handle the tick any way suitable, but most importantly is must NOT persist the tick into Kafka. I ran into the scenario that a non-Storm consumer of this Kafka topic gave errors because it got a message (= a Tick) that did not adhere to the agreed interface (i.e. I got deserialization exceptions). So that's why. KafkaBolt doesn't handle ticks properly --- Key: STORM-512 URL: https://issues.apache.org/jira/browse/STORM-512 Project: Apache Storm Issue Type: Bug Reporter: Niels Basjes I found that when using the KafkaBolt the tick tuples are not handled properly. They should be ignored and the reality is that they are not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-512) KafkaBolt doesn't handle ticks properly
[ https://issues.apache.org/jira/browse/STORM-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215010#comment-14215010 ] ASF GitHub Bot commented on STORM-512: -- Github user nielsbasjes commented on the pull request: https://github.com/apache/storm/pull/275#issuecomment-63358387 I think you are right. It seems like the assumption that was made during the development of that test is not 100% accurate in all scenarios. KafkaBolt doesn't handle ticks properly --- Key: STORM-512 URL: https://issues.apache.org/jira/browse/STORM-512 Project: Apache Storm Issue Type: Bug Reporter: Niels Basjes I found that when using the KafkaBolt the tick tuples are not handled properly. They should be ignored and the reality is that they are not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Storm555: Storm json response should set chars...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/313#issuecomment-63367628 Thanks @Parth-Brahmbhatt. I merged this into the master and 0.9.3 branches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-558 change swap! to reset! to fix as...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/315#issuecomment-63373190 Thanks @xiaokang. I merged this into the master and 0.9.3 branches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (STORM-558) supervisor's assignment-versions memory cache for ZK is always empty
[ https://issues.apache.org/jira/browse/STORM-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] P. Taylor Goetz resolved STORM-558. --- Resolution: Fixed Fix Version/s: 0.9.3 supervisor's assignment-versions memory cache for ZK is always empty Key: STORM-558 URL: https://issues.apache.org/jira/browse/STORM-558 Project: Apache Storm Issue Type: Bug Reporter: Kang Xiao Assignee: Kang Xiao Fix For: 0.9.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-558 change swap! to reset! to fix as...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/315 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [DISCUSS] Release Apache Storm 0.9.3
Okay, I’ve reverted the commit in the 0.9.3 branch. I did not revert it in master. For master we can either choose to revert it there as well, or leave it and continue to look for a root cause. I’m open to either and would like to hear what other people think. In addition to the revert of STORM-350, I also merged the following to the master and 0.9.3 branches: * STORM-558 change swap! to reset! to fix assignment-versions in supervisor * STORM-555: Storm json response should set charset to UTF-8 What are everyone’s thoughts on proceeding with a 0.9.3 release? -Taylor On Nov 17, 2014, at 11:09 AM, Bobby Evans ev...@yahoo-inc.com.INVALID wrote: Even if we cannot reproduce it in all cases, the change was rather small not totally necessary. +1 for reverting. - Bobby On Sunday, November 16, 2014 2:23 PM, Harsha st...@harsha.io wrote: I am able to reproduce this using Sean's topology. I am +1 on reverting this. Thanks, Harsha On Fri, Nov 14, 2014, at 06:43 PM, P. Taylor Goetz wrote: I agree completely. However, I've been unable to reproduce the issue. I think we need to get more eyes on this. If anyone has the resources to do so please do. I'm +1 for reverting, but I'd also like to have a root cause. -Taylor On Nov 14, 2014, at 9:10 PM, 임정택 kabh...@gmail.com wrote: Agree with Nathan. Seems like it doesn't prepared because we should take care of STORM-350. 2014-11-15 10:54 GMT+09:00 Nathan Marz nat...@nathanmarz.com: -1. Looking at https://issues.apache.org/jira/browse/STORM-350 it seems the upgrade to disruptor caused message loss issues. That upgrade should be reverted, or I'd like @clockfly to provide more insight. On Fri, Nov 14, 2014 at 5:19 PM, Harsha st...@harsha.io wrote: I am +1 releasing 0.9.3 +1 on including STORM-555. Thanks, Harsha On Fri, Nov 14, 2014, at 02:29 PM, P. Taylor Goetz wrote: I’d like to get the community’s opinion on releasing 0.9.3 with what is currently in the 0.9.3 branch. This would be the official release (skipping the unofficial rc2). The only addition I’d like to include is STORM-555, which should be eligible for merging early next week. Thoughts? -Taylor Email had 1 attachment: + signature.asc 1k (application/pgp-signature) -- Twitter: @nathanmarz http://nathanmarz.com -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior signature.asc Description: Message signed with OpenPGP using GPGMail
[jira] [Commented] (STORM-558) supervisor's assignment-versions memory cache for ZK is always empty
[ https://issues.apache.org/jira/browse/STORM-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215163#comment-14215163 ] ASF GitHub Bot commented on STORM-558: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/315 supervisor's assignment-versions memory cache for ZK is always empty Key: STORM-558 URL: https://issues.apache.org/jira/browse/STORM-558 Project: Apache Storm Issue Type: Bug Reporter: Kang Xiao Assignee: Kang Xiao Fix For: 0.9.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Release Apache Storm 0.9.3
Thanks for the input Bobby and Nathan. I’ll revert it there as well unless anyone proposes a compelling reason not to. What are your thoughts on moving forward with the 0.9.3 release? -Taylor On Nov 17, 2014, at 4:45 PM, Nathan Marz nat...@nathanmarz.com wrote: I agree, it should be reverted on master until we can figure out what's going on. On Mon, Nov 17, 2014 at 1:33 PM, Bobby Evans ev...@yahoo-inc.com.invalid wrote: I would prefer to revert it on master too, just so that if anyone decides to use master storm, they have a chance of getting something that, at least as far as we know, works. - Bobby On Monday, November 17, 2014 2:58 PM, P. Taylor Goetz ptgo...@gmail.com wrote: Okay, I’ve reverted the commit in the 0.9.3 branch. I did not revert it in master. For master we can either choose to revert it there as well, or leave it and continue to look for a root cause. I’m open to either and would like to hear what other people think. In addition to the revert of STORM-350, I also merged the following to the master and 0.9.3 branches: * STORM-558 change swap! to reset! to fix assignment-versions in supervisor * STORM-555: Storm json response should set charset to UTF-8 What are everyone’s thoughts on proceeding with a 0.9.3 release? -Taylor On Nov 17, 2014, at 11:09 AM, Bobby Evans ev...@yahoo-inc.com.INVALID wrote: Even if we cannot reproduce it in all cases, the change was rather small not totally necessary. +1 for reverting. - Bobby On Sunday, November 16, 2014 2:23 PM, Harsha st...@harsha.io wrote: I am able to reproduce this using Sean's topology. I am +1 on reverting this. Thanks, Harsha On Fri, Nov 14, 2014, at 06:43 PM, P. Taylor Goetz wrote: I agree completely. However, I've been unable to reproduce the issue. I think we need to get more eyes on this. If anyone has the resources to do so please do. I'm +1 for reverting, but I'd also like to have a root cause. -Taylor On Nov 14, 2014, at 9:10 PM, 임정택 kabh...@gmail.com wrote: Agree with Nathan. Seems like it doesn't prepared because we should take care of STORM-350. 2014-11-15 10:54 GMT+09:00 Nathan Marz nat...@nathanmarz.com: -1. Looking at https://issues.apache.org/jira/browse/STORM-350 it seems the upgrade to disruptor caused message loss issues. That upgrade should be reverted, or I'd like @clockfly to provide more insight. On Fri, Nov 14, 2014 at 5:19 PM, Harsha st...@harsha.io wrote: I am +1 releasing 0.9.3 +1 on including STORM-555. Thanks, Harsha On Fri, Nov 14, 2014, at 02:29 PM, P. Taylor Goetz wrote: I’d like to get the community’s opinion on releasing 0.9.3 with what is currently in the 0.9.3 branch. This would be the official release (skipping the unofficial rc2). The only addition I’d like to include is STORM-555, which should be eligible for merging early next week. Thoughts? -Taylor Email had 1 attachment: + signature.asc 1k (application/pgp-signature) -- Twitter: @nathanmarz http://nathanmarz.com -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior -- Twitter: @nathanmarz http://nathanmarz.com signature.asc Description: Message signed with OpenPGP using GPGMail
[jira] [Updated] (STORM-350) Update disruptor to latest version (3.2.1)
[ https://issues.apache.org/jira/browse/STORM-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated STORM-350: - Fix Version/s: 0.10.0 Update disruptor to latest version (3.2.1) -- Key: STORM-350 URL: https://issues.apache.org/jira/browse/STORM-350 Project: Apache Storm Issue Type: Dependency upgrade Reporter: Boris Aksenov Fix For: 0.10.0 Attachments: 20141117-0.9.3-rc1-3-worker-separate-1-spout-and-2-bolts-failing-tuples.png, 20141117-0.9.3-rc1-one-worker-failing-tuples.png, 20141117-0.9.3-rc1-three-workers-1-spout-3-bolts-failing-tuples.png, 20141118-0.9.3-branch-3-worker-separate-1-spout-and-2-bolts-ok.png, 20141118-0.9.3-branch-one-worker-ok.png, 20141118-0.9.3-branch-three-workers-1-spout-3-bolts-ok.png, Storm UI1.pdf, Storm UI2.pdf, storm-0.9.3-rc1-failing-tuples.png, storm-0_9_2-incubating-failing-tuples.png, storm-0_9_2-incubating-no-failing-tuples.png, storm-failed-tuples-multi-node.png, storm-multi-node-without-350.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-350) Update disruptor to latest version (3.2.1)
[ https://issues.apache.org/jira/browse/STORM-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated STORM-350: - Fix Version/s: (was: 0.9.3) Update disruptor to latest version (3.2.1) -- Key: STORM-350 URL: https://issues.apache.org/jira/browse/STORM-350 Project: Apache Storm Issue Type: Dependency upgrade Reporter: Boris Aksenov Fix For: 0.10.0 Attachments: 20141117-0.9.3-rc1-3-worker-separate-1-spout-and-2-bolts-failing-tuples.png, 20141117-0.9.3-rc1-one-worker-failing-tuples.png, 20141117-0.9.3-rc1-three-workers-1-spout-3-bolts-failing-tuples.png, 20141118-0.9.3-branch-3-worker-separate-1-spout-and-2-bolts-ok.png, 20141118-0.9.3-branch-one-worker-ok.png, 20141118-0.9.3-branch-three-workers-1-spout-3-bolts-ok.png, Storm UI1.pdf, Storm UI2.pdf, storm-0.9.3-rc1-failing-tuples.png, storm-0_9_2-incubating-failing-tuples.png, storm-0_9_2-incubating-no-failing-tuples.png, storm-failed-tuples-multi-node.png, storm-multi-node-without-350.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-350) Update disruptor to latest version (3.2.1)
[ https://issues.apache.org/jira/browse/STORM-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated STORM-350: - Issue Type: Bug (was: Dependency upgrade) Update disruptor to latest version (3.2.1) -- Key: STORM-350 URL: https://issues.apache.org/jira/browse/STORM-350 Project: Apache Storm Issue Type: Bug Reporter: Boris Aksenov Assignee: Boris Aksenov Fix For: 0.10.0 Attachments: 20141117-0.9.3-rc1-3-worker-separate-1-spout-and-2-bolts-failing-tuples.png, 20141117-0.9.3-rc1-one-worker-failing-tuples.png, 20141117-0.9.3-rc1-three-workers-1-spout-3-bolts-failing-tuples.png, 20141118-0.9.3-branch-3-worker-separate-1-spout-and-2-bolts-ok.png, 20141118-0.9.3-branch-one-worker-ok.png, 20141118-0.9.3-branch-three-workers-1-spout-3-bolts-ok.png, Storm UI1.pdf, Storm UI2.pdf, storm-0.9.3-rc1-failing-tuples.png, storm-0_9_2-incubating-failing-tuples.png, storm-0_9_2-incubating-no-failing-tuples.png, storm-failed-tuples-multi-node.png, storm-multi-node-without-350.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (STORM-350) Update disruptor to latest version (3.2.1)
[ https://issues.apache.org/jira/browse/STORM-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong reopened STORM-350: -- Assignee: (was: Boris Aksenov) Update disruptor to latest version (3.2.1) -- Key: STORM-350 URL: https://issues.apache.org/jira/browse/STORM-350 Project: Apache Storm Issue Type: Dependency upgrade Reporter: Boris Aksenov Fix For: 0.9.3 Attachments: 20141117-0.9.3-rc1-3-worker-separate-1-spout-and-2-bolts-failing-tuples.png, 20141117-0.9.3-rc1-one-worker-failing-tuples.png, 20141117-0.9.3-rc1-three-workers-1-spout-3-bolts-failing-tuples.png, 20141118-0.9.3-branch-3-worker-separate-1-spout-and-2-bolts-ok.png, 20141118-0.9.3-branch-one-worker-ok.png, 20141118-0.9.3-branch-three-workers-1-spout-3-bolts-ok.png, Storm UI1.pdf, Storm UI2.pdf, storm-0.9.3-rc1-failing-tuples.png, storm-0_9_2-incubating-failing-tuples.png, storm-0_9_2-incubating-no-failing-tuples.png, storm-failed-tuples-multi-node.png, storm-multi-node-without-350.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-350) Update disruptor to latest version (3.2.1)
[ https://issues.apache.org/jira/browse/STORM-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated STORM-350: - Assignee: Boris Aksenov Update disruptor to latest version (3.2.1) -- Key: STORM-350 URL: https://issues.apache.org/jira/browse/STORM-350 Project: Apache Storm Issue Type: Dependency upgrade Reporter: Boris Aksenov Assignee: Boris Aksenov Fix For: 0.10.0 Attachments: 20141117-0.9.3-rc1-3-worker-separate-1-spout-and-2-bolts-failing-tuples.png, 20141117-0.9.3-rc1-one-worker-failing-tuples.png, 20141117-0.9.3-rc1-three-workers-1-spout-3-bolts-failing-tuples.png, 20141118-0.9.3-branch-3-worker-separate-1-spout-and-2-bolts-ok.png, 20141118-0.9.3-branch-one-worker-ok.png, 20141118-0.9.3-branch-three-workers-1-spout-3-bolts-ok.png, Storm UI1.pdf, Storm UI2.pdf, storm-0.9.3-rc1-failing-tuples.png, storm-0_9_2-incubating-failing-tuples.png, storm-0_9_2-incubating-no-failing-tuples.png, storm-failed-tuples-multi-node.png, storm-multi-node-without-350.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-329 : buffer message in client and recon...
Github user tedxia commented on the pull request: https://github.com/apache/storm/pull/268#issuecomment-63424599 @ptgoetz I have merge storm/master to this patch and all tests pass except storm-core/test/clj/backtype/storm/multilang_test.clj, test results are(after I removed multilang_test.clj): ``` [INFO] Reactor Summary: [INFO] [INFO] Storm . SUCCESS [1.340s] [INFO] maven-shade-clojure-transformer ... SUCCESS [1.712s] [INFO] Storm Core SUCCESS [6:04.103s] [INFO] storm-starter . SUCCESS [7.010s] [INFO] storm-kafka ... SUCCESS [1:03.183s] [INFO] storm-hdfs SUCCESS [2.056s] [INFO] storm-hbase ... SUCCESS [2.186s] [INFO] Storm Binary Distribution . SUCCESS [0.185s] [INFO] Storm Source Distribution . SUCCESS [0.136s] ``` When I run multilang_test.clj, I got exception like this: ``` java.lang.Exception: Shell Process Exception: Exception in bolt: \xE4 on US-ASCII - /usr/lib/ruby/1.9.1/json/common.rb:148:in `encode'\n/usr/lib/ruby/1.9.1/json/common.rb:148:in `initialize'\n/usr/lib/ruby/1.9.1/json/common.rb:148:in `new'\n/usr/lib/ruby/1.9.1/json/common.rb:148:in `parse'\n/tmp/81b49de0-4ee0-493a-afe0-6286e393fb14/supervisor/stormdist/test-1-1416288370/resources/storm.rb:39:in `read_message'\n/tmp/81b49de0-4ee0-493a-afe0-6286e393fb14/supervisor/stormdist/test-1-1416288370/resources/storm.rb:57:in `read_command'\n/tmp/81b49de0-4ee0-493a-afe0-6286e393fb14/supervisor/stormdist/test-1-1416288370/resources/storm.rb:190:in `run'\ntester_bolt.rb:37:in `main' at backtype.storm.task.ShellBolt.handleError(ShellBolt.java:188) [classes/:na] at backtype.storm.task.ShellBolt.access$1100(ShellBolt.java:69) [classes/:na] at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:331) [classes/:na] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_37] 124526 [Thread-1209] ERROR backtype.storm.task.ShellBolt - Halting process: ShellBolt died. java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read. Serializer Exception: at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:101) ~[classes/:na] at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) ~[classes/:na] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_37] 124527 [Thread-1209] ERROR backtype.storm.daemon.executor - java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read. Serializer Exception: at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:101) ~[classes/:na] at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) ~[classes/:na] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_37] ``` When I run multilang_test.clj at storm/master, I got the same exception, I think this may be my personal environment problem, can you merge this to 0.9.3 branch and run tests again, thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-329) Add Option to Config Message handling strategy when connection timeout
[
https://issues.apache.org/jira/browse/STORM-329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215731#comment-14215731
]
ASF GitHub Bot commented on STORM-329:
--
Github user tedxia commented on the pull request:
https://github.com/apache/storm/pull/268#issuecomment-63424599
@ptgoetz I have merge storm/master to this patch and all tests pass except
storm-core/test/clj/backtype/storm/multilang_test.clj, test results are(after
I removed multilang_test.clj):
```
[INFO] Reactor Summary:
[INFO]
[INFO] Storm . SUCCESS [1.340s]
[INFO] maven-shade-clojure-transformer ... SUCCESS [1.712s]
[INFO] Storm Core SUCCESS
[6:04.103s]
[INFO] storm-starter . SUCCESS [7.010s]
[INFO] storm-kafka ... SUCCESS
[1:03.183s]
[INFO] storm-hdfs SUCCESS [2.056s]
[INFO] storm-hbase ... SUCCESS [2.186s]
[INFO] Storm Binary Distribution . SUCCESS [0.185s]
[INFO] Storm Source Distribution . SUCCESS [0.136s]
```
When I run multilang_test.clj, I got exception like this:
```
java.lang.Exception: Shell Process Exception: Exception in bolt: \xE4 on
US-ASCII - /usr/lib/ruby/1.9.1/json/common.rb:148:in
`encode'\n/usr/lib/ruby/1.9.1/json/common.rb:148:in
`initialize'\n/usr/lib/ruby/1.9.1/json/common.rb:148:in
`new'\n/usr/lib/ruby/1.9.1/json/common.rb:148:in
`parse'\n/tmp/81b49de0-4ee0-493a-afe0-6286e393fb14/supervisor/stormdist/test-1-1416288370/resources/storm.rb:39:in
`read_message'\n/tmp/81b49de0-4ee0-493a-afe0-6286e393fb14/supervisor/stormdist/test-1-1416288370/resources/storm.rb:57:in
`read_command'\n/tmp/81b49de0-4ee0-493a-afe0-6286e393fb14/supervisor/stormdist/test-1-1416288370/resources/storm.rb:190:in
`run'\ntester_bolt.rb:37:in `main'
at backtype.storm.task.ShellBolt.handleError(ShellBolt.java:188)
[classes/:na]
at backtype.storm.task.ShellBolt.access$1100(ShellBolt.java:69)
[classes/:na]
at
backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:331)
[classes/:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_37]
124526 [Thread-1209] ERROR backtype.storm.task.ShellBolt - Halting process:
ShellBolt died.
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException:
Pipe to subprocess seems to be broken! No output read.
Serializer Exception:
at
backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:101)
~[classes/:na]
at
backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318)
~[classes/:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_37]
124527 [Thread-1209] ERROR backtype.storm.daemon.executor -
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException:
Pipe to subprocess seems to be broken! No output read.
Serializer Exception:
at
backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:101)
~[classes/:na]
at
backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318)
~[classes/:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_37]
```
When I run multilang_test.clj at storm/master, I got the same exception, I
think this may be my personal environment problem, can you merge this to 0.9.3
branch and run tests again, thanks a lot.
Add Option to Config Message handling strategy when connection timeout
--
Key: STORM-329
URL: https://issues.apache.org/jira/browse/STORM-329
Project: Apache Storm
Issue Type: Improvement
Affects Versions: 0.9.2-incubating
Reporter: Sean Zhong
Priority: Minor
Labels: Netty
Fix For: 0.9.3-rc2
Attachments: storm-329.patch, worker-kill-recover3.jpg
This is to address a [concern brought
up|https://github.com/apache/incubator-storm/pull/103#issuecomment-43632986]
during the work at STORM-297:
{quote}
[~revans2] wrote: Your logic makes since to me on why these calls are
blocking. My biggest concern around the blocking is in the case of a worker
crashing. If a single worker crashes this can block the entire topology from
executing until that worker comes back up. In some cases I can see that being
something that you would want. In other cases I can see speed being the
primary concern and some users would like to get partial data fast, rather
then accurate data later.
Could we make it configurable on a follow up JIRA where we can have a max
limit
[GitHub] storm pull request: STORM-329 : buffer message in client and recon...
Github user tedxia commented on the pull request: https://github.com/apache/storm/pull/268#issuecomment-63430061 ``` I tested this by applying it to the 0.9.3 branch and found problems with the unit tests (never-ending cycle of zookeeper reconnects, tests never complete). ``` @ptgoetz Can you figure out which test cause this, I will check it again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-188: Allow user to specifiy full configu...
Github user clockfly commented on the pull request: https://github.com/apache/storm/pull/120#issuecomment-63431309 The original behavior of findAndReadConfigFile() is to locate config file on classpath. findResources(name) will not be empty when name exists on classpath. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
