Stormprocessor StormCV (Storm + OpenCV)
Hi Storm dev's, A couple of colleagues and myself have developed a computer vision specific 'extension' of Storm called StormCV which can be used to create large scale video analysis topologies. It includes its own model, serializers and supports the use of the OpenCV library for image enhancement/analysis. For more information and the first release of the library please visit https://github.com/sensorstorm/StormCV It would be great if you can make the StormCV project part of stormprocessor or otherwise link to it to improve its visibility. Any feedback on the project is welcome as well! Kind regards, Corne Versloot
Re: Stormprocessor StormCV (Storm + OpenCV)
Hi, Quite interested in this project. do we need to explore OpenCV to run the examples ? On Tue, Dec 23, 2014 at 3:35 PM, Corné Versloot corne.versl...@gmail.com wrote: Hi Storm dev's, A couple of colleagues and myself have developed a computer vision specific 'extension' of Storm called StormCV which can be used to create large scale video analysis topologies. It includes its own model, serializers and supports the use of the OpenCV library for image enhancement/analysis. For more information and the first release of the library please visit https://github.com/sensorstorm/StormCV It would be great if you can make the StormCV project part of stormprocessor or otherwise link to it to improve its visibility. Any feedback on the project is welcome as well! Kind regards, Corne Versloot
Re: Stormprocessor StormCV (Storm + OpenCV)
You could add yourself to the powered by page http://storm.apache.org/documentation/Powered-By.html We recently moved the documentation source into git so you can fork storm and update https://github.com/apache/storm/blob/master/docs/documentation/Powered-By.md Then put up a pull request for the change. If you want to update the site to have a new section for extensions/libraries for storm I think that would be fine too. - Bobby On Tuesday, December 23, 2014 5:27 AM, padma priya chitturi padmapriy...@gmail.com wrote: Hi, Quite interested in this project. do we need to explore OpenCV to run the examples ? On Tue, Dec 23, 2014 at 3:35 PM, Corné Versloot corne.versl...@gmail.com wrote: Hi Storm dev's, A couple of colleagues and myself have developed a computer vision specific 'extension' of Storm called StormCV which can be used to create large scale video analysis topologies. It includes its own model, serializers and supports the use of the OpenCV library for image enhancement/analysis. For more information and the first release of the library please visit https://github.com/sensorstorm/StormCV It would be great if you can make the StormCV project part of stormprocessor or otherwise link to it to improve its visibility. Any feedback on the project is welcome as well! Kind regards, Corne Versloot
[jira] [Created] (STORM-601) Have jira-github-join ignore case in matting JIRA number
Robert Joseph Evans created STORM-601: - Summary: Have jira-github-join ignore case in matting JIRA number Key: STORM-601 URL: https://issues.apache.org/jira/browse/STORM-601 Project: Apache Storm Issue Type: Improvement Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Just a little change to make the pull request script a little more user friendly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-601: Make jira-github-join ignore case.
GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/358 STORM-601: Make jira-github-join ignore case. You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/incubator-storm STORM-601 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/358.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #358 commit 3ad05ca3565f0d33693e7bff6a5af6fd20afce46 Author: Robert (Bobby) Evans ev...@yahoo-inc.com Date: 2014-12-23T15:29:36Z STORM-601: Make jira-github-join ignore case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-601) Have jira-github-join ignore case in matting JIRA number
[ https://issues.apache.org/jira/browse/STORM-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257087#comment-14257087 ] ASF GitHub Bot commented on STORM-601: -- GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/358 STORM-601: Make jira-github-join ignore case. You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/incubator-storm STORM-601 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/358.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #358 commit 3ad05ca3565f0d33693e7bff6a5af6fd20afce46 Author: Robert (Bobby) Evans ev...@yahoo-inc.com Date: 2014-12-23T15:29:36Z STORM-601: Make jira-github-join ignore case. Have jira-github-join ignore case in matting JIRA number Key: STORM-601 URL: https://issues.apache.org/jira/browse/STORM-601 Project: Apache Storm Issue Type: Improvement Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Just a little change to make the pull request script a little more user friendly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-544 Fix outdated documents
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/340#issuecomment-67962843 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-544) Some documents from website are outdated
[ https://issues.apache.org/jira/browse/STORM-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257100#comment-14257100 ] Robert Joseph Evans commented on STORM-544: --- I just merged this into the master branch, but I am going to try and update the web page itself. I think I have permissions to do it. Thanks for finding and fixing these issues. Some documents from website are outdated Key: STORM-544 URL: https://issues.apache.org/jira/browse/STORM-544 Project: Apache Storm Issue Type: Documentation Reporter: Jungtaek Lim Assignee: Jungtaek Lim Actually I've mailed this to Storm Dev mailing list, but nobody cares so I repost it to JIRA. I found that some documents are outdated. 1. storm-starter link from all (or most? I don't know) documents refers to https://github.com/nathanmarz/storm-starter, it should be https://github.com/apache/storm/tree/master/examples/storm-starter . 2. It seems that Storm project (including storm-starter) no longer uses Leiningen, right? It it does, some documents should reflect that change. - https://storm.apache.org/documentation/Creating-a-new-Storm-project.html says that storm project uses Leiningen yet. 3. https://storm.apache.org/documentation/Maven.html contains outdated contents. - It should reflect recent stable version of groupId and artifactId, and version. - Would it better to remove building user's project with Leiningen because Storm project doesn't use Leiningen? - Developing Storm section is completely outdated. 4. https://storm.apache.org/documentation/Maven.html Markdown for code block is not working. xml tag is removed so we can't understand what it means. 5. Some repository links refer to incubator-storm, not storm. 6. multilang documents are outdated a. Explanation for adapters is outdated. Storm have javascript implemenation so it doesn't need to rely on external, and many projects seems not maintained, or redirect us to forked project. b. multilang-protocol is outdated. - Spout subprocess can take a sleep in next() but shouldn't much because of heartbeat check. - Bolt subprocess should handle heartbeat tuple, and send sync to let ShellBolt know it's alive. 7. Storm-kafka project is in Storm project now, not storm-contrib. I'll be happy to contribute when modification needs a hand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-544) Some documents from website are outdated
[ https://issues.apache.org/jira/browse/STORM-544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-544. --- Resolution: Fixed Fix Version/s: 0.10.0 Some documents from website are outdated Key: STORM-544 URL: https://issues.apache.org/jira/browse/STORM-544 Project: Apache Storm Issue Type: Documentation Reporter: Jungtaek Lim Assignee: Jungtaek Lim Fix For: 0.10.0 Actually I've mailed this to Storm Dev mailing list, but nobody cares so I repost it to JIRA. I found that some documents are outdated. 1. storm-starter link from all (or most? I don't know) documents refers to https://github.com/nathanmarz/storm-starter, it should be https://github.com/apache/storm/tree/master/examples/storm-starter . 2. It seems that Storm project (including storm-starter) no longer uses Leiningen, right? It it does, some documents should reflect that change. - https://storm.apache.org/documentation/Creating-a-new-Storm-project.html says that storm project uses Leiningen yet. 3. https://storm.apache.org/documentation/Maven.html contains outdated contents. - It should reflect recent stable version of groupId and artifactId, and version. - Would it better to remove building user's project with Leiningen because Storm project doesn't use Leiningen? - Developing Storm section is completely outdated. 4. https://storm.apache.org/documentation/Maven.html Markdown for code block is not working. xml tag is removed so we can't understand what it means. 5. Some repository links refer to incubator-storm, not storm. 6. multilang documents are outdated a. Explanation for adapters is outdated. Storm have javascript implemenation so it doesn't need to rely on external, and many projects seems not maintained, or redirect us to forked project. b. multilang-protocol is outdated. - Spout subprocess can take a sleep in next() but shouldn't much because of heartbeat check. - Bolt subprocess should handle heartbeat tuple, and send sync to let ShellBolt know it's alive. 7. Storm-kafka project is in Storm project now, not storm-contrib. I'll be happy to contribute when modification needs a hand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-544) Some documents from website are outdated
[ https://issues.apache.org/jira/browse/STORM-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257133#comment-14257133 ] Robert Joseph Evans commented on STORM-544: --- I checked the changes into http://svn.apache.org/viewvc/storm/site/. I tried to follow the directions in the docs directory, but it deleted a number of files, so I kept the deleted files around, just in case, and only added/updated files. [~ptgoetz] if I messed something up please let me know, otherwise the new site should be up in a few hours at the most. Some documents from website are outdated Key: STORM-544 URL: https://issues.apache.org/jira/browse/STORM-544 Project: Apache Storm Issue Type: Documentation Reporter: Jungtaek Lim Assignee: Jungtaek Lim Fix For: 0.10.0 Actually I've mailed this to Storm Dev mailing list, but nobody cares so I repost it to JIRA. I found that some documents are outdated. 1. storm-starter link from all (or most? I don't know) documents refers to https://github.com/nathanmarz/storm-starter, it should be https://github.com/apache/storm/tree/master/examples/storm-starter . 2. It seems that Storm project (including storm-starter) no longer uses Leiningen, right? It it does, some documents should reflect that change. - https://storm.apache.org/documentation/Creating-a-new-Storm-project.html says that storm project uses Leiningen yet. 3. https://storm.apache.org/documentation/Maven.html contains outdated contents. - It should reflect recent stable version of groupId and artifactId, and version. - Would it better to remove building user's project with Leiningen because Storm project doesn't use Leiningen? - Developing Storm section is completely outdated. 4. https://storm.apache.org/documentation/Maven.html Markdown for code block is not working. xml tag is removed so we can't understand what it means. 5. Some repository links refer to incubator-storm, not storm. 6. multilang documents are outdated a. Explanation for adapters is outdated. Storm have javascript implemenation so it doesn't need to rely on external, and many projects seems not maintained, or redirect us to forked project. b. multilang-protocol is outdated. - Spout subprocess can take a sleep in next() but shouldn't much because of heartbeat check. - Bolt subprocess should handle heartbeat tuple, and send sync to let ShellBolt know it's alive. 7. Storm-kafka project is in Storm project now, not storm-contrib. I'll be happy to contribute when modification needs a hand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-595) storm-hdfs can only work with sequence files that use Writables
[ https://issues.apache.org/jira/browse/STORM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen resolved STORM-595. Resolution: Implemented storm-hdfs can only work with sequence files that use Writables --- Key: STORM-595 URL: https://issues.apache.org/jira/browse/STORM-595 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.3 Reporter: Mike Thomsen Assignee: Mike Thomsen Priority: Minor The current SequenceFormat interface requires that key() and value() return a class that implements Writable. This limitation makes it impossible to use object serialization systems like Avro or even Java serialization with the HDFS SequenceFileBolt. Proposed solution: change SequenceFormat so that key() and value() return an Object, not a Writable. This would keep existing functionality for those implementing Writable support while allowing serialization support for those of us that need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-575) Ability to specify Jetty host to bind to
[ https://issues.apache.org/jira/browse/STORM-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated STORM-575: -- Assignee: Matthew Siegel Ability to specify Jetty host to bind to Key: STORM-575 URL: https://issues.apache.org/jira/browse/STORM-575 Project: Apache Storm Issue Type: Improvement Reporter: Matthew Siegel Assignee: Matthew Siegel Priority: Minor In addition to being able to specify the port that the UI (Jetty) binds to, I need to be able to specify the host/interface that the UI binds to. Currently defaults to '0.0.0.0', all interfaces. Default will remain 0.0.0.0, and 'ui.host' config item to be added -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-575: Specify UI/Jetty interface to bind ...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/334 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-575) Ability to specify Jetty host to bind to
[ https://issues.apache.org/jira/browse/STORM-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257151#comment-14257151 ] ASF GitHub Bot commented on STORM-575: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/334 Ability to specify Jetty host to bind to Key: STORM-575 URL: https://issues.apache.org/jira/browse/STORM-575 Project: Apache Storm Issue Type: Improvement Reporter: Matthew Siegel Assignee: Matthew Siegel Priority: Minor Fix For: 0.10.0 In addition to being able to specify the port that the UI (Jetty) binds to, I need to be able to specify the host/interface that the UI binds to. Currently defaults to '0.0.0.0', all interfaces. Default will remain 0.0.0.0, and 'ui.host' config item to be added -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-595) storm-hdfs can only work with sequence files that use Writables
[ https://issues.apache.org/jira/browse/STORM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257159#comment-14257159 ] ASF GitHub Bot commented on STORM-595: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/351#issuecomment-67968645 +1. looks good to me. storm-hdfs can only work with sequence files that use Writables --- Key: STORM-595 URL: https://issues.apache.org/jira/browse/STORM-595 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.3 Reporter: Mike Thomsen Assignee: Mike Thomsen Priority: Minor The current SequenceFormat interface requires that key() and value() return a class that implements Writable. This limitation makes it impossible to use object serialization systems like Avro or even Java serialization with the HDFS SequenceFileBolt. Proposed solution: change SequenceFormat so that key() and value() return an Object, not a Writable. This would keep existing functionality for those implementing Writable support while allowing serialization support for those of us that need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-595) storm-hdfs can only work with sequence files that use Writables
[ https://issues.apache.org/jira/browse/STORM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257180#comment-14257180 ] ASF GitHub Bot commented on STORM-595: -- Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/351#issuecomment-67970628 +1 storm-hdfs can only work with sequence files that use Writables --- Key: STORM-595 URL: https://issues.apache.org/jira/browse/STORM-595 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.3 Reporter: Mike Thomsen Assignee: Mike Thomsen Priority: Minor The current SequenceFormat interface requires that key() and value() return a class that implements Writable. This limitation makes it impossible to use object serialization systems like Avro or even Java serialization with the HDFS SequenceFileBolt. Proposed solution: change SequenceFormat so that key() and value() return an Object, not a Writable. This would keep existing functionality for those implementing Writable support while allowing serialization support for those of us that need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-577：supervisor heartbeat and handle ev...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/332 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-577) long time launch worker will block supervisor heartbeat
[ https://issues.apache.org/jira/browse/STORM-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257219#comment-14257219 ] ASF GitHub Bot commented on STORM-577: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/332 long time launch worker will block supervisor heartbeat --- Key: STORM-577 URL: https://issues.apache.org/jira/browse/STORM-577 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0, 0.9.3-rc2 Reporter: xiajun Assignee: xiajun Priority: Minor Supervisor heartbeat and sync-processes use the same timer, when sync-processed use a lot time that greater than nimbus.supervisor.timeout.secs, nimbus will this supervisor was dead and move worker from this supervisor to others. At most time, block happens when worker launch consume too long time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-577) long time launch worker will block supervisor heartbeat
[ https://issues.apache.org/jira/browse/STORM-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-577. --- Resolution: Fixed Fix Version/s: 0.10.0 Thanks [~xiajun] I merged this into master. long time launch worker will block supervisor heartbeat --- Key: STORM-577 URL: https://issues.apache.org/jira/browse/STORM-577 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0, 0.9.3-rc2 Reporter: xiajun Assignee: xiajun Priority: Minor Fix For: 0.10.0 Supervisor heartbeat and sync-processes use the same timer, when sync-processed use a lot time that greater than nimbus.supervisor.timeout.secs, nimbus will this supervisor was dead and move worker from this supervisor to others. At most time, block happens when worker launch consume too long time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Storm-456 cannot navigate to topology page whe...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/318#issuecomment-67976452 The issue appears to be that when we url decode the topology name the '+' is staying as a '+' if I replace it with a space ' ' in the URL it seems to work for me on both firefox and chrome. I think we want to replace the space with '%20', not a '+'. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-552:add new config storm.messaging.netty...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/311#issuecomment-67976914 @caofangkun any update on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-552) add new config storm.messaging.netty.backlog
[ https://issues.apache.org/jira/browse/STORM-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257254#comment-14257254 ] ASF GitHub Bot commented on STORM-552: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/311#issuecomment-67976914 @caofangkun any update on this? add new config storm.messaging.netty.backlog Key: STORM-552 URL: https://issues.apache.org/jira/browse/STORM-552 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.9.3-rc2 Reporter: caofangkun Assignee: caofangkun Priority: Minor In Netty 3.7 backlog deault value is 50 for JDK 1.6 backlog = net.core.somaxconn -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-505: Fix debug string construction
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/266#issuecomment-67978120 Will open a New PR against master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-505) Debug Log Message in logviewer causes exception.
[ https://issues.apache.org/jira/browse/STORM-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257261#comment-14257261 ] ASF GitHub Bot commented on STORM-505: -- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/266#issuecomment-67978120 Will open a New PR against master. Debug Log Message in logviewer causes exception. Key: STORM-505 URL: https://issues.apache.org/jira/browse/STORM-505 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor Labels: security An incorrectly-constructed list of file names passed to a log-debug is causing a run-time exception when DEBUG log level is used. This prevents log clean-up from happening correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-505: Fix debug string construction
Github user d2r closed the pull request at: https://github.com/apache/storm/pull/266 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
hdfsbolt
Hi All, Why HdfsBolt doesn't retry when the hadoop node is down or not accessible and dies brings down the topology with it too? I can catch the run time exception and keep the topology going, but was wondering why the retry is not built into HdfsBolt. thank you Clay
[jira] [Commented] (STORM-505) Debug Log Message in logviewer causes exception.
[ https://issues.apache.org/jira/browse/STORM-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257260#comment-14257260 ] ASF GitHub Bot commented on STORM-505: -- Github user d2r closed the pull request at: https://github.com/apache/storm/pull/266 Debug Log Message in logviewer causes exception. Key: STORM-505 URL: https://issues.apache.org/jira/browse/STORM-505 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor Labels: security An incorrectly-constructed list of file names passed to a log-debug is causing a run-time exception when DEBUG log level is used. This prevents log clean-up from happening correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-505: Fix debug string construction
GitHub user d2r opened a pull request: https://github.com/apache/storm/pull/359 STORM-505: Fix debug string construction This pull request supersedes #266. You can merge this pull request into a Git repository by running: $ git pull https://github.com/d2r/storm STORM-505 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/359.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #359 commit 7d795f15ffae0ebc46e19934c0fdd411b2816cf0 Author: Derek Dagit der...@yahoo-inc.com Date: 2014-09-23T14:24:39Z Fix debug string construction commit 679d1103c12c6ac7f6da88261b406c3e4ec5dbd5 Author: Derek Dagit der...@yahoo-inc.com Date: 2014-09-23T18:50:55Z fix for lazy evaluation of log string commit 095373c7c8730d2e850eb5906933ea3b30bd28e8 Author: Derek Dagit der...@yahoo-inc.com Date: 2014-09-24T17:14:39Z let pr-str do some of the formatting --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-505) Debug Log Message in logviewer causes exception.
[ https://issues.apache.org/jira/browse/STORM-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257265#comment-14257265 ] ASF GitHub Bot commented on STORM-505: -- GitHub user d2r opened a pull request: https://github.com/apache/storm/pull/359 STORM-505: Fix debug string construction This pull request supersedes #266. You can merge this pull request into a Git repository by running: $ git pull https://github.com/d2r/storm STORM-505 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/359.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #359 commit 7d795f15ffae0ebc46e19934c0fdd411b2816cf0 Author: Derek Dagit der...@yahoo-inc.com Date: 2014-09-23T14:24:39Z Fix debug string construction commit 679d1103c12c6ac7f6da88261b406c3e4ec5dbd5 Author: Derek Dagit der...@yahoo-inc.com Date: 2014-09-23T18:50:55Z fix for lazy evaluation of log string commit 095373c7c8730d2e850eb5906933ea3b30bd28e8 Author: Derek Dagit der...@yahoo-inc.com Date: 2014-09-24T17:14:39Z let pr-str do some of the formatting Debug Log Message in logviewer causes exception. Key: STORM-505 URL: https://issues.apache.org/jira/browse/STORM-505 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor Labels: security An incorrectly-constructed list of file names passed to a log-debug is causing a run-time exception when DEBUG log level is used. This prevents log clean-up from happening correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-505: Fix debug string construction
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/359#issuecomment-67979768 +1 this is a rebase of https://github.com/apache/storm/pull/266 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-505) Debug Log Message in logviewer causes exception.
[ https://issues.apache.org/jira/browse/STORM-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257272#comment-14257272 ] ASF GitHub Bot commented on STORM-505: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/359#issuecomment-67979768 +1 this is a rebase of https://github.com/apache/storm/pull/266 Debug Log Message in logviewer causes exception. Key: STORM-505 URL: https://issues.apache.org/jira/browse/STORM-505 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor Labels: security An incorrectly-constructed list of file names passed to a log-debug is causing a run-time exception when DEBUG log level is used. This prevents log clean-up from happening correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-505) Debug Log Message in logviewer causes exception.
[ https://issues.apache.org/jira/browse/STORM-505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-505. --- Resolution: Fixed Fix Version/s: 0.10.0 Thanks Derek, I merged this into master. Debug Log Message in logviewer causes exception. Key: STORM-505 URL: https://issues.apache.org/jira/browse/STORM-505 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor Labels: security Fix For: 0.10.0 An incorrectly-constructed list of file names passed to a log-debug is causing a run-time exception when DEBUG log level is used. This prevents log clean-up from happening correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: hdfsbolt
Probably the best thing to do for this is to file a JIRA at issues.apache.org under STORM and include what it takes to reproduce the issue. - Bobby On Tuesday, December 23, 2014 11:42 AM, clay teahouse clayteaho...@gmail.com wrote: Hi All, Why HdfsBolt doesn't retry when the hadoop node is down or not accessible and dies brings down the topology with it too? I can catch the run time exception and keep the topology going, but was wondering why the retry is not built into HdfsBolt. thank you Clay
[GitHub] storm pull request: STORM-248 STORM-322 proposal for fix
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/353#issuecomment-67982494 +1 the changes look fine to me. Please upmerge. the storm python script has changed since you did the pull request and there are some minor merge conflicts that should be simple to fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-248) cluster.xml location is hardcoded for workers.
[ https://issues.apache.org/jira/browse/STORM-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257304#comment-14257304 ] ASF GitHub Bot commented on STORM-248: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/353#issuecomment-67982494 +1 the changes look fine to me. Please upmerge. the storm python script has changed since you did the pull request and there are some minor merge conflicts that should be simple to fix. cluster.xml location is hardcoded for workers. -- Key: STORM-248 URL: https://issues.apache.org/jira/browse/STORM-248 Project: Apache Storm Issue Type: Bug Reporter: Edward Goslin Assignee: Frantz Mazoyer Priority: Trivial when the supervisor spawns a worker process, it assumes the cluster.xml is in -Dlogback.configurationFile= storm-home /logback/cluster.xml It should take the VM arguement for the supervisor and pass it down to the worker. System.get(logback.configurationFile, storm-home + /logback/cluster.xml) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-243,Record version and revision informat...
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/294#discussion_r8158 --- Diff: storm-core/src/clj/backtype/storm/ui/core.clj --- @@ -52,15 +52,9 @@ (map #(.get_stats ^ExecutorSummary %)) (filter not-nil?))) -(defn read-storm-version +(def read-storm-version Returns a string containing the Storm version or 'Unknown'. - [] - (let [storm-home (System/getProperty storm.home) -release-path (format %s/RELEASE storm-home) -release-file (File. release-path)] -(if (and (.exists release-file) (.isFile release-file)) - (trim (slurp release-path)) - Unknown))) + (str VersionInfo/getVersion)) --- End diff -- This is not valid clojure. It should probably be something like ```(str (VersionInfo/getVersion))``` you are also going to have to explicitly import VersionInfo info core.clj. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-243) Record version and revision information in builds
[ https://issues.apache.org/jira/browse/STORM-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257325#comment-14257325 ] ASF GitHub Bot commented on STORM-243: -- Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/294#discussion_r8158 --- Diff: storm-core/src/clj/backtype/storm/ui/core.clj --- @@ -52,15 +52,9 @@ (map #(.get_stats ^ExecutorSummary %)) (filter not-nil?))) -(defn read-storm-version +(def read-storm-version Returns a string containing the Storm version or 'Unknown'. - [] - (let [storm-home (System/getProperty storm.home) -release-path (format %s/RELEASE storm-home) -release-file (File. release-path)] -(if (and (.exists release-file) (.isFile release-file)) - (trim (slurp release-path)) - Unknown))) + (str VersionInfo/getVersion)) --- End diff -- This is not valid clojure. It should probably be something like ```(str (VersionInfo/getVersion))``` you are also going to have to explicitly import VersionInfo info core.clj. Record version and revision information in builds -- Key: STORM-243 URL: https://issues.apache.org/jira/browse/STORM-243 Project: Apache Storm Issue Type: Improvement Reporter: caofangkun Assignee: caofangkun Priority: Minor Attachments: STORM-243-1.patch Patch atteThe effect shown below, $ storm version Storm 0.9.2-incubating-SNAPSHOT Subversion https://github.com/apache/incubator-storm/trunk/storm-core -r 1959 Compiled by somebody on Wed Feb 19 11:23:38 CST 2014 From source with checksum 9347aded8a39f3ddf8e8f2f9bf56186f or $ java -classpath storm-core-0.9.2-incubating-SNAPSHOT.jar backtype.storm.utils.VersionInfo Storm 0.9.2-incubating-SNAPSHOT Subversion https://github.com/apache/incubator-storm/trunk/storm-core -r 1959 Compiled by somebody on Wed Feb 19 11:23:38 CST 2014 From source with checksum 9347aded8a39f3ddf8e8f2f9bf56186f -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-243) Record version and revision information in builds
[ https://issues.apache.org/jira/browse/STORM-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257328#comment-14257328 ] ASF GitHub Bot commented on STORM-243: -- Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/294#discussion_r8250 --- Diff: storm-core/src/saveVersion.sh --- @@ -0,0 +1,42 @@ +s file is used to generate the package-info.java class that +# records the version, revision, branch, user, timestamp, and url +unset LANG +unset LC_CTYPE +unset LC_TIME +version=$1 +build_dir=$2 +user=`whoami` +date=`date` +cwd=`pwd` +if [ -d .git ]; then --- End diff -- This does not work with how the code is run from maven. The working directory, at least for me, seems to be storm-core, so it never finds .git and is not able to detect that the code is in git. subversion should not have the same issues. Record version and revision information in builds -- Key: STORM-243 URL: https://issues.apache.org/jira/browse/STORM-243 Project: Apache Storm Issue Type: Improvement Reporter: caofangkun Assignee: caofangkun Priority: Minor Attachments: STORM-243-1.patch Patch atteThe effect shown below, $ storm version Storm 0.9.2-incubating-SNAPSHOT Subversion https://github.com/apache/incubator-storm/trunk/storm-core -r 1959 Compiled by somebody on Wed Feb 19 11:23:38 CST 2014 From source with checksum 9347aded8a39f3ddf8e8f2f9bf56186f or $ java -classpath storm-core-0.9.2-incubating-SNAPSHOT.jar backtype.storm.utils.VersionInfo Storm 0.9.2-incubating-SNAPSHOT Subversion https://github.com/apache/incubator-storm/trunk/storm-core -r 1959 Compiled by somebody on Wed Feb 19 11:23:38 CST 2014 From source with checksum 9347aded8a39f3ddf8e8f2f9bf56186f -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-243) Record version and revision information in builds
[
https://issues.apache.org/jira/browse/STORM-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257335#comment-14257335
]
ASF GitHub Bot commented on STORM-243:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/294#discussion_r8357
--- Diff: storm-dist/binary/src/main/assembly/binary.xml ---
@@ -131,13 +131,6 @@
/file
!-- TODO this should be a generated file from target --
file
-source${project.basedir}/../../VERSION/source
-outputDirectory//outputDirectory
-destNameRELEASE/destName
--- End diff --
There are explicit checks elsewhere in the code looking for the RELEASE
file. If it is not present some parts of the code assume they are not part of
a valid release.
https://github.com/caofangkun/apache-storm/blob/storm-243/bin/storm#L86-91
is one example of where this happens.
Record version and revision information in builds
--
Key: STORM-243
URL: https://issues.apache.org/jira/browse/STORM-243
Project: Apache Storm
Issue Type: Improvement
Reporter: caofangkun
Assignee: caofangkun
Priority: Minor
Attachments: STORM-243-1.patch
Patch atteThe effect shown below,
$ storm version
Storm 0.9.2-incubating-SNAPSHOT
Subversion https://github.com/apache/incubator-storm/trunk/storm-core -r 1959
Compiled by somebody on Wed Feb 19 11:23:38 CST 2014
From source with checksum 9347aded8a39f3ddf8e8f2f9bf56186f
or
$ java -classpath storm-core-0.9.2-incubating-SNAPSHOT.jar
backtype.storm.utils.VersionInfo
Storm 0.9.2-incubating-SNAPSHOT
Subversion https://github.com/apache/incubator-storm/trunk/storm-core -r 1959
Compiled by somebody on Wed Feb 19 11:23:38 CST 2014
From source with checksum 9347aded8a39f3ddf8e8f2f9bf56186f
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-243,Record version and revision informat...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/294#issuecomment-67985049 @caofangkun thank you for submitting this patch. I appreciate you trying to help. In the future it would be good though, to be sure that your patch complies and the unit tests pass before submitting it for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-243) Record version and revision information in builds
[ https://issues.apache.org/jira/browse/STORM-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257341#comment-14257341 ] ASF GitHub Bot commented on STORM-243: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/294#issuecomment-67985049 @caofangkun thank you for submitting this patch. I appreciate you trying to help. In the future it would be good though, to be sure that your patch complies and the unit tests pass before submitting it for review. Record version and revision information in builds -- Key: STORM-243 URL: https://issues.apache.org/jira/browse/STORM-243 Project: Apache Storm Issue Type: Improvement Reporter: caofangkun Assignee: caofangkun Priority: Minor Attachments: STORM-243-1.patch Patch atteThe effect shown below, $ storm version Storm 0.9.2-incubating-SNAPSHOT Subversion https://github.com/apache/incubator-storm/trunk/storm-core -r 1959 Compiled by somebody on Wed Feb 19 11:23:38 CST 2014 From source with checksum 9347aded8a39f3ddf8e8f2f9bf56186f or $ java -classpath storm-core-0.9.2-incubating-SNAPSHOT.jar backtype.storm.utils.VersionInfo Storm 0.9.2-incubating-SNAPSHOT Subversion https://github.com/apache/incubator-storm/trunk/storm-core -r 1959 Compiled by somebody on Wed Feb 19 11:23:38 CST 2014 From source with checksum 9347aded8a39f3ddf8e8f2f9bf56186f -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-602) HdfsBolt dies when the hadoop node is not available
clay teahouse created STORM-602: --- Summary: HdfsBolt dies when the hadoop node is not available Key: STORM-602 URL: https://issues.apache.org/jira/browse/STORM-602 Project: Apache Storm Issue Type: Bug Components: storm-hdfs Affects Versions: 0.9.3 Environment: Ubuntu 14.04 Reporter: clay teahouse When the hadoop nodes are not available, HdfsBolt generates the following run time error, and dies and the topology dies with it too. 12154 [Thread-50-hdfsBolt2] ERROR backtype.storm.util - Halting process: (Worker died) java.lang.RuntimeException: (Worker died) at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:319) [storm-core-0.9.3-SNAPSHOT.jar:0.9.3-SNAPSHOT] at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$fn__4770$fn__4771.invoke(worker.clj:452) [storm-core-0.9.3-SNAPSHOT.jar:0.9.3-SNAPSHOT] at backtype.storm.daemon.executor$mk_executor_data$fn__3287$fn__3288.invoke(executor.clj:239) [storm-core-0.9.3-SNAPSHOT.jar:0.9.3-SNAPSHOT] at backtype.storm.util$async_loop$fn__458.invoke(util.clj:467) [storm-core-0.9.3-SNAPSHOT.jar:0.9.3-SNAPSHOT] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-442] multilang ShellBolt/ShellSpout die...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/305#issuecomment-67989780 I see this has not had that much traffic in quite a while. The code to me looks acceptable. I am not really happy with calling .available() twice, but I can live with it. There is the possibility of a race if two different threads call getErrorsString or logErrorStream at the same time, but as the way the code is currently used that should not be a problem. I am +1 on checking this in, but would like to know if others have any option. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-442) multilang ShellBolt/ShellSpout die() can be hang when Exception happened
[
https://issues.apache.org/jira/browse/STORM-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257411#comment-14257411
]
ASF GitHub Bot commented on STORM-442:
--
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/305#issuecomment-67989780
I see this has not had that much traffic in quite a while. The code to me
looks acceptable. I am not really happy with calling .available() twice, but I
can live with it. There is the possibility of a race if two different threads
call getErrorsString or logErrorStream at the same time, but as the way the
code is currently used that should not be a problem. I am +1 on checking this
in, but would like to know if others have any option.
multilang ShellBolt/ShellSpout die() can be hang when Exception happened
Key: STORM-442
URL: https://issues.apache.org/jira/browse/STORM-442
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3
Reporter: DashengJu
In ShellBolt, the _readerThread read command from python/shell process, and
handle like this:
try {
ShellMsg shellMsg = _process.readShellMsg();
...
} catch (InterruptedException e) {
} catch (Throwable t) {
die(t);
}
And in the die function, getProcessTerminationInfoString will read
getErrorsString() from processErrorStream.
private void die(Throwable exception) {
String processInfo = _process.getProcessInfoString() +
_process.getProcessTerminationInfoString();
_exception = new RuntimeException(processInfo, exception);
}
so when ShellBolt got exception(for example, readShellMsg() throw NPE ) ,
but it is not an error from sub process, then
getProcessTerminationInfoString will be hang because processErrorStream have
no data to read.
On the other hand, as [~xiaokang] says ShellBolt should fail fast on
exception ( https://github.com/apache/incubator-storm/pull/46 ) , I think it
is not a good idea to read error info from stream.
Because [~xiaokang] 's PR is based old version, so I will move his code to
this PR, and modify some other place in ShellSpout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: Storm-456: changing encoding to ring/url-encod...
GitHub user Parth-Brahmbhatt opened a pull request: https://github.com/apache/storm/pull/360 Storm-456: changing encoding to ring/url-encode. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Parth-Brahmbhatt/incubator-storm Storm-456 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/360.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #360 commit 16b7382dc7675d7a8b7aa0108af836913eef0560 Author: Parth Brahmbhatt brahmbhatt.pa...@gmail.com Date: 2014-12-23T19:56:08Z STORM-456: Changing the url-encode method to use ring ecnode as ring auto decodes the requests with its own scheme. commit d42d8dab3a7e5524f96ed1e9985381ca2936544e Author: Parth Brahmbhatt brahmbhatt.pa...@gmail.com Date: 2014-12-23T20:19:58Z Merge remote-tracking branch 'upstream/master' into Storm-456 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-591] exclude logback.xml from jar
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/352#issuecomment-67997538 This seems to be used by the tests, after this change see a lot of DEBUG and INFO logs from zookeeper. Instead of deleting it, can we move it to a directory that is only on the classpath for tests? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-591) logback.xml
[ https://issues.apache.org/jira/browse/STORM-591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257521#comment-14257521 ] ASF GitHub Bot commented on STORM-591: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/352#issuecomment-67997538 This seems to be used by the tests, after this change see a lot of DEBUG and INFO logs from zookeeper. Instead of deleting it, can we move it to a directory that is only on the classpath for tests? logback.xml --- Key: STORM-591 URL: https://issues.apache.org/jira/browse/STORM-591 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.3 Reporter: Shaun Thompson logback.xml is now being included in storm-core.jar. This causes issues for any application that might include storm-core in it's classpath, having to override the config file location. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-599] Use use nimbus's cached heartbeats...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/356#issuecomment-67997939 The changes look fine to me. My only concern would be around what happens if the cache has not been updated and is empty, but it looks like it will return an empty map, and that looks like what happened before when there were no heartbeats. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-599) UI is slow due to fetching heartbeats from ZK
[ https://issues.apache.org/jira/browse/STORM-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257528#comment-14257528 ] ASF GitHub Bot commented on STORM-599: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/356#issuecomment-67997939 The changes look fine to me. My only concern would be around what happens if the cache has not been updated and is empty, but it looks like it will return an empty map, and that looks like what happened before when there were no heartbeats. +1 UI is slow due to fetching heartbeats from ZK - Key: STORM-599 URL: https://issues.apache.org/jira/browse/STORM-599 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.9.3 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor The method getTopologyInfo fetches every heartbeat from ZooKeeper in order to gather statistics. The UI calls this method via thrift to render the topology page. When topologies are launched with thousands of executors, this fetching from ZK dramatically slows things down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-598:Newly submitted topologies do not sh...
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/355#discussion_r22236169 --- Diff: storm-core/src/clj/backtype/storm/daemon/nimbus.clj --- @@ -1248,7 +1256,7 @@ (when-let [owner (:owner base)] (.set_owner topo-summ owner)) (when-let [sched-status (.get @(:id-sched-status nimbus) id)] (.set_sched_status topo-summ sched-status)) topo-summ - ))] + )))] --- End diff -- The new ')' is in the wrong place. It should be up on line 1255 instead. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-598:Newly submitted topologies do not sh...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/355#issuecomment-68001550 Also I tried to reproduce the issue on master and I am seeing the second topology come up on the ui without any issues. Could you try to reproduce the error on master and let me know if it is still an issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-598) Newly submitted topologies do not show up on the storm ui cluser page when the Storm Cluster run out of worker slots
[
https://issues.apache.org/jira/browse/STORM-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257578#comment-14257578
]
ASF GitHub Bot commented on STORM-598:
--
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/355#issuecomment-68001550
Also I tried to reproduce the issue on master and I am seeing the second
topology come up on the ui without any issues. Could you try to reproduce the
error on master and let me know if it is still an issue?
Newly submitted topologies do not show up on the storm ui cluser page when
the Storm Cluster run out of worker slots
-
Key: STORM-598
URL: https://issues.apache.org/jira/browse/STORM-598
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3-rc2
Reporter: caofangkun
Assignee: caofangkun
Priority: Minor
1: Set up a Storm Cluster with 1 Supervisor(4 worker slots)
2: submit topologyA and use 4 workers
3: submit topologyB with 4 workers
topologyB does not but should show up on the storm ui cluster page.
See Code Line 232：
https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/daemon/nimbus.clj#L1232
If assignment is null, just new Assignmnet with 0 workers and 0 tasks?
{code:title=nimbus.clj|borderStyle=solid}
Index: src/clj/backtype/storm/daemon/nimbus.clj
===
--- src/clj/backtype/storm/daemon/nimbus.clj (revision 4324)
+++ src/clj/backtype/storm/daemon/nimbus.clj (working copy)
@@ -1230,7 +1230,9 @@
bases (topology-bases storm-cluster-state)
topology-summaries (dofor [[id base] bases :when base]
(let [assignment (.assignment-info
storm-cluster-state id nil)
-topo-summ (TopologySummary.
id
+topo-summ ( if (nil?
assignment)
+ (TopologySummary.
id (:storm-name base) 0 0 0 (time-delta (:launch-time-secs base))
(extract-status-str base))
+ (TopologySummary.
id
(:storm-name
base)
(-
(:executor-node+port assignment)
keys
@@ -1244,7 +1246,7 @@
set
count)
(time-delta
(:launch-time-secs base))
-
(extract-status-str base))]
+
(extract-status-str base)))]
(when-let [owner (:owner
base)] (.set_owner topo-summ owner))
(when-let [sched-status (.get
@(:id-sched-status nimbus) id)] (.set_sched_status topo-summ sched-status))
topo-summ
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-603) storm-kafka: Log errors when missing required configuration fields
Curtis Allen created STORM-603:
--
Summary: storm-kafka: Log errors when missing required
configuration fields
Key: STORM-603
URL: https://issues.apache.org/jira/browse/STORM-603
Project: Apache Storm
Issue Type: Bug
Reporter: Curtis Allen
Assignee: Curtis Allen
I was upgrading our topologies to storm-0.9.3 this
[commit|https://github.com/apache/storm/commit/2596e335f27a57784a93a57823bd93dde587909f]
introduced a change that threw me for a loop. When submitting my topology I
got the following error.
{code}
[main] ERROR storm.kafka.DynamicBrokersReader - Couldn't connect to zookeeper
java.lang.IllegalArgumentException: Don't know how to convert null to int
at backtype.storm.utils.Utils.getInt(Utils.java:301)
~[storm-core-0.9.3.jar:0.9.3]
at
storm.kafka.DynamicBrokersReader.init(DynamicBrokersReader.java:47)
~[gambit-storm-threads-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na]
at
com.pearson.gambit.threads.storm.ThreadsTopology.main(ThreadsTopology.java:45)
[gambit-storm-threads-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na]
Exception in thread main java.lang.RuntimeException:
java.lang.RuntimeException: java.lang.NullPointerException
at
storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:81)
at
com.pearson.gambit.threads.storm.ThreadsTopology.main(ThreadsTopology.java:48)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:94)
at
storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65)
... 1 more
Caused by: java.lang.NullPointerException
at
storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:91)
... 2 more
{code}
It took me a while to figure out that this error stems from missing the
{code}backtype.storm.Config.STORM_ZOOKEEPER_CONNECTION_TIMEOUT{code} property
in the conf map provided to the {code}storm.kafka.DynamicBrokersReader{code}
constructer. It would be nice to check the required configuration parameters
and throw a RuntimeException if any are missing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-603 Log errors when required kafka param...
GitHub user curtisallen opened a pull request: https://github.com/apache/storm/pull/361 STORM-603 Log errors when required kafka params are missing https://issues.apache.org/jira/browse/STORM-603 I was upgrading our topologies to storm-0.9.3 this [commit](https://github.com/apache/storm/commit/2596e335f27a57784a93a57823bd93dde587909f) introduced a change that threw me for a loop. When submitting my topology I got the following error. ``` [main] ERROR storm.kafka.DynamicBrokersReader - Couldn't connect to zookeeper java.lang.IllegalArgumentException: Don't know how to convert null to int at backtype.storm.utils.Utils.getInt(Utils.java:301) ~[storm-core-0.9.3.jar:0.9.3] at storm.kafka.DynamicBrokersReader.init(DynamicBrokersReader.java:47) ~[gambit-storm-threads-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at com.pearson.gambit.threads.storm.ThreadsTopology.main(ThreadsTopology.java:45) [gambit-storm-threads-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] Exception in thread main java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:81) at com.pearson.gambit.threads.storm.ThreadsTopology.main(ThreadsTopology.java:48) Caused by: java.lang.RuntimeException: java.lang.NullPointerException at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:94) at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ... 1 more Caused by: java.lang.NullPointerException at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:91) ... 2 more ``` It took me a while to figure out that this error stems from missing the `backtype.storm.Config.STORM_ZOOKEEPER_CONNECTION_TIMEOUT` property in the `conf` map provided to the `storm.kafka.DynamicBrokersReader` constructer. It would be nice to check the required configuration parameters and throw a RuntimeException if any are missing. This pull requests adds logging when required errors are missing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/curtisallen/storm STORM-603 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/361.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #361 commit 326cc1c796d073c768de4b5aced1669bd47c6891 Author: Curtis Allen curtis.al...@pearson.com Date: 2014-12-23T23:13:31Z Log errors when required kafka params are missing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Merge pull request #2 from apache/master
GitHub user caofangkun opened a pull request: https://github.com/apache/storm/pull/362 Merge pull request #2 from apache/master Merge from trunk to caofangkun You can merge this pull request into a Git repository by running: $ git pull https://github.com/caofangkun/apache-storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/362.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #362 commit 97ae36e42abc961fbcc54dd77c80bafff30dce7e Author: caofangkun caofang...@gmail.com Date: 2014-12-22T08:18:08Z Merge pull request #2 from apache/master Merge from trunk to caofangkun --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-552:add new config storm.messaging.netty...
Github user caofangkun commented on the pull request: https://github.com/apache/storm/pull/311#issuecomment-68022108 @revans2 Thank you for your explanation and good advise . I have renamed storm.messaging.netty.backlog,default to storm.messaging.netty.socket.backlog and update the patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-552) add new config storm.messaging.netty.backlog
[ https://issues.apache.org/jira/browse/STORM-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257879#comment-14257879 ] ASF GitHub Bot commented on STORM-552: -- Github user caofangkun commented on the pull request: https://github.com/apache/storm/pull/311#issuecomment-68022108 @revans2 Thank you for your explanation and good advise . I have renamed storm.messaging.netty.backlog,default to storm.messaging.netty.socket.backlog and update the patch. add new config storm.messaging.netty.backlog Key: STORM-552 URL: https://issues.apache.org/jira/browse/STORM-552 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.9.3-rc2 Reporter: caofangkun Assignee: caofangkun Priority: Minor In Netty 3.7 backlog deault value is 50 for JDK 1.6 backlog = net.core.somaxconn -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Merge pull request #2 from apache/master
Github user caofangkun closed the pull request at: https://github.com/apache/storm/pull/362 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (STORM-552) add new config storm.messaging.netty.socket.backlog
[ https://issues.apache.org/jira/browse/STORM-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caofangkun updated STORM-552: - Summary: add new config storm.messaging.netty.socket.backlog (was: add new config storm.messaging.netty.backlog) add new config storm.messaging.netty.socket.backlog --- Key: STORM-552 URL: https://issues.apache.org/jira/browse/STORM-552 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.9.3-rc2 Reporter: caofangkun Assignee: caofangkun Priority: Minor In Netty 3.7 backlog deault value is 50 for JDK 1.6 backlog = net.core.somaxconn -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-596) topology.receiver.buffer.size has no effect
[ https://issues.apache.org/jira/browse/STORM-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257928#comment-14257928 ] ASF GitHub Bot commented on STORM-596: -- GitHub user caofangkun opened a pull request: https://github.com/apache/storm/pull/363 STORM-596: remove config topology.receiver.buffer.size topology.receiver.buffer.size has no effect and not necessary for now. Should we just remove it ? You can merge this pull request into a Git repository by running: $ git pull https://github.com/caofangkun/apache-storm storm-596 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/363.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #363 commit 97ae36e42abc961fbcc54dd77c80bafff30dce7e Author: caofangkun caofang...@gmail.com Date: 2014-12-22T08:18:08Z Merge pull request #2 from apache/master Merge from trunk to caofangkun commit cad1505deb214facb0fb70e24034e30f79a973da Author: caofangkun caofang...@gmail.com Date: 2014-12-24T03:44:25Z STORM-596: remove config topology.receiver.buffer.size commit 986ad60bf065c8c3dd965b5c2cfd33d71229385b Author: caofangkun caofang...@gmail.com Date: 2014-12-24T03:56:12Z STORM-596: rm defaults.yaml.orig Config.java.orig topology.receiver.buffer.size has no effect -- Key: STORM-596 URL: https://issues.apache.org/jira/browse/STORM-596 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0, 0.9.3-rc2 Reporter: caofangkun Assignee: caofangkun Priority: Minor https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/messaging/loader.clj#L27 backtype.storm.messaging.loader#mk-receive-thread accepts max-buffer-size as an input but the value isn't used within the function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
