[jira] [Commented] (STORM-995) ILocalCluster topology methods can throw NotAliveException
[ https://issues.apache.org/jira/browse/STORM-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701005#comment-14701005 ] Simon Cooper commented on STORM-995: I've also seen shutdown() throw checked exceptions, but I'm not sure which ones it can throw from the clojure... ILocalCluster topology methods can throw NotAliveException -- Key: STORM-995 URL: https://issues.apache.org/jira/browse/STORM-995 Project: Apache Storm Issue Type: Bug Reporter: Simon Cooper Priority: Minor The getTopology*(String) methods in ILocalCluster can all throw the checked exception NotAliveException through nimbus, but it's not declared in the method signatures, which confuses the Java compiler. They are declared correctly in Nimbus.Iface. Affected methods - getTopologyConf, getTopology, getTopologyInfo -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-976) Config storm.logback.conf.dir is specific to previous logging framework
[ https://issues.apache.org/jira/browse/STORM-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yvonne Ironberg reassigned STORM-976: - Assignee: Yvonne Ironberg Config storm.logback.conf.dir is specific to previous logging framework --- Key: STORM-976 URL: https://issues.apache.org/jira/browse/STORM-976 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Yvonne Ironberg Priority: Minor Labels: Newbie Fix For: 0.11.0 Storm has migrated from logback to log4j2, so we should rename this config and code that uses it. https://github.com/apache/storm/blob/544e55cb8ab8878c4af500aab49bd35d4b69cd3e/storm-core/src/clj/backtype/storm/daemon/supervisor.clj#L664 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-976) Config storm.logback.conf.dir is specific to previous logging framework
[ https://issues.apache.org/jira/browse/STORM-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701639#comment-14701639 ] ASF GitHub Bot commented on STORM-976: -- Github user YvonneIronberg commented on the pull request: https://github.com/apache/storm/pull/684#issuecomment-132286396 Thanks, @d2r! I've assigned STORM-976 to me on Apache jira. Config storm.logback.conf.dir is specific to previous logging framework --- Key: STORM-976 URL: https://issues.apache.org/jira/browse/STORM-976 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Yvonne Ironberg Priority: Minor Labels: Newbie Fix For: 0.11.0 Storm has migrated from logback to log4j2, so we should rename this config and code that uses it. https://github.com/apache/storm/blob/544e55cb8ab8878c4af500aab49bd35d4b69cd3e/storm-core/src/clj/backtype/storm/daemon/supervisor.clj#L664 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-976
Github user YvonneIronberg commented on the pull request: https://github.com/apache/storm/pull/684#issuecomment-132286396 Thanks, @d2r! I've assigned STORM-976 to me on Apache jira. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-829
GitHub user YvonneIronberg opened a pull request: https://github.com/apache/storm/pull/689 STORM-829 You can merge this pull request into a Git repository by running: $ git pull https://github.com/YvonneIronberg/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/689.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #689 commit 6a30be80a895c63d5e0c53183a9807b8e8c3a8b5 Author: YvonneIronberg yvonne.ironb...@gmail.com Date: 2015-08-18T17:37:31Z STORM-829. commit 8d487852e51c67d943ee6ea2b778e9e68da5f062 Author: YvonneIronberg yvonne.ironb...@gmail.com Date: 2015-08-18T17:44:29Z STORM-829. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-998) Add support for user specified UGI - (UserGroupInformation) for storm hbase connector
Priyank Shah created STORM-998: -- Summary: Add support for user specified UGI - (UserGroupInformation) for storm hbase connector Key: STORM-998 URL: https://issues.apache.org/jira/browse/STORM-998 Project: Apache Storm Issue Type: Sub-task Reporter: Priyank Shah Assignee: Priyank Shah -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-998) Add support for user specified UGI - (UserGroupInformation) for storm hbase connector
[ https://issues.apache.org/jira/browse/STORM-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyank Shah updated STORM-998: --- Description: In a non-secure environment, Storm HBase component that provides interaction with HBase from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with hdfs as the user provided instead of user running the worker process Add support for user specified UGI - (UserGroupInformation) for storm hbase connector - Key: STORM-998 URL: https://issues.apache.org/jira/browse/STORM-998 Project: Apache Storm Issue Type: Sub-task Reporter: Priyank Shah Assignee: Priyank Shah In a non-secure environment, Storm HBase component that provides interaction with HBase from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with hdfs as the user provided instead of user running the worker process -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-829) Hadoop dependency confusion
[ https://issues.apache.org/jira/browse/STORM-829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701682#comment-14701682 ] ASF GitHub Bot commented on STORM-829: -- GitHub user YvonneIronberg opened a pull request: https://github.com/apache/storm/pull/689 STORM-829 You can merge this pull request into a Git repository by running: $ git pull https://github.com/YvonneIronberg/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/689.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #689 commit 6a30be80a895c63d5e0c53183a9807b8e8c3a8b5 Author: YvonneIronberg yvonne.ironb...@gmail.com Date: 2015-08-18T17:37:31Z STORM-829. commit 8d487852e51c67d943ee6ea2b778e9e68da5f062 Author: YvonneIronberg yvonne.ironb...@gmail.com Date: 2015-08-18T17:44:29Z STORM-829. Hadoop dependency confusion --- Key: STORM-829 URL: https://issues.apache.org/jira/browse/STORM-829 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans The hadoop dependencies seem a bit messed up. We have a hadoop.version set in the main pom.xml to 2.6.0 but it does not really appear to be used. storm-core uses hadoop-auth set to 2.4.0, storm-hdfs hard codes it to 2.2.0, storm-hbase does the same. We should have all of the versions be consistent and come from the main pom.xml. Also what are we using hadoop-auth for in storm-core. Also it looks like we should remove the hadoop-auth dependency. When AutoHDFS and AutoHBase moved out of storm-core it looks like it was left there by mistake. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (STORM-996) netty-unit-tests/test-batch demonstrates out-of-order delivery
[
https://issues.apache.org/jira/browse/STORM-996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701818#comment-14701818
]
Derek Dagit edited comment on STORM-996 at 8/18/15 7:19 PM:
Suspecting maybe the [asynchronous
write|https://github.com/apache/storm/blob/3b1b3bfa595851afd468594079caa7c772dafd48/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java#L323]
in Client#flushMesages might be the issue. We create a future here to write
the data asynchronously, but the new thread does not have to be scheduled to
run before subsequent batches are written, and so batches can be sent in the
wrong order.
EDIT: spelling
was (Author: dagit):
Suspecting maybe the [asynchronous
write|https://github.com/apache/storm/blob/3b1b3bfa595851afd468594079caa7c772dafd48/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java#L323]
in Client#flushMesages might be the issue. We create a future here to write
the data asynchronously, but the new thread does not have to be scheduled to
run before subsequent batches are written, and so batches can be send in the
wrong order.
netty-unit-tests/test-batch demonstrates out-of-order delivery
--
Key: STORM-996
URL: https://issues.apache.org/jira/browse/STORM-996
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Derek Dagit
Assignee: Derek Dagit
Priority: Blocker
backtype.storm.messaging.netty-unit-test/test-batch
One example of output. Similar things happen sporadically and vary widely by
number of failed assertions.
Tuples are not just skewed, but actually seem to come in out-of-order.
{quote}
actual: (not (= 66040 66041))
at: test_runner.clj:105
expected: (= req_msg resp_msg)
actual: (not (= 66041 66042))
at: test_runner.clj:105
expected: (= req_msg resp_msg)
actual: (not (= 66042 66040))
at: test_runner.clj:105
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-999) Add support for user specified UGI - (UserGroupInformation) for storm hive connector
[ https://issues.apache.org/jira/browse/STORM-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyank Shah updated STORM-999: --- Description: In a non-secure environment, Storm Hive component that provides interaction with Hive from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with Hive as the user provided instead of user running the worker process Add support for user specified UGI - (UserGroupInformation) for storm hive connector Key: STORM-999 URL: https://issues.apache.org/jira/browse/STORM-999 Project: Apache Storm Issue Type: Sub-task Reporter: Priyank Shah Assignee: Priyank Shah In a non-secure environment, Storm Hive component that provides interaction with Hive from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with Hive as the user provided instead of user running the worker process -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-998) Add support for user specified UGI - (UserGroupInformation) for storm hbase connector
[ https://issues.apache.org/jira/browse/STORM-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyank Shah updated STORM-998: --- Description: In a non-secure environment, Storm HBase component that provides interaction with HBase from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with HBase as the user provided instead of user running the worker process (was: In a non-secure environment, Storm HBase component that provides interaction with HBase from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with hdfs as the user provided instead of user running the worker process) Add support for user specified UGI - (UserGroupInformation) for storm hbase connector - Key: STORM-998 URL: https://issues.apache.org/jira/browse/STORM-998 Project: Apache Storm Issue Type: Sub-task Reporter: Priyank Shah Assignee: Priyank Shah In a non-secure environment, Storm HBase component that provides interaction with HBase from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with HBase as the user provided instead of user running the worker process -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (STORM-997) Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector
[ https://issues.apache.org/jira/browse/STORM-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyank Shah updated STORM-997: --- Comment: was deleted (was: In a non-secure environment, Storm HDFS component that provides interaction with HDFS from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with hdfs as the user provided instead of user running the worker process.) Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector Key: STORM-997 URL: https://issues.apache.org/jira/browse/STORM-997 Project: Apache Storm Issue Type: Sub-task Reporter: Priyank Shah Assignee: Priyank Shah -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-997) Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector
Priyank Shah created STORM-997: -- Summary: Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector Key: STORM-997 URL: https://issues.apache.org/jira/browse/STORM-997 Project: Apache Storm Issue Type: Sub-task Reporter: Priyank Shah Assignee: Priyank Shah -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-999) Add support for user specified UGI - (UserGroupInformation) for storm hive connector
Priyank Shah created STORM-999: -- Summary: Add support for user specified UGI - (UserGroupInformation) for storm hive connector Key: STORM-999 URL: https://issues.apache.org/jira/browse/STORM-999 Project: Apache Storm Issue Type: Sub-task Reporter: Priyank Shah Assignee: Priyank Shah -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-997) Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector
[ https://issues.apache.org/jira/browse/STORM-997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701690#comment-14701690 ] Priyank Shah commented on STORM-997: In a non-secure environment, Storm HDFS component that provides interaction with HDFS from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with hdfs as the user provided instead of user running the worker process. Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector Key: STORM-997 URL: https://issues.apache.org/jira/browse/STORM-997 Project: Apache Storm Issue Type: Sub-task Reporter: Priyank Shah Assignee: Priyank Shah -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-997) Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector
[ https://issues.apache.org/jira/browse/STORM-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyank Shah updated STORM-997: --- Description: In a non-secure environment, Storm HDFS component that provides interaction with HDFS from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with hdfs as the user provided instead of user running the worker process Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector Key: STORM-997 URL: https://issues.apache.org/jira/browse/STORM-997 Project: Apache Storm Issue Type: Sub-task Reporter: Priyank Shah Assignee: Priyank Shah In a non-secure environment, Storm HDFS component that provides interaction with HDFS from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with hdfs as the user provided instead of user running the worker process -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-996) netty-unit-tests/test-batch demonstrates out-of-order delivery
[
https://issues.apache.org/jira/browse/STORM-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Derek Dagit reassigned STORM-996:
-
Assignee: Derek Dagit
netty-unit-tests/test-batch demonstrates out-of-order delivery
--
Key: STORM-996
URL: https://issues.apache.org/jira/browse/STORM-996
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Derek Dagit
Assignee: Derek Dagit
Priority: Blocker
backtype.storm.messaging.netty-unit-test/test-batch
One example of output. Similar things happen sporadically and vary widely by
number of failed assertions.
Tuples are not just skewed, but actually seem to come in out-of-order.
{quote}
actual: (not (= 66040 66041))
at: test_runner.clj:105
expected: (= req_msg resp_msg)
actual: (not (= 66041 66042))
at: test_runner.clj:105
expected: (= req_msg resp_msg)
actual: (not (= 66042 66040))
at: test_runner.clj:105
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-996) netty-unit-tests/test-batch demonstrates out-of-order delivery
[
https://issues.apache.org/jira/browse/STORM-996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701818#comment-14701818
]
Derek Dagit commented on STORM-996:
---
Suspecting maybe the [asynchronous
write|https://github.com/apache/storm/blob/3b1b3bfa595851afd468594079caa7c772dafd48/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java#L323]
in Client#flushMesages might be the issue. We create a future here to write
the data asynchronously, but the new thread does not have to be scheduled to
run before subsequent batches are written, and so batches can be send in the
wrong order.
netty-unit-tests/test-batch demonstrates out-of-order delivery
--
Key: STORM-996
URL: https://issues.apache.org/jira/browse/STORM-996
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Derek Dagit
Assignee: Derek Dagit
Priority: Blocker
backtype.storm.messaging.netty-unit-test/test-batch
One example of output. Similar things happen sporadically and vary widely by
number of failed assertions.
Tuples are not just skewed, but actually seem to come in out-of-order.
{quote}
actual: (not (= 66040 66041))
at: test_runner.clj:105
expected: (= req_msg resp_msg)
actual: (not (= 66041 66042))
at: test_runner.clj:105
expected: (= req_msg resp_msg)
actual: (not (= 66042 66040))
at: test_runner.clj:105
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Cannot run WordCountExample in Intellij
Hi Matthias, sorry to respond lately. Unfortunately, AFAIK you can't run multilang feature with LocalCluster without having packaged file. ShellProcess relies on codeDir of TopologyContext, which is used by supervisor. Workers are serialized to stormcode.ser, but multilang files should extracted to outside of serialized file so that python/ruby/node/etc can load it. Accomplishing this with distribute mode is easy because there's always user submitted jar, and supervisor can know it is what user submitted. But accomplishing this with local mode is not easy cause supervisor cannot know user submitted jar, and users can run topology to local mode without packaging. So, Supervisor in local mode finds resource directory (resources) from each jars (which ends with jar) in classpath, and copy first occurrence to codeDir. storm jar places user topology jar to the first of classpath, so it can be run without issue. So normally, it's natural for ShellProcess to not find splitsentence.py. Maybe your working directory or PYTHONPATH do the trick. Hope this helps. Best, Jungtaek Lim (HeartSaVioR) ps. I also respond to your SO question with same content. http://stackoverflow.com/a/32085316/3480122 2015-08-10 6:49 GMT+09:00 Matthias J. Sax mj...@informatik.hu-berlin.de: Hi, I work with Storm for a while already, but want to get started with development. As suggested, I am using Intellij (up to now, I was using Eclipse). I was also looking at https://github.com/apache/storm/tree/master/examples/storm-starter#intellij-idea This documentation is not complete. I was not able to run anything in Intellij first. I could figure out, that I need to remove the scope of storm-core dependency (in storm-starter pom.xml). (found here: https://stackoverflow.com/questions/30724424/storm-starter-with-intellij-idea-maven-project-could-not-find-class ) After that I wass able to build the project. I can also run ExclamationTopology with no problems within Intellij. However, WordCountTopology fails. First I got the following error: java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read. Serializer Exception: Traceback (most recent call last): File splitsentence.py, line 16, in module import storm ImportError: No module named storm I was able to resolve it via: apt-get install python-storm (from StackOverflow) However, I don't speak Python and was wondering what the problem is and why I could resolve it like this. Just want to get deeper into it. Maybe someone can explain. Unfortunately, I am getting a different error now: java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read. Serializer Exception: Traceback (most recent call last): File splitsentence.py, line 16, in module import storm ImportError: No module named storm I did not find any solution on the Internet. And as I am not familiar with Python and never used Storm differently as low-level Java API I am stuck now. Because ExclamationTopology runs, I guess my basic setup is correct. What do I do wrong? -Matthias -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
[jira] [Commented] (STORM-958) Add config for init params of group mapping service
[ https://issues.apache.org/jira/browse/STORM-958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702384#comment-14702384 ] ASF GitHub Bot commented on STORM-958: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/649#issuecomment-132434528 +1 Add config for init params of group mapping service --- Key: STORM-958 URL: https://issues.apache.org/jira/browse/STORM-958 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor As a user, I would like a config added for initialization parameters of the group mapping service, so that I can initialize my own plug-in properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-958) Add config for init params of group mapping service
[ https://issues.apache.org/jira/browse/STORM-958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702388#comment-14702388 ] ASF GitHub Bot commented on STORM-958: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/649 Add config for init params of group mapping service --- Key: STORM-958 URL: https://issues.apache.org/jira/browse/STORM-958 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor Fix For: 0.11.0 As a user, I would like a config added for initialization parameters of the group mapping service, so that I can initialize my own plug-in properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-958) Add config for init params of group mapping service
[ https://issues.apache.org/jira/browse/STORM-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-958. Resolution: Fixed Fix Version/s: 0.11.0 Thanks [~dagit], I merged it into master. Add config for init params of group mapping service --- Key: STORM-958 URL: https://issues.apache.org/jira/browse/STORM-958 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor Fix For: 0.11.0 As a user, I would like a config added for initialization parameters of the group mapping service, so that I can initialize my own plug-in properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-958] adds config for group.mapping.serv...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/649 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-958] adds config for group.mapping.serv...
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/649#issuecomment-132429355 @HeartSaVioR: updated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (STORM-963) Frozen topology (KafkaSpout + Multilang bolt)
[
https://issues.apache.org/jira/browse/STORM-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Agarwal updated STORM-963:
---
Attachment: jstack.txt
I seem to have run into similar issue. I am using kafka spout along with a bolt
written in java. There is only one bolt in topology. I see no errors at all in
the logs. However, it behaves as if the spout thread has died. Kafka spout
nextTuple is not being called at all.
Luckily I took the jstack -
Frozen topology (KafkaSpout + Multilang bolt)
-
Key: STORM-963
URL: https://issues.apache.org/jira/browse/STORM-963
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.4, 0.9.5, 0.9.6
Environment: - VMware ESX 5.5
- Ubuntu Server 14.04 LTS (kernel 3.16.0-41-generic)
- Java (TM) SE Runtime Environment (build 1.8.0_45-b14)
- Python 2.7.6 (default, Jun 22 2015, 17:58:13)
- Zookeeper 3.4.6
Reporter: Alex Sobrino
Labels: multilang
Attachments: jstack.txt
Hi,
We've got a pretty simple topology running with Storm 0.9.5 (tried also with
0.9.4 and 0.9.6-INCUBATING) in a 3 machine cluster:
{code}kafkaSpout (3) - processBolt (12){code}
Some info:
- kafkaSpout reads from a topic with 3 partitions and 2 replications
- processBolt iterates throught the message and saves the results in MongoDB
- processBolt is implemented in Python and has a storm.log(I'm doing
something) just to add a simple debug message in the logs
- The messages can be quite big (~25-40 MB) and are in JSON format
- The kafka topic has a retention of 2 hours
- We use the same ZooKeeper cluster to both Kafka and Storm
The topology gets frozen after several hours (not days) running. We don't see
any message in the logs... In fact, the periodic message from s.k.KafkaUtils
and s.k.ZkCoordinator disapears. As you can imagine, the message from the
Bolt also dissapears. Logs are copy/pasted further on. If we redeploy the
topology everything starts to work again until it becomes frozen again.
Our kafkaSpout config is:
{code}
ZkHosts zkHosts = new ZkHosts(zkhost01:2181,zkhost02:2181,zkhost03:2181);
SpoutConfig kafkaConfig = new SpoutConfig(zkHosts, topic,
/topic/ourclientid, ourclientid);
kafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
kafkaConfig.fetchSizeBytes = 50*1024*1024;
kafkaConfig.bufferSizeBytes = 50*1024*1024;
{code}
We've also tried setting the following options
{code}
kafkaConfig.forceFromStart = true;
kafkaConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime(); // Also
with kafka.api.OffsetRequest.LatestTime();
kafkaConfig.useStartOffsetTimeIfOffsetOutOfRange = true;
{code}
Right now the topology is running without acking the messages since there's a
bug in kafkaSpout with failed messages and deleted offsets in Kafka.
This is what can be seen in the logs in one of the workers:
{code}
2015-07-23T12:37:38.008+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:37:39.079+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:37:51.013+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:37:51.091+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:38:02.684+0200 s.k.ZkCoordinator [INFO] Task [2/3] Refreshing
partition manager connections
2015-07-23T12:38:02.687+0200 s.k.DynamicBrokersReader [INFO] Read partition
info from zookeeper: GlobalPartitionInformation{partitionMap={0=kafka1:9092,
1=kafka2:9092, 2=kafka3:9092}}
2015-07-23T12:38:02.687+0200 s.k.KafkaUtils [INFO] Task [2/3] assigned
[Partition{host=kafka2, partition=1}]
2015-07-23T12:38:02.687+0200 s.k.ZkCoordinator [INFO] Task [2/3] Deleted
partition managers: []
2015-07-23T12:38:02.687+0200 s.k.ZkCoordinator [INFO] Task [2/3] New
partition managers: []
2015-07-23T12:38:02.687+0200 s.k.ZkCoordinator [INFO] Task [2/3] Finished
refreshing
2015-07-23T12:38:09.012+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:38:41.878+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:39:02.688+0200 s.k.ZkCoordinator [INFO] Task [2/3] Refreshing
partition manager connections
2015-07-23T12:39:02.691+0200 s.k.DynamicBrokersReader [INFO] Read partition
info from zookeeper: GlobalPartitionInformation{partitionMap={0=kafka1:9092,
1=kafka2:9092, 2=kafka3:9092}}
2015-07-23T12:39:02.691+0200 s.k.KafkaUtils [INFO] Task [2/3] assigned
[Partition{host=kafka2:9092, partition=1}]
2015-07-23T12:39:02.691+0200 s.k.ZkCoordinator [INFO] Task [2/3] Deleted
partition managers: []
2015-07-23T12:39:02.691+0200 s.k.ZkCoordinator [INFO]
[jira] [Commented] (STORM-976) Config storm.logback.conf.dir is specific to previous logging framework
[ https://issues.apache.org/jira/browse/STORM-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701371#comment-14701371 ] ASF GitHub Bot commented on STORM-976: -- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/684#issuecomment-132237904 Looks good to me, +1 Thanks @YvonneIronberg Config storm.logback.conf.dir is specific to previous logging framework --- Key: STORM-976 URL: https://issues.apache.org/jira/browse/STORM-976 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Priority: Minor Labels: Newbie Storm has migrated from logback to log4j2, so we should rename this config and code that uses it. https://github.com/apache/storm/blob/544e55cb8ab8878c4af500aab49bd35d4b69cd3e/storm-core/src/clj/backtype/storm/daemon/supervisor.clj#L664 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-976
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/684#issuecomment-132237904 Looks good to me, +1 Thanks @YvonneIronberg --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Comment Edited] (STORM-963) Frozen topology (KafkaSpout + Multilang bolt)
[
https://issues.apache.org/jira/browse/STORM-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701411#comment-14701411
]
Abhishek Agarwal edited comment on STORM-963 at 8/18/15 3:13 PM:
-
I seem to have run into similar issue. I am using kafka spout along with a bolt
written in java. There is only one bolt in topology. I see no errors at all in
the logs. However, it behaves as if the spout thread has died. Kafka spout
nextTuple is not being called at all.
Luckily I took the jstack. I have attached it to JIRA.
was (Author: abhishek.agarwal):
I seem to have run into similar issue. I am using kafka spout along with a bolt
written in java. There is only one bolt in topology. I see no errors at all in
the logs. However, it behaves as if the spout thread has died. Kafka spout
nextTuple is not being called at all.
Luckily I took the jstack -
Frozen topology (KafkaSpout + Multilang bolt)
-
Key: STORM-963
URL: https://issues.apache.org/jira/browse/STORM-963
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.4, 0.9.5, 0.9.6
Environment: - VMware ESX 5.5
- Ubuntu Server 14.04 LTS (kernel 3.16.0-41-generic)
- Java (TM) SE Runtime Environment (build 1.8.0_45-b14)
- Python 2.7.6 (default, Jun 22 2015, 17:58:13)
- Zookeeper 3.4.6
Reporter: Alex Sobrino
Labels: multilang
Attachments: jstack.txt
Hi,
We've got a pretty simple topology running with Storm 0.9.5 (tried also with
0.9.4 and 0.9.6-INCUBATING) in a 3 machine cluster:
{code}kafkaSpout (3) - processBolt (12){code}
Some info:
- kafkaSpout reads from a topic with 3 partitions and 2 replications
- processBolt iterates throught the message and saves the results in MongoDB
- processBolt is implemented in Python and has a storm.log(I'm doing
something) just to add a simple debug message in the logs
- The messages can be quite big (~25-40 MB) and are in JSON format
- The kafka topic has a retention of 2 hours
- We use the same ZooKeeper cluster to both Kafka and Storm
The topology gets frozen after several hours (not days) running. We don't see
any message in the logs... In fact, the periodic message from s.k.KafkaUtils
and s.k.ZkCoordinator disapears. As you can imagine, the message from the
Bolt also dissapears. Logs are copy/pasted further on. If we redeploy the
topology everything starts to work again until it becomes frozen again.
Our kafkaSpout config is:
{code}
ZkHosts zkHosts = new ZkHosts(zkhost01:2181,zkhost02:2181,zkhost03:2181);
SpoutConfig kafkaConfig = new SpoutConfig(zkHosts, topic,
/topic/ourclientid, ourclientid);
kafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
kafkaConfig.fetchSizeBytes = 50*1024*1024;
kafkaConfig.bufferSizeBytes = 50*1024*1024;
{code}
We've also tried setting the following options
{code}
kafkaConfig.forceFromStart = true;
kafkaConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime(); // Also
with kafka.api.OffsetRequest.LatestTime();
kafkaConfig.useStartOffsetTimeIfOffsetOutOfRange = true;
{code}
Right now the topology is running without acking the messages since there's a
bug in kafkaSpout with failed messages and deleted offsets in Kafka.
This is what can be seen in the logs in one of the workers:
{code}
2015-07-23T12:37:38.008+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:37:39.079+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:37:51.013+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:37:51.091+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:38:02.684+0200 s.k.ZkCoordinator [INFO] Task [2/3] Refreshing
partition manager connections
2015-07-23T12:38:02.687+0200 s.k.DynamicBrokersReader [INFO] Read partition
info from zookeeper: GlobalPartitionInformation{partitionMap={0=kafka1:9092,
1=kafka2:9092, 2=kafka3:9092}}
2015-07-23T12:38:02.687+0200 s.k.KafkaUtils [INFO] Task [2/3] assigned
[Partition{host=kafka2, partition=1}]
2015-07-23T12:38:02.687+0200 s.k.ZkCoordinator [INFO] Task [2/3] Deleted
partition managers: []
2015-07-23T12:38:02.687+0200 s.k.ZkCoordinator [INFO] Task [2/3] New
partition managers: []
2015-07-23T12:38:02.687+0200 s.k.ZkCoordinator [INFO] Task [2/3] Finished
refreshing
2015-07-23T12:38:09.012+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:38:41.878+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364,
name:processBolt I'm doing something
2015-07-23T12:39:02.688+0200 s.k.ZkCoordinator [INFO] Task [2/3] Refreshing
partition manager connections
[jira] [Updated] (STORM-996) netty-unit-tests/test-batch demonstrates out-of-order delivery
[
https://issues.apache.org/jira/browse/STORM-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated STORM-996:
--
Priority: Blocker (was: Critical)
netty-unit-tests/test-batch demonstrates out-of-order delivery
--
Key: STORM-996
URL: https://issues.apache.org/jira/browse/STORM-996
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Derek Dagit
Priority: Blocker
backtype.storm.messaging.netty-unit-test/test-batch
One example of output. Similar things happen sporadically and vary widely by
number of failed assertions.
Tuples are not just skewed, but actually seem to come in out-of-order.
{quote}
actual: (not (= 66040 66041))
at: test_runner.clj:105
expected: (= req_msg resp_msg)
actual: (not (= 66041 66042))
at: test_runner.clj:105
expected: (= req_msg resp_msg)
actual: (not (= 66042 66040))
at: test_runner.clj:105
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-954] Topology event inspector
Github user lazyval commented on a diff in the pull request: https://github.com/apache/storm/pull/673#discussion_r37303658 --- Diff: storm-core/src/clj/backtype/storm/cluster.clj --- @@ -240,7 +240,8 @@ (when cb (cb id -(defn- maybe-deserialize +;; public for stubbing in nimbus_test +(defn maybe-deserialize --- End diff -- I don't see any change in tests, are you sure it's necessary? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-954) Toplogy Event Inspector
[ https://issues.apache.org/jira/browse/STORM-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701326#comment-14701326 ] ASF GitHub Bot commented on STORM-954: -- Github user lazyval commented on a diff in the pull request: https://github.com/apache/storm/pull/673#discussion_r37303658 --- Diff: storm-core/src/clj/backtype/storm/cluster.clj --- @@ -240,7 +240,8 @@ (when cb (cb id -(defn- maybe-deserialize +;; public for stubbing in nimbus_test +(defn maybe-deserialize --- End diff -- I don't see any change in tests, are you sure it's necessary? Toplogy Event Inspector --- Key: STORM-954 URL: https://issues.apache.org/jira/browse/STORM-954 Project: Apache Storm Issue Type: Improvement Reporter: Sriharsha Chintalapani Assignee: Arun Mahadevan •Ability to view tuples flowing through the topology •Ability to turn on/off debug events without having to stop/restart topology •Default debug events is off •User should be able to select a specific Spout or Bolt and see incoming events and outgoing events •We could put a configurable numbers of events to view (e.g. last 100 events or last 1 minute) •Tuple stream to have following info •Message id, batch/transaction id, name/value pair, timestamp, acked (boolean) •All the above to be available from Storm UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-990] Refactored TimeCacheMap to extend ...
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/682#issuecomment-132271316 +1 Looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-990) Refactored TimeCacheMap to extend RotatingMap
[ https://issues.apache.org/jira/browse/STORM-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701557#comment-14701557 ] ASF GitHub Bot commented on STORM-990: -- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/682#issuecomment-132271316 +1 Looks good to me. Refactored TimeCacheMap to extend RotatingMap - Key: STORM-990 URL: https://issues.apache.org/jira/browse/STORM-990 Project: Apache Storm Issue Type: Improvement Reporter: Dean de Bree Assignee: Dean de Bree Priority: Minor Original Estimate: 1h Remaining Estimate: 1h The TimeCacheMap has been deprecated in favour of the RotatingMap class. To simplify the transition to the new class it is suggested that the TimeCacheMap be changed such that it extends the RotatingMap. This will also improves code reuse since many of the methods are shared between the two classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-976
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/684#issuecomment-132257022 @YvonneIronberg , I did not see an Apache JIRA user name for you, so I have resolved STORM-976 as Unassigned. If you would like, share your JIRA user name so that we can assign the STORM-976 to you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-976) Config storm.logback.conf.dir is specific to previous logging framework
[ https://issues.apache.org/jira/browse/STORM-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701459#comment-14701459 ] ASF GitHub Bot commented on STORM-976: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/684 Config storm.logback.conf.dir is specific to previous logging framework --- Key: STORM-976 URL: https://issues.apache.org/jira/browse/STORM-976 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Priority: Minor Labels: Newbie Storm has migrated from logback to log4j2, so we should rename this config and code that uses it. https://github.com/apache/storm/blob/544e55cb8ab8878c4af500aab49bd35d4b69cd3e/storm-core/src/clj/backtype/storm/daemon/supervisor.clj#L664 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-976) Config storm.logback.conf.dir is specific to previous logging framework
[ https://issues.apache.org/jira/browse/STORM-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701463#comment-14701463 ] ASF GitHub Bot commented on STORM-976: -- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/684#issuecomment-132257022 @YvonneIronberg , I did not see an Apache JIRA user name for you, so I have resolved STORM-976 as Unassigned. If you would like, share your JIRA user name so that we can assign the STORM-976 to you. Config storm.logback.conf.dir is specific to previous logging framework --- Key: STORM-976 URL: https://issues.apache.org/jira/browse/STORM-976 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Priority: Minor Labels: Newbie Fix For: 0.11.0 Storm has migrated from logback to log4j2, so we should rename this config and code that uses it. https://github.com/apache/storm/blob/544e55cb8ab8878c4af500aab49bd35d4b69cd3e/storm-core/src/clj/backtype/storm/daemon/supervisor.clj#L664 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-976
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/684 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Storm UI addition
I am leaning toward date/time instead of elapsed time. Maybe we could put the elapsed time in a tool tip, such that when we hover over date/time, it displays elapsed time for convenience? This way the field is still sort-able, and the red error text will still show at a glance if the error is recent ( 5min ago). -- Derek From: Jerry Peng jerry.boyang.p...@gmail.com To: dev@storm.apache.org Sent: Monday, August 17, 2015 9:04 AM Subject: Storm UI addition Hello everyone, From a Jira [STORM-949], I made a UI addition to include the column that tells you the elapsed time since error of the last error for each component on the topology summary page. I got some feedback suggesting that the this additional column should just present the date/time of the last error instead of a elapsed time. However, the date/time of the error can be viewed if the user drills down to the specific component thus I think its better to have a elapsed time. I would like some feedback to what everyone would like to see in this additional column. I have include a snapshot of what this additional column would look like with the elapsed time implemented. Best, Boyang Jerry Peng
[jira] [Resolved] (STORM-976) Config storm.logback.conf.dir is specific to previous logging framework
[ https://issues.apache.org/jira/browse/STORM-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit resolved STORM-976. --- Resolution: Fixed Fix Version/s: 0.11.0 Thanks Yvonne, I merged this to master. Config storm.logback.conf.dir is specific to previous logging framework --- Key: STORM-976 URL: https://issues.apache.org/jira/browse/STORM-976 Project: Apache Storm Issue Type: Bug Affects Versions: 0.10.0 Reporter: Derek Dagit Priority: Minor Labels: Newbie Fix For: 0.11.0 Storm has migrated from logback to log4j2, so we should rename this config and code that uses it. https://github.com/apache/storm/blob/544e55cb8ab8878c4af500aab49bd35d4b69cd3e/storm-core/src/clj/backtype/storm/daemon/supervisor.clj#L664 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
