[jira] [Commented] (STORM-1278) port backtype.storm.daemon.worker to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448475#comment-15448475
 ] 

Abhishek Agarwal commented on STORM-1278:
-

FYI, I am working on this. 

> port backtype.storm.daemon.worker to java
> -
>
> Key: STORM-1278
> URL: https://issues.apache.org/jira/browse/STORM-1278
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/worker
>  as an example



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1278) port backtype.storm.daemon.worker to java

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1278:
---

Assignee: Abhishek Agarwal

> port backtype.storm.daemon.worker to java
> -
>
> Key: STORM-1278
> URL: https://issues.apache.org/jira/browse/STORM-1278
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/worker
>  as an example



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1240) port backtype.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer-test to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418661#comment-15418661
 ] 

Abhishek Agarwal commented on STORM-1240:
-

Most of the items assigned to me are tests. LocalCluster and testing.clj is 
important class but that is blocked. Feel free to reassign if someone else 
wants to take it up immediately. 
Here is a PR for some tests which are not blocked as of now - 
https://github.com/apache/storm/pull/1625

> port  backtype.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer-test to 
> java
> --
>
> Key: STORM-1240
> URL: https://issues.apache.org/jira/browse/STORM-1240
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1282) port backtype.storm.LocalCluster to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397190#comment-15397190
 ] 

Abhishek Agarwal commented on STORM-1282:
-

[~kabhwan] mk-local-storm-cluster is still in clojure (testing.clj). 
testing.clj has lots of helper methods and some code that is dependent on 
daemons. That's why I had not started on it yet. Do you plan to start it 
immediately? 

In my opinion, the priority should be get the daemons ported first. Executor is 
still pending for code review (I have taken two passes but would like more 
people to review it). Then nimbus and worker pull requests are yet to be 
submitted. 

> port backtype.storm.LocalCluster to java
> 
>
> Key: STORM-1282
> URL: https://issues.apache.org/jira/browse/STORM-1282
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/LocalCluster.java
>  as an example



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1876) Provide option to build storm-kafka against different kafka clients

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393239#comment-15393239
 ] 

Abhishek Agarwal commented on STORM-1876:
-

Thank you [~ptgoetz]

> Provide option to build storm-kafka against different kafka clients
> ---
>
> Key: STORM-1876
> URL: https://issues.apache.org/jira/browse/STORM-1876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: documentation, storm-kafka
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
> Fix For: 2.0.0, 1.0.2, 1.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1435) Build a single jar with dependency for StormSQL dependency

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387480#comment-15387480
 ] 

Abhishek Agarwal commented on STORM-1435:
-

There are two types of jars - application jars and system jars. System jars 
have to be added implicitly by the sql implementation. Application jars should 
be added explicitly by user. Btw, I consider storm-sql-kafka an application 
jar, and storm-sql-core a system jar. 

> Build a single jar with dependency for StormSQL dependency
> --
>
> Key: STORM-1435
> URL: https://issues.apache.org/jira/browse/STORM-1435
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: Haohui Mai
>
> Currently StormSQL requires all dependency of the topology to reside in 
> either the `lib` or the `extlib` directory. It will greatly improve the 
> usability if StormSQL can provide a mechanism to pack all dependency with the 
> jar compiled from the topology.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1435) Build a single jar with dependency for StormSQL dependency

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387339#comment-15387339
 ] 

Abhishek Agarwal commented on STORM-1435:
-

I will suggest an option to specify the jars via some command line option - 
storm sql -sql.jars=

These jars can be added to classpath using the technique suggested by Bobby 
here -
http://mail-archives.us.apache.org/mod_mbox/storm-dev/201605.mbox/%3c1080588044.4838953.1463665180081.javamail.ya...@mail.yahoo.com%3E



> Build a single jar with dependency for StormSQL dependency
> --
>
> Key: STORM-1435
> URL: https://issues.apache.org/jira/browse/STORM-1435
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: Haohui Mai
>
> Currently StormSQL requires all dependency of the topology to reside in 
> either the `lib` or the `extlib` directory. It will greatly improve the 
> usability if StormSQL can provide a mechanism to pack all dependency with the 
> jar compiled from the topology.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1730) LocalCluster#shutdown() does not terminate all storm threads/thread pools.

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384230#comment-15384230
 ] 

Abhishek Agarwal commented on STORM-1730:
-

if RC3 passes the vote, we can put it in 1.0.3. I wouldn't consider this issue 
as release breaker. 

> LocalCluster#shutdown() does not terminate all storm threads/thread pools.
> --
>
> Key: STORM-1730
> URL: https://issues.apache.org/jira/browse/STORM-1730
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
> Environment: Windows 7 x64
> Oracle Java 1.8.0 u92 x64
>Reporter: Fergus Byrne
>Assignee: Fergus Byrne
>Priority: Minor
> Fix For: 2.0.0, 1.1.0
>
> Attachments: Thread Pool '47' remaining..png, storm-shutdown-issue.zip
>
>
> When using the LocalCluster in test setup.  LocalCluster#shutdown() does not 
> shutdown all executor services it starts.  In my test case, there is a single 
> thread pool executor service that is not shutdown and not daemon.  This keeps 
> the jvm alive when it is expected to terminate.
> Please see attached test case.  In my example, thread pool 47 is not 
> shutdown.  Naming here is conditional on threading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1982) Topology running Local Cluster is not properly destroyed

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384120#comment-15384120
 ] 

Abhishek Agarwal commented on STORM-1982:
-

Is there a way to reproduce this issue? 

> Topology running Local Cluster is not properly destroyed
> 
>
> Key: STORM-1982
> URL: https://issues.apache.org/jira/browse/STORM-1982
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Jungtaek Lim
>Priority: Minor
> Attachments: 15659.jstack, local-cluster-shutdown-hang.log
>
>
> The process which run LocalCluster is not properly destroyed.
> I'll attach the console log after kill is triggered, and jstack dump.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1951) Move example topologies of modules into examples directory

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal resolved STORM-1951.
-
Resolution: Duplicate

> Move example topologies of modules into examples directory
> --
>
> Key: STORM-1951
> URL: https://issues.apache.org/jira/browse/STORM-1951
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> Right now, example topologies for different modules are being put up in test 
> code of that module. This example code should be placed in a separate 
> directory. After discussion, we have agreed to have aforementioned directory 
> in examples. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1086) Make FailedMsgRetryManager configurable when setting up KafkaSpout

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal resolved STORM-1086.
-
   Resolution: Fixed
Fix Version/s: 1.1.0
   1.0.2
   2.0.0

> Make FailedMsgRetryManager configurable when setting up KafkaSpout
> --
>
> Key: STORM-1086
> URL: https://issues.apache.org/jira/browse/STORM-1086
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Nick Kleinschmidt
>Assignee: Abhishek Agarwal
>Priority: Minor
> Fix For: 2.0.0, 1.0.2, 1.1.0
>
>
> The FailedMsgRetryManager interface makes replay behavior configurable and we 
> default to using ExponentialBackoffMsgRetryManager, but there's no way to 
> define your own FailedMsgRetryManager and plug it in when setting up 
> KafkaSpout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1600) Do not report errors when the worker shutdown is in progress

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376752#comment-15376752
 ] 

Abhishek Agarwal commented on STORM-1600:
-

Incorporate the check into 
https://github.com/apache/storm/pull/1543

> Do not report errors when the worker shutdown is in progress
> 
>
> Key: STORM-1600
> URL: https://issues.apache.org/jira/browse/STORM-1600
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> Usually in a worker, some uncaught exception in an executor threads leads to 
> process exit. Process exit is not instantaneous and it first triggers the 
> shutdown. The shutdown initiation usually results in network connections 
> closing e.g. zookeeper, hdfs in other threads causing other exceptions. These 
> threads end up reporting their exceptions as well. It confuses the user who 
> can these errors on UI but not the actual root cause of shutdown hidden 
> beneath new errors. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1967) Kafka 0.8 Incompatible with Storm 1.0.1

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375236#comment-15375236
 ] 

Abhishek Agarwal commented on STORM-1967:
-

You shouldn't have to. Are you using maven to build uber jar from your 
topology? If yes, can you share your project pom?

> Kafka 0.8 Incompatible with Storm 1.0.1
> ---
>
> Key: STORM-1967
> URL: https://issues.apache.org/jira/browse/STORM-1967
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 1.0.1
>Reporter: James Finkle
>
> Had a Storm Cluster deployed and functioning with storm 0.9.5. Updated the 
> cluster and the topology to 1.0.1, leaving the same kafka version (0.8). 
> Kafka Spout would not function, threw Kafka Buffer Underflow exceptions.
> Updating to Kafka 0.9 fixed this issue, but I'm told I shouldn't have had to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372729#comment-15372729
 ] 

Abhishek Agarwal commented on STORM-1949:
-

I see. You mean to say that consumer may be slow but transfer thread will 
continue to populate receive queues. Since transfer thread doesn't slow down, 
spout too will keep pumping out the data. is that correct?

> Backpressure can cause spout to stop emitting and stall topology
> 
>
> Key: STORM-1949
> URL: https://issues.apache.org/jira/browse/STORM-1949
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Roshan Naik
>
> Problem can be reproduced by this [Word count 
> topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java]
> within a IDE.
> I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt 
> instances.
> The problem is more easily reproduced with WC topology as it causes an 
> explosion of tuples due to splitting a sentence tuple into word tuples. As 
> the bolts have to process more tuples than the  spout is producing, spout 
> needs to operate slower.
> The amount of time it takes for the topology to stall can vary.. but 
> typically under 10 mins. 
> *My theory:*  I suspect there is a race condition in the way ZK is being 
> utilized to enable/disable back pressure. When congested (i.e pressure 
> exceeds high water mark), the bolt's worker records this congested situation 
> in ZK by creating a node. Once the congestion is reduced below the low water 
> mark, it deletes this node. 
> The spout's worker has setup a watch on the parent node, expecting a callback 
> whenever there is change in the child nodes. On receiving the callback the 
> spout's worker lists the parent node to check if there are 0 or more child 
> nodes it is essentially trying to figure out the nature of state change 
> in ZK to determine whether to throttle or not. Subsequently  it setsup 
> another watch in ZK to keep an eye on future changes.
> When there are multiple bolts, there can be rapid creation/deletion of these 
> ZK nodes. Between the time the worker receives a callback and sets up the 
> next watch.. many changes may have undergone in ZK which will go unnoticed by 
> the spout. 
> The condition that the bolts are no longer congested may not get noticed as a 
> result of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371055#comment-15371055
 ] 

Abhishek Agarwal commented on STORM-1949:
-

ok. my understanding was that spout will continue emitting new tuples even if 
consumer is not consuming fast enough. But that won't happen because spout 
doesn't emit further if transfer queue is full - 
https://github.com/apache/storm/blob/1.x-branch/storm-core/src/clj/org/apache/storm/daemon/executor.clj#L647
[~kabhwan] doesn't this address your concern? 



> Backpressure can cause spout to stop emitting and stall topology
> 
>
> Key: STORM-1949
> URL: https://issues.apache.org/jira/browse/STORM-1949
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Roshan Naik
>
> Problem can be reproduced by this [Word count 
> topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java]
> within a IDE.
> I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt 
> instances.
> The problem is more easily reproduced with WC topology as it causes an 
> explosion of tuples due to splitting a sentence tuple into word tuples. As 
> the bolts have to process more tuples than the  spout is producing, spout 
> needs to operate slower.
> The amount of time it takes for the topology to stall can vary.. but 
> typically under 10 mins. 
> *My theory:*  I suspect there is a race condition in the way ZK is being 
> utilized to enable/disable back pressure. When congested (i.e pressure 
> exceeds high water mark), the bolt's worker records this congested situation 
> in ZK by creating a node. Once the congestion is reduced below the low water 
> mark, it deletes this node. 
> The spout's worker has setup a watch on the parent node, expecting a callback 
> whenever there is change in the child nodes. On receiving the callback the 
> spout's worker lists the parent node to check if there are 0 or more child 
> nodes it is essentially trying to figure out the nature of state change 
> in ZK to determine whether to throttle or not. Subsequently  it setsup 
> another watch in ZK to keep an eye on future changes.
> When there are multiple bolts, there can be rapid creation/deletion of these 
> ZK nodes. Between the time the worker receives a callback and sets up the 
> next watch.. many changes may have undergone in ZK which will go unnoticed by 
> the spout. 
> The condition that the bolts are no longer congested may not get noticed as a 
> result of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370998#comment-15370998
 ] 

Abhishek Agarwal commented on STORM-1949:
-

[~sriharsha] We can disable the back-pressure. But I want to know what will be 
the alternative for topologies with non-acking? If backpressure is not enabled, 
there should be some other way to bound the send/receive queues.

> Backpressure can cause spout to stop emitting and stall topology
> 
>
> Key: STORM-1949
> URL: https://issues.apache.org/jira/browse/STORM-1949
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Roshan Naik
>
> Problem can be reproduced by this [Word count 
> topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java]
> within a IDE.
> I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt 
> instances.
> The problem is more easily reproduced with WC topology as it causes an 
> explosion of tuples due to splitting a sentence tuple into word tuples. As 
> the bolts have to process more tuples than the  spout is producing, spout 
> needs to operate slower.
> The amount of time it takes for the topology to stall can vary.. but 
> typically under 10 mins. 
> *My theory:*  I suspect there is a race condition in the way ZK is being 
> utilized to enable/disable back pressure. When congested (i.e pressure 
> exceeds high water mark), the bolt's worker records this congested situation 
> in ZK by creating a node. Once the congestion is reduced below the low water 
> mark, it deletes this node. 
> The spout's worker has setup a watch on the parent node, expecting a callback 
> whenever there is change in the child nodes. On receiving the callback the 
> spout's worker lists the parent node to check if there are 0 or more child 
> nodes it is essentially trying to figure out the nature of state change 
> in ZK to determine whether to throttle or not. Subsequently  it setsup 
> another watch in ZK to keep an eye on future changes.
> When there are multiple bolts, there can be rapid creation/deletion of these 
> ZK nodes. Between the time the worker receives a callback and sets up the 
> next watch.. many changes may have undergone in ZK which will go unnoticed by 
> the spout. 
> The condition that the bolts are no longer congested may not get noticed as a 
> result of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370949#comment-15370949
 ] 

Abhishek Agarwal commented on STORM-1949:
-

I have the same concern as Jungtake with disabling backpressue by default. If 
my understanding is correct, transfer queues are not bounded anymore and if 
acking is not enabled, can keep growing. This will result in out of memory 
exceptions. 

> Backpressure can cause spout to stop emitting and stall topology
> 
>
> Key: STORM-1949
> URL: https://issues.apache.org/jira/browse/STORM-1949
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Roshan Naik
>
> Problem can be reproduced by this [Word count 
> topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java]
> within a IDE.
> I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt 
> instances.
> The problem is more easily reproduced with WC topology as it causes an 
> explosion of tuples due to splitting a sentence tuple into word tuples. As 
> the bolts have to process more tuples than the  spout is producing, spout 
> needs to operate slower.
> The amount of time it takes for the topology to stall can vary.. but 
> typically under 10 mins. 
> *My theory:*  I suspect there is a race condition in the way ZK is being 
> utilized to enable/disable back pressure. When congested (i.e pressure 
> exceeds high water mark), the bolt's worker records this congested situation 
> in ZK by creating a node. Once the congestion is reduced below the low water 
> mark, it deletes this node. 
> The spout's worker has setup a watch on the parent node, expecting a callback 
> whenever there is change in the child nodes. On receiving the callback the 
> spout's worker lists the parent node to check if there are 0 or more child 
> nodes it is essentially trying to figure out the nature of state change 
> in ZK to determine whether to throttle or not. Subsequently  it setsup 
> another watch in ZK to keep an eye on future changes.
> When there are multiple bolts, there can be rapid creation/deletion of these 
> ZK nodes. Between the time the worker receives a callback and sets up the 
> next watch.. many changes may have undergone in ZK which will go unnoticed by 
> the spout. 
> The condition that the bolts are no longer congested may not get noticed as a 
> result of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1281) port backtype.storm.testing to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367159#comment-15367159
 ] 

Abhishek Agarwal commented on STORM-1281:
-

[~kabhwan] It depends on components which haven't been ported already. That's 
why it hasn't been taken up. The major work is in getting executor, worker and 
nimbus ported. These are the big pieces. Executor is pending for code review. I 
had done two passes over than PR but I want someone else to take a look as well 
(given how critical executor is) 

> port  backtype.storm.testing to java
> 
>
> Key: STORM-1281
> URL: https://issues.apache.org/jira/browse/STORM-1281
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> lots of helper functions/macros for running tests.  Some might need to stay 
> in cojure with a java equivalent that can be used when other tests are ported 
> over,



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1951) Move example topologies of modules into examples directory

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1951:
---

 Summary: Move example topologies of modules into examples directory
 Key: STORM-1951
 URL: https://issues.apache.org/jira/browse/STORM-1951
 Project: Apache Storm
  Issue Type: Task
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal


Right now, example topologies for different modules are being put up in test 
code of that module. This example code should be placed in a separate 
directory. After discussion, we have agreed to have aforementioned directory in 
examples. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1235) port backtype.storm.security.auth.ReqContext-test to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365890#comment-15365890
 ] 

Abhishek Agarwal commented on STORM-1235:
-

[~Johnbaba]  can you review this PR?

> port  backtype.storm.security.auth.ReqContext-test to java
> --
>
> Key: STORM-1235
> URL: https://issues.apache.org/jira/browse/STORM-1235
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit test migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365694#comment-15365694
 ] 

Abhishek Agarwal commented on STORM-1949:
-

can we also have a thread stack taken after the topology gets stalled? It may 
be of help in debugging the problem further. 

> Backpressure can cause spout to stop emitting and stall topology
> 
>
> Key: STORM-1949
> URL: https://issues.apache.org/jira/browse/STORM-1949
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Roshan Naik
>
> Problem can be reproduced by this [Word count 
> topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java]
> within a IDE.
> I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt 
> instances.
> The problem is more easily reproduced with WC topology as it causes an 
> explosion of tuples due to splitting a sentence tuple into word tuples. As 
> the bolts have to process more tuples than the  spout is producing, spout 
> needs to operate slower.
> The amount of time it takes for the topology to stall can vary.. but 
> typically under 10 mins. 
> *My theory:*  I suspect there is a race condition in the way ZK is being 
> utilized to enable/disable back pressure. When congested (i.e pressure 
> exceeds high water mark), the bolt's worker records this congested situation 
> in ZK by creating a node. Once the congestion is reduced below the low water 
> mark, it deletes this node. 
> The spout's worker has setup a watch on the parent node, expecting a callback 
> whenever there is change in the child nodes. On receiving the callback the 
> spout's worker lists the parent node to check if there are 0 or more child 
> nodes it is essentially trying to figure out the nature of state change 
> in ZK to determine whether to throttle or not. Subsequently  it setsup 
> another watch in ZK to keep an eye on future changes.
> When there are multiple bolts, there can be rapid creation/deletion of these 
> ZK nodes. Between the time the worker receives a callback and sets up the 
> next watch.. many changes may have undergone in ZK which will go unnoticed by 
> the spout. 
> The condition that the bolts are no longer congested may not get noticed as a 
> result of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1770) Pluggable status storage in storm-Kafka

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348074#comment-15348074
 ] 

Abhishek Agarwal commented on STORM-1770:
-

That wouldn't be right [~darion] since offset as a concept exists only for 
certain type of spouts. By the way, which other spouts are using OffsetManager?

> Pluggable status storage in storm-Kafka
> ---
>
> Key: STORM-1770
> URL: https://issues.apache.org/jira/browse/STORM-1770
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 0.9.6
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> Storm-Kafka spout store consumer offset in zookeeper . When a lot topology 
> running in cluster , zookeeper snapshot frequently . This ticket is to make 
> status storage pluggable , support store in zookeeper (default) and Redis .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1903) Intermittent Travis test failures in org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1903:
---

 Summary: Intermittent Travis test failures in 
org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
 Key: STORM-1903
 URL: https://issues.apache.org/jira/browse/STORM-1903
 Project: Apache Storm
  Issue Type: Test
  Components: storm-core
Reporter: Abhishek Agarwal


testSubmitTopologyToLocalNimbus fails with following error
{noformat}
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at 
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at 
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at 
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at 
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at 
org.apache.storm.generated.Nimbus$Client.recv_getTopologyInfo(Nimbus.java:1182)
at 
org.apache.storm.generated.Nimbus$Client.getTopologyInfo(Nimbus.java:1169)
at org.apache.storm.utils.Utils.getTopologyInfo(Utils.java:1465)
at org.apache.storm.LocalCluster$submit_hook.invoke(LocalCluster.clj:44)
at 
org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:52)
at org.apache.storm.LocalCluster.submitTopology(Unknown Source)
at 
org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus(LocalNimbusTest.java:65)
{noformat}

The exception on server is 
{noformat}
283607 [pool-75-thread-3] ERROR o.a.t.s.AbstractNonblockingServer$FrameBuffer - 
Unexpected throwable while invoking!
java.lang.NullPointerException
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:26) 
~[clojure-1.7.0.jar:?]
at 
org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758$iter__4869__4873$fn__4874.invoke(nimbus.clj:1870)
 ~[classes/:?]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
at clojure.core$dorun.invoke(core.clj:3009) ~[clojure-1.7.0.jar:?]
at clojure.core$doall.invoke(core.clj:3025) ~[clojure-1.7.0.jar:?]
at 
org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfoWithOpts(nimbus.clj:1868)
 ~[classes/:?]
at 
org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfo(nimbus.clj:1907)
 ~[classes/:?]
at 
org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3748)
 ~[classes/:?]
at 
org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3732)
 ~[classes/:?]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[libthrift-0.9.3.jar:0.9.3]
at 
org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
 ~[classes/:?]
at 
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
 [libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.server.Invocation.run(Invocation.java:18) 
[libthrift-0.9.3.jar:0.9.3]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_31]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_31]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1876) Provide option to build storm-kafka against different kafka clients

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal updated STORM-1876:

Component/s: storm-kafka

> Provide option to build storm-kafka against different kafka clients
> ---
>
> Key: STORM-1876
> URL: https://issues.apache.org/jira/browse/STORM-1876
> Project: Apache Storm
>  Issue Type: Documentation
>  Components: documentation, storm-kafka
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1876) Provide option to build storm-kafka against different kafka clients

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal updated STORM-1876:

Summary: Provide option to build storm-kafka against different kafka 
clients  (was: Add documentation on FailedMsgRetryManager and storm-kafka 
interoperability with kafka clients)

> Provide option to build storm-kafka against different kafka clients
> ---
>
> Key: STORM-1876
> URL: https://issues.apache.org/jira/browse/STORM-1876
> Project: Apache Storm
>  Issue Type: Documentation
>  Components: documentation
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1876) Add documentation on FailedMsgRetryManager and storm-kafka interoperability with kafka clients

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1876:
---

 Summary: Add documentation on FailedMsgRetryManager and 
storm-kafka interoperability with kafka clients
 Key: STORM-1876
 URL: https://issues.apache.org/jira/browse/STORM-1876
 Project: Apache Storm
  Issue Type: Documentation
  Components: documentation
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1755) Revert the kafka client version upgrade in storm-kafka module

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309981#comment-15309981
 ] 

Abhishek Agarwal commented on STORM-1755:
-

Done. https://issues.apache.org/jira/browse/STORM-1876

> Revert the kafka client version upgrade in storm-kafka module
> -
>
> Key: STORM-1755
> URL: https://issues.apache.org/jira/browse/STORM-1755
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Affects Versions: 1.0.1
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
> Fix For: 2.0.0, 1.0.2, 1.1.0
>
>
> storm-kafka module does not use any feature of new API. The newer kafka 
> client (0.9.x) is not backward compatible with 0.8.x brokers and upgrading 
> storm-kafka version in topology could break the currently running topologies. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1755) Revert the kafka client version upgrade in storm-kafka module

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309932#comment-15309932
 ] 

Abhishek Agarwal commented on STORM-1755:
-

Thanks [~kabhwan] I will add something along with documentation on 
FailedMsgRetryManager

> Revert the kafka client version upgrade in storm-kafka module
> -
>
> Key: STORM-1755
> URL: https://issues.apache.org/jira/browse/STORM-1755
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Affects Versions: 1.0.1
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
> Fix For: 2.0.0, 1.0.2, 1.1.0
>
>
> storm-kafka module does not use any feature of new API. The newer kafka 
> client (0.9.x) is not backward compatible with 0.8.x brokers and upgrading 
> storm-kafka version in topology could break the currently running topologies. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1705) Cap on number of retries for a failed message in kafka spout

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306773#comment-15306773
 ] 

Abhishek Agarwal commented on STORM-1705:
-

[~spo...@salesforce.com] Making retry manager configurable was intended to 
solve the use cases you have listed. People can put up their own implementation 
of failedMsgRetryManager and can do anything with the message if number of 
retries is exhausted. These other implementations can be generic and shared as 
well. 

> Cap on number of retries for a failed message in kafka spout
> 
>
> Key: STORM-1705
> URL: https://issues.apache.org/jira/browse/STORM-1705
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-kafka
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> The kafka-spout module based on newer APIs has a cap on the number of times, 
> a message is to be retried. It will be a good feature add in the older kafka 
> spout code as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1086) Make FailedMsgRetryManager configurable when setting up KafkaSpout

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1086:
---

Assignee: Abhishek Agarwal  (was: Nick Kleinschmidt)

> Make FailedMsgRetryManager configurable when setting up KafkaSpout
> --
>
> Key: STORM-1086
> URL: https://issues.apache.org/jira/browse/STORM-1086
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Nick Kleinschmidt
>Assignee: Abhishek Agarwal
>Priority: Minor
>
> The FailedMsgRetryManager interface makes replay behavior configurable and we 
> default to using ExponentialBackoffMsgRetryManager, but there's no way to 
> define your own FailedMsgRetryManager and plug it in when setting up 
> KafkaSpout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1869) Add ability for Kafka Spout to do limited retries on failed tuples, and redirect down a "failed" stream

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306241#comment-15306241
 ] 

Abhishek Agarwal commented on STORM-1869:
-

There is already a PR open to achieve the limited retries functionality 
https://github.com/apache/storm/pull/1331

You can also implement a custom RetryManager to act on the message which has 
exhausted all the retries. 

> Add ability for Kafka Spout to do limited retries on failed tuples, and 
> redirect down a "failed" stream
> ---
>
> Key: STORM-1869
> URL: https://issues.apache.org/jira/browse/STORM-1869
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Stephen Powis
>
> h4. Summary
> KafkaSpout provides no way to limit the number of times a tuple can fail 
> within your topology.  I think it'd be great to allow configuring a hard 
> limit to the number of times a tuple can be failed before it is no longer 
> retried.
> Additionally, it'd be fantastic if the spout could do a one-time emit of this 
> tuple down a "failed" stream.  This stream would allow you to handle these 
> failed tuples in some way -- persist the failed tuple to some other storage 
> system to be replayed later?  logged?  ignored? tons of things you could 
> possibly want to do with these.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-433) Give users visibility to the depth of queues at each bolt

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-433:
--

Assignee: Abhishek Agarwal  (was: Kyle Nusbaum)

> Give users visibility to the depth of queues at each bolt
> -
>
> Key: STORM-433
> URL: https://issues.apache.org/jira/browse/STORM-433
> Project: Apache Storm
>  Issue Type: Wish
>  Components: storm-core
>Reporter: Dane Hammer
>Assignee: Abhishek Agarwal
>Priority: Minor
>
> I envision being able to browse the Storm UI and see where queues of tuples 
> are backing up.
> Today if I see latencies increasing at a bolt, it may not be due to anything 
> specific to that bolt, but that it is backed up behind an overwhelmed bolt 
> (which has too low of parallelism or too high of latency).
> I would expect this could use sampling like the metrics reported to the UI 
> today, and just retrieve data from netty about the state of the queues. I 
> wouldn't imagine supporting zeromq on the first pass.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1853) Deserialization issues in Utils.javaDeserialize()

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal updated STORM-1853:

Priority: Critical  (was: Major)

> Deserialization issues in Utils.javaDeserialize()
> -
>
> Key: STORM-1853
> URL: https://issues.apache.org/jira/browse/STORM-1853
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Critical
>
> Utils.javaDeserialize uses a custom implementation of ObjectInputStream which 
> can be inconsistent with ObjectOutputStream class used in javaSerialize. 
> One resulting issue e.g
> http://mail-archives.apache.org/mod_mbox/storm-user/201605.mbox/%3CCAGOmOn0RJ33RZ0tj-%3DoKPkqNunkS4Q2Nx0ZSKHcNAMPLowuc3w%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1705) Cap on number of retries for a failed message in kafka spout

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal updated STORM-1705:


It is not a bug but still want to be included in 1.0.2 release. 

> Cap on number of retries for a failed message in kafka spout
> 
>
> Key: STORM-1705
> URL: https://issues.apache.org/jira/browse/STORM-1705
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-kafka
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> The kafka-spout module based on newer APIs has a cap on the number of times, 
> a message is to be retried. It will be a good feature add in the older kafka 
> spout code as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1853) Deserialization issues in Utils.javaDeserialize()

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1853:
---

 Summary: Deserialization issues in Utils.javaDeserialize()
 Key: STORM-1853
 URL: https://issues.apache.org/jira/browse/STORM-1853
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Affects Versions: 1.0.0
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal


Utils.javaDeserialize uses a custom implementation of ObjectInputStream which 
can be inconsistent with ObjectOutputStream class used in javaSerialize. 

One resulting issue e.g
http://mail-archives.apache.org/mod_mbox/storm-user/201605.mbox/%3CCAGOmOn0RJ33RZ0tj-%3DoKPkqNunkS4Q2Nx0ZSKHcNAMPLowuc3w%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1277) port backtype.storm.daemon.executor to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275609#comment-15275609
 ] 

Abhishek Agarwal commented on STORM-1277:
-

[~Cody] can you publish the class hierarchy of executor that you have in mind? 
Executor class seems to be good place for using object oriented design. 

> port backtype.storm.daemon.executor to java
> ---
>
> Key: STORM-1277
> URL: https://issues.apache.org/jira/browse/STORM-1277
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Cody
>  Labels: java-migration, jstorm-merger
>
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/task
>  kind of.  Tasks and executors are combined in jstorm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1770) Pluggable status storage in storm-Kafka

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274436#comment-15274436
 ] 

Abhishek Agarwal commented on STORM-1770:
-

+1. Making the state storage pluggable will be useful. 

> Pluggable status storage in storm-Kafka
> ---
>
> Key: STORM-1770
> URL: https://issues.apache.org/jira/browse/STORM-1770
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 0.9.6
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> Storm-Kafka spout store consumer offset in zookeeper . When a lot topology 
> running in cluster , zookeeper snapshot frequently . This ticket is to make 
> status storage pluggable , support store in zookeeper (default) and Redis .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1755) Revert the kafka client version upgrade in storm-kafka module

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1755:
---

 Summary: Revert the kafka client version upgrade in storm-kafka 
module
 Key: STORM-1755
 URL: https://issues.apache.org/jira/browse/STORM-1755
 Project: Apache Storm
  Issue Type: Improvement
  Components: storm-kafka
Affects Versions: 1.0.1
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal


storm-kafka module does not use any feature of new API. The newer kafka client 
(0.9.x) is not backward compatible with 0.8.x brokers and upgrading storm-kafka 
version in topology could break the currently running topologies. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1753) Add documentation for LoadAwareShuffleGrouping

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1753:
---

 Summary: Add documentation for LoadAwareShuffleGrouping
 Key: STORM-1753
 URL: https://issues.apache.org/jira/browse/STORM-1753
 Project: Apache Storm
  Issue Type: Improvement
  Components: documentation
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1752) Correct the nimbus.host property name in 0.10.0 docs

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1752:
---

 Summary: Correct the nimbus.host property name in 0.10.0 docs
 Key: STORM-1752
 URL: https://issues.apache.org/jira/browse/STORM-1752
 Project: Apache Storm
  Issue Type: Bug
  Components: documentation
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal


http://storm.apache.org/releases/0.10.0/Setting-up-a-Storm-cluster.html

nimbus.seeds should be nimbus.host. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1751) Remove references to external repositories in storm documentation

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1751:
---

 Summary: Remove references to external repositories in storm 
documentation
 Key: STORM-1751
 URL: https://issues.apache.org/jira/browse/STORM-1751
 Project: Apache Storm
  Issue Type: Bug
  Components: documentation
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1749) Fix storm-starter links in the github code

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1749:
---

 Summary: Fix storm-starter links in the github code
 Key: STORM-1749
 URL: https://issues.apache.org/jira/browse/STORM-1749
 Project: Apache Storm
  Issue Type: Bug
  Components: examples
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal


github links in storm-starter readme.md are still pointing to the old package 
name. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1278) port backtype.storm.daemon.worker to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263622#comment-15263622
 ] 

Abhishek Agarwal commented on STORM-1278:
-

No Problem [~jark]. Thank you for update. 

> port backtype.storm.daemon.worker to java
> -
>
> Key: STORM-1278
> URL: https://issues.apache.org/jira/browse/STORM-1278
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Jark Wu
>  Labels: java-migration, jstorm-merger
>
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/worker
>  as an example



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1278) port backtype.storm.daemon.worker to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262647#comment-15262647
 ] 

Abhishek Agarwal commented on STORM-1278:
-

[~jark] Have you started working on this? Do you mind if I pick it up?

> port backtype.storm.daemon.worker to java
> -
>
> Key: STORM-1278
> URL: https://issues.apache.org/jira/browse/STORM-1278
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Jark Wu
>  Labels: java-migration, jstorm-merger
>
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/worker
>  as an example



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1276) port backtype.storm.daemon.nimbus to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262644#comment-15262644
 ] 

Abhishek Agarwal commented on STORM-1276:
-

Sounds like a good idea. Let me know which parts you have not ported and we can 
create separate JIRA for them. 

> port backtype.storm.daemon.nimbus to java
> -
>
> Key: STORM-1276
> URL: https://issues.apache.org/jira/browse/STORM-1276
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Basti Liu
>  Labels: java-migration, jstorm-merger
> Attachments: Nimbus_Diff.pdf
>
>
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/nimbus
> as a possible example



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1276) port backtype.storm.daemon.nimbus to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261881#comment-15261881
 ] 

Abhishek Agarwal commented on STORM-1276:
-

[~basti.lj] If you are not woking on this right now, do you mind if I take it 
up?

> port backtype.storm.daemon.nimbus to java
> -
>
> Key: STORM-1276
> URL: https://issues.apache.org/jira/browse/STORM-1276
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Basti Liu
>  Labels: java-migration, jstorm-merger
> Attachments: Nimbus_Diff.pdf
>
>
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/nimbus
> as a possible example



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1676) NullPointerException while serializing ClusterWorkerHearbeat

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal updated STORM-1676:

Fix Version/s: 2.0.0

> NullPointerException while serializing ClusterWorkerHearbeat
> 
>
> Key: STORM-1676
> URL: https://issues.apache.org/jira/browse/STORM-1676
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
> Fix For: 2.0.0
>
>
> `Map executor_stats` had null value in the key 
> which was causing NPE during serialization. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1676) NullPointerException while serializing ClusterWorkerHearbeat

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249683#comment-15249683
 ] 

Abhishek Agarwal commented on STORM-1676:
-

[~kabhwan] You are right. I have removed the epic links and updated the 
versions accordingly. 

> NullPointerException while serializing ClusterWorkerHearbeat
> 
>
> Key: STORM-1676
> URL: https://issues.apache.org/jira/browse/STORM-1676
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
> Fix For: 2.0.0
>
>
> `Map executor_stats` had null value in the key 
> which was causing NPE during serialization. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1676) NullPointerException while serializing ClusterWorkerHearbeat

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal updated STORM-1676:

Affects Version/s: 2.0.0

> NullPointerException while serializing ClusterWorkerHearbeat
> 
>
> Key: STORM-1676
> URL: https://issues.apache.org/jira/browse/STORM-1676
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
> Fix For: 2.0.0
>
>
> `Map executor_stats` had null value in the key 
> which was causing NPE during serialization. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1710) java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241184#comment-15241184
 ] 

Abhishek Agarwal commented on STORM-1710:
-

If you don't want to clean the older storm, you can choose an other directory 
for storm 1.0.0. To do that, change the *storm.local.dir* and 
*storm.zookeeper.root* e.g.

storm.local.dir: "storm-local-1"
storm.zookeeper.root: "/storm-1"

> java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId
> -
>
> Key: STORM-1710
> URL: https://issues.apache.org/jira/browse/STORM-1710
> Project: Apache Storm
>  Issue Type: Question
>  Components: storm-core
>Affects Versions: 1.0.0
> Environment: Linux  2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 
> EST 2013 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Guangxin Zhang
>Assignee: Robert Joseph Evans
>
> 2016-04-14 09:07:22.226 o.a.s.d.supervisor [ERROR] Error on initialization of 
> server mk-supervisor
> java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> backtype.storm.generated.LSSupervisorId
>   at org.apache.storm.utils.LocalState.deserialize(LocalState.java:83)
>   at org.apache.storm.utils.LocalState.get(LocalState.java:130)
>   at 
> org.apache.storm.local_state$ls_supervisor_id.invoke(local_state.clj:61)
>   at 
> org.apache.storm.daemon.supervisor$standalone_supervisor$reify__9646.prepare(supervisor.clj:1228)
>   at 
> org.apache.storm.daemon.supervisor$fn__9503$exec_fn__1826__auto9504.invoke(supervisor.clj:781)
>   at clojure.lang.AFn.applyToHelper(AFn.java:160)
>   at clojure.lang.AFn.applyTo(AFn.java:144)
>   at clojure.core$apply.invoke(core.clj:630)
>   at 
> org.apache.storm.daemon.supervisor$fn__9503$mk_supervisor__9548.doInvoke(supervisor.clj:779)
>   at clojure.lang.RestFn.invoke(RestFn.java:436)
>   at 
> org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1216)
>   at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1249)
>   at clojure.lang.AFn.applyToHelper(AFn.java:152)
>   at clojure.lang.AFn.applyTo(AFn.java:144)
>   at org.apache.storm.daemon.supervisor.main(Unknown Source)
> Caused by: java.lang.ClassNotFoundException: 
> backtype.storm.generated.LSSupervisorId
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:186)
>   at org.apache.storm.utils.LocalState.deserialize(LocalState.java:78)
>   ... 14 more
> 2016-04-14 09:07:22.239 o.a.s.util [ERROR] Halting process: ("Error on 
> initialization")
> java.lang.RuntimeException: ("Error on initialization")
>   at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
>   at clojure.lang.RestFn.invoke(RestFn.java:423)
>   at 
> org.apache.storm.daemon.supervisor$fn__9503$mk_supervisor__9548.doInvoke(supervisor.clj:779)
>   at clojure.lang.RestFn.invoke(RestFn.java:436)
>   at 
> org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1216)
>   at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1249)
>   at clojure.lang.AFn.applyToHelper(AFn.java:152)
>   at clojure.lang.AFn.applyTo(AFn.java:144)
>   at org.apache.storm.daemon.supervisor.main(Unknown Source)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1560) Topology stops processing after Netty catches/swallows Throwable

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236940#comment-15236940
 ] 

Abhishek Agarwal commented on STORM-1560:
-

That could possibly be because the topology workers are getting restarted 
frequently. It usually happens when there is an uncaught exception in the 
topology. 

> Topology stops processing after Netty catches/swallows Throwable
> 
>
> Key: STORM-1560
> URL: https://issues.apache.org/jira/browse/STORM-1560
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: P. Taylor Goetz
>
> In some scenarios, netty connection problems can leave a topology in an 
> unrecoverable state. The likely culprit is the Netty {{HashedWheelTimer}} 
> class that contains the following code:
> {code}
> public void expire() {
> if(this.compareAndSetState(0, 2)) {
> try {
> this.task.run(this);
> } catch (Throwable var2) {
> if(HashedWheelTimer.logger.isWarnEnabled()) {
> HashedWheelTimer.logger.warn("An exception was thrown 
> by " + TimerTask.class.getSimpleName() + '.', var2);
> }
> }
> }
> }
> {code}
> The exception being swallowed can be seen below:
> {code}
> 2016-02-18 08:46:59.116 o.a.s.m.n.Client [INFO] closing Netty Client 
> Netty-Client-/192.168.202.6:6701
> 2016-02-18 08:46:59.173 o.a.s.m.n.Client [INFO] waiting up to 60 ms to 
> send 0 pending messages to Netty-Client-/192.168.202.6:6701
> 2016-02-18 08:46:59.271 STDIO [ERROR] Feb 18, 2016 8:46:59 AM 
> org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer
> WARNING: An exception was thrown by TimerTask.
> java.lang.RuntimeException: Giving up to scheduleConnect to 
> Netty-Client-/192.168.202.6:6701 after 44 failed attempts. 3 messages were 
> lost
>   at org.apache.storm.messaging.netty.Client$Connect.run(Client.java:573)
>   at 
> org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546)
>   at 
> org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446)
>   at 
> org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395)
>   at 
> org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> The netty client then never recovers, and the follows messages repeat forever:
> {code}
> 2016-02-18 09:42:56.251 o.a.s.m.n.Client [ERROR] discarding 1 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> 2016-02-18 09:43:25.248 o.a.s.m.n.Client [ERROR] discarding 1 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> 2016-02-18 09:43:55.248 o.a.s.m.n.Client [ERROR] discarding 1 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> 2016-02-18 09:43:55.752 o.a.s.m.n.Client [ERROR] discarding 2 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> 2016-02-18 09:43:56.252 o.a.s.m.n.Client [ERROR] discarding 1 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> 2016-02-18 09:44:25.249 o.a.s.m.n.Client [ERROR] discarding 1 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1705) Cap on number of retries for a failed message in kafka spout

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1705:
---

 Summary: Cap on number of retries for a failed message in kafka 
spout
 Key: STORM-1705
 URL: https://issues.apache.org/jira/browse/STORM-1705
 Project: Apache Storm
  Issue Type: New Feature
  Components: storm-kafka
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal


The kafka-spout module based on newer APIs has a cap on the number of times, a 
message is to be retried. It will be a good feature add in the older kafka 
spout code as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1676) NullPointerException while serializing ClusterWorkerHearbeat

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1676:
---

Assignee: Abhishek Agarwal

> NullPointerException while serializing ClusterWorkerHearbeat
> 
>
> Key: STORM-1676
> URL: https://issues.apache.org/jira/browse/STORM-1676
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> `Map executor_stats` had null value in the key 
> which was causing NPE during serialization. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1693) Negative counts in the UI for __metrics stream

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15231870#comment-15231870
 ] 

Abhishek Agarwal commented on STORM-1693:
-

[~revans2] can you help me in debugging this? Since `_bucketStart` is being 
updated every `600/19` seconds, why would timeSpent be higher than 600 seconds. 

> Negative counts in the UI for __metrics stream
> --
>
> Key: STORM-1693
> URL: https://issues.apache.org/jira/browse/STORM-1693
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Critical
> Attachments: Screen Shot 2016-04-07 at 7.05.01 PM.png
>
>
> Metrics reported by UI are not correct. I am seeing negative counts for 
> output stats in the bolt. The same application code works fine on 0.9.6 
> version of storm. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (STORM-1693) Negative counts in the UI for __metrics stream

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230352#comment-15230352
 ] 

Abhishek Agarwal edited comment on STORM-1693 at 4/7/16 2:52 PM:
-

I put up some code to catch any negative percentage here -
https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/metric/internal/CountStatAndMetric.java#L189

``
 if (pct < 0) {
 throw new IllegalStateException("This is never expected. " + timeSpent 
+ "#" + timeNeeded + "#" + bucketTime[i]);
 }
``

The exception was thrown in the topology with following statements
{noformat}
java.lang.IllegalStateException: This is never expected. 602745#-2745#0
java.lang.IllegalStateException: This is never expected. 602421#-2421#0
java.lang.IllegalStateException: This is never expected. 602467#-2467#0
{noformat}


was (Author: abhishek.agarwal):
I put up some code to catch any negative percentage here -
https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/metric/internal/CountStatAndMetric.java#L189

```
 if (pct < 0) {
 throw new IllegalStateException("This is never expected. " + timeSpent 
+ "#" + timeNeeded + "#" + bucketTime[i]);
 }
```

The exception was thrown in the topology with following statements
{noformat}
java.lang.IllegalStateException: This is never expected. 602745#-2745#0
java.lang.IllegalStateException: This is never expected. 602421#-2421#0
java.lang.IllegalStateException: This is never expected. 602467#-2467#0
{noformat}

> Negative counts in the UI for __metrics stream
> --
>
> Key: STORM-1693
> URL: https://issues.apache.org/jira/browse/STORM-1693
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Critical
> Attachments: Screen Shot 2016-04-07 at 7.05.01 PM.png
>
>
> Metrics reported by UI are not correct. I am seeing negative counts for 
> output stats in the bolt. The same application code works fine on 0.9.6 
> version of storm. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1693) Negative counts in the UI for __metrics stream

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230352#comment-15230352
 ] 

Abhishek Agarwal commented on STORM-1693:
-

I put up some code to catch any negative percentage here -
https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/metric/internal/CountStatAndMetric.java#L189

```
 if (pct < 0) {
 throw new IllegalStateException("This is never expected. " + timeSpent 
+ "#" + timeNeeded + "#" + bucketTime[i]);
 }
```

The exception was thrown in the topology with following statements
{noformat}
java.lang.IllegalStateException: This is never expected. 602745#-2745#0
java.lang.IllegalStateException: This is never expected. 602421#-2421#0
java.lang.IllegalStateException: This is never expected. 602467#-2467#0
{noformat}

> Negative counts in the UI for __metrics stream
> --
>
> Key: STORM-1693
> URL: https://issues.apache.org/jira/browse/STORM-1693
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Critical
> Attachments: Screen Shot 2016-04-07 at 7.05.01 PM.png
>
>
> Metrics reported by UI are not correct. I am seeing negative counts for 
> output stats in the bolt. The same application code works fine on 0.9.6 
> version of storm. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1693) Negative counts in the UI for __metrics stream

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1693:
---

Assignee: Abhishek Agarwal

> Negative counts in the UI for __metrics stream
> --
>
> Key: STORM-1693
> URL: https://issues.apache.org/jira/browse/STORM-1693
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Critical
> Attachments: Screen Shot 2016-04-07 at 7.05.01 PM.png
>
>
> Metrics reported by UI are not correct. I am seeing negative counts for 
> output stats in the bolt. The same application code works fine on 0.9.6 
> version of storm. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1693) Negative counts in the UI for __metrics stream

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal updated STORM-1693:

Attachment: Screen Shot 2016-04-07 at 7.05.01 PM.png

> Negative counts in the UI for __metrics stream
> --
>
> Key: STORM-1693
> URL: https://issues.apache.org/jira/browse/STORM-1693
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>Priority: Critical
> Attachments: Screen Shot 2016-04-07 at 7.05.01 PM.png
>
>
> Metrics reported by UI are not correct. I am seeing negative counts for 
> output stats in the bolt. The same application code works fine on 0.9.6 
> version of storm. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1693) Negative counts in the UI for __metrics stream

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1693:
---

 Summary: Negative counts in the UI for __metrics stream
 Key: STORM-1693
 URL: https://issues.apache.org/jira/browse/STORM-1693
 Project: Apache Storm
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Abhishek Agarwal
Priority: Critical


Metrics reported by UI are not correct. I am seeing negative counts for output 
stats in the bolt. The same application code works fine on 0.9.6 version of 
storm. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1680) Provide configuration to set min fetch size in KafkaSpout

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1680:
---

Assignee: Abhishek Agarwal

> Provide configuration to set min fetch size in KafkaSpout
> -
>
> Key: STORM-1680
> URL: https://issues.apache.org/jira/browse/STORM-1680
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka, trident
>Reporter: Sachin Pasalkar
>Assignee: Abhishek Agarwal
>
> Kafka consumer has provided the configuration to set minimum fetch size. 
> However, storms kafka spout is not exposing these functionality.  This is 
> helpful in some case where someone writing data to hdfs & want file size of 
> X. 
> Below are changes needs to be done
> 1.In KafkaUtils class update fetchMessages API with below change
> FetchRequest fetchRequest = builder.addFetch(topic, partitionId, offset, 
> config.fetchSizeBytes).clientId(config.clientId).maxWait(config.fetchMaxWait).minBytes(config.minFetchByte).build();
> 2. Update KafkaConfig class with instance variable as minFetchByte 
> (Default value is 0 as mentioned in FetchRequestBuilder class)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1676) NullPointerException while serializing ClusterWorkerHearbeat

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1676:
---

 Summary: NullPointerException while serializing 
ClusterWorkerHearbeat
 Key: STORM-1676
 URL: https://issues.apache.org/jira/browse/STORM-1676
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Reporter: Abhishek Agarwal


`Map executor_stats` had null value in the key 
which was causing NPE during serialization. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1675) Ability to submit multiple jars from client to topology

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1675:
---

 Summary: Ability to submit multiple jars from client to topology
 Key: STORM-1675
 URL: https://issues.apache.org/jira/browse/STORM-1675
 Project: Apache Storm
  Issue Type: New Feature
  Components: storm-core
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal


Though submitting shaded jar works in most of the cases, in some cases such an 
ability may be very helpful. e.g. storm sql project. It currently packs the 
classes into one jar and submits it as a single topology jar. A generic 
capability may be helpful for other tools such as storm-sql. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1656) Acking-Framework-implementation.md is not checked in the new documentation

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1656:
---

 Summary: Acking-Framework-implementation.md is not checked in the 
new documentation
 Key: STORM-1656
 URL: https://issues.apache.org/jira/browse/STORM-1656
 Project: Apache Storm
  Issue Type: Bug
  Components: documentation
Reporter: Abhishek Agarwal


Looks like Acking-framework-implementation was missed out while the 
documentation was moved to svn. I can't find it in the storm documentation and 
the link has been removed from the implementation-docs.html. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (STORM-1537) Upgrade to Kryo 3

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reopened STORM-1537:
-

Reopening so that changes are also merged into master branch. 

> Upgrade to Kryo 3
> -
>
> Key: STORM-1537
> URL: https://issues.apache.org/jira/browse/STORM-1537
> Project: Apache Storm
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Oscar Boykin
>Assignee: Abhishek Agarwal
> Fix For: 1.0.0
>
>
> In storm, Kryo (2.21) is used for serialization:
> https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L231
> The user must use the same version storm does, or there will be a java class 
> error at runtime.
> Storm depends on a quasi-abandoned library: carbonite:
> https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L210
> which depends on Kryo 2.21 and Twitter chill 0.3.6:
> https://github.com/sritchie/carbonite/blob/master/project.clj#L8
> Chill, currently on 0.7.3, would like to upgrade to Kryo 3.0.3:
> https://github.com/twitter/chill/pull/245
> because Spark, also depending on chill, would like to upgrade for performance 
> improvements and bugfixes.
> https://issues.apache.org/jira/browse/SPARK-11416
> Unfortunately, summingbird depends on storm:
> https://github.com/twitter/summingbird/blob/develop/build.sbt#L34
> so, if chill is upgraded, and that gets on the classpath, summingbird will 
> break at runtime.
> I propose:
> 1) copy the carbonite code into storm. It is likely the only consumer.
> 2) bump the storm kryo dependency after chill upgrades: recall that storm 
> actually depends on chill-java. A dependency that could possibly be removed 
> after you pull carbonite in.
> 3) once a new version of storm is published, summingbird (and scalding) can 
> upgrade to the latest chill.
> Also, I hope for:
> 4) we as a JVM community get better about classpath isolation and versioning. 
> Diamonds like this in one big classpath make large codebases very fragile.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1537) Upgrade to Kryo 3

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1537:
---

Assignee: Abhishek Agarwal  (was: P. Taylor Goetz)

> Upgrade to Kryo 3
> -
>
> Key: STORM-1537
> URL: https://issues.apache.org/jira/browse/STORM-1537
> Project: Apache Storm
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Oscar Boykin
>Assignee: Abhishek Agarwal
> Fix For: 1.0.0
>
>
> In storm, Kryo (2.21) is used for serialization:
> https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L231
> The user must use the same version storm does, or there will be a java class 
> error at runtime.
> Storm depends on a quasi-abandoned library: carbonite:
> https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L210
> which depends on Kryo 2.21 and Twitter chill 0.3.6:
> https://github.com/sritchie/carbonite/blob/master/project.clj#L8
> Chill, currently on 0.7.3, would like to upgrade to Kryo 3.0.3:
> https://github.com/twitter/chill/pull/245
> because Spark, also depending on chill, would like to upgrade for performance 
> improvements and bugfixes.
> https://issues.apache.org/jira/browse/SPARK-11416
> Unfortunately, summingbird depends on storm:
> https://github.com/twitter/summingbird/blob/develop/build.sbt#L34
> so, if chill is upgraded, and that gets on the classpath, summingbird will 
> break at runtime.
> I propose:
> 1) copy the carbonite code into storm. It is likely the only consumer.
> 2) bump the storm kryo dependency after chill upgrades: recall that storm 
> actually depends on chill-java. A dependency that could possibly be removed 
> after you pull carbonite in.
> 3) once a new version of storm is published, summingbird (and scalding) can 
> upgrade to the latest chill.
> Also, I hope for:
> 4) we as a JVM community get better about classpath isolation and versioning. 
> Diamonds like this in one big classpath make large codebases very fragile.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1645) nimbus.thrift.port command line argument leads to java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206201#comment-15206201
 ] 

Abhishek Agarwal commented on STORM-1645:
-

This problem exists only for the 0.10.x branch and it is a minor bug. So I am 
not completely sure if this needs to be fixed. 

> nimbus.thrift.port command line argument leads to 
> java.lang.ClassCastException: java.lang.Long cannot be cast to 
> java.lang.Integer
> --
>
> Key: STORM-1645
> URL: https://issues.apache.org/jira/browse/STORM-1645
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.10.0
>Reporter: Waatze Goldewijk
>Assignee: Abhishek Agarwal
>
> When you supply the commandline parameter for a custom Nimbus thrift port, 
> the number value is interpreted as a Long, but internally used as an Integer. 
> This leads to a ClassCastException.
> This is executed (redacted):
> /opt/storm/apache-storm-0.10.0/bin/storm kill -w 10 -c 
> nimbus.thrift.port=6627 -c nimbus.host=vm0009 #{topology}"
> This is the output:
>  ** [out :: vm0009] 2627 [main] INFO  b.s.u.Utils - Using defaults.yaml from 
> resources
>  ** [out :: vm0009] 2795 [main] INFO  b.s.u.Utils - Using storm.yaml from 
> resources
>  ** [out :: vm0009] 4262 [main] INFO  b.s.u.Utils - Using defaults.yaml from 
> resources
>  ** [out :: vm0009] 4287 [main] INFO  b.s.u.Utils - Using storm.yaml from 
> resources
>  ** [out :: vm0009] 4328 [main] INFO  b.s.thrift - Connecting to Nimbus at 
> vm0009:6627 as user:
>  ** [out :: vm0009] 4328 [main] INFO  b.s.u.Utils - Using defaults.yaml from 
> resources
>  ** [out :: vm0009] 4348 [main] INFO  b.s.u.Utils - Using storm.yaml from 
> resources
>  ** [out :: vm0009] Exception in thread "main"
>  ** [out :: vm0009] java.lang.ClassCastException: java.lang.Long cannot be 
> cast to java.lang.Integer
>  ** [out :: vm0009] 
>  ** [out :: vm0009] at 
> backtype.storm.thrift$nimbus_client_and_conn.invoke(thrift.clj:75)
>  ** [out :: vm0009] 
>  ** [out :: vm0009] at 
> backtype.storm.thrift$nimbus_client_and_conn.invoke(thrift.clj:72)
>  ** [out :: vm0009] 
>  ** [out :: vm0009] 
>  ** [out :: vm0009] at 
> backtype.storm.command.kill_topology$_main.doInvoke(kill_topology.clj:26)
>  ** [out :: vm0009] 
>  ** [out :: vm0009] at clojure.lang.RestFn.applyTo(RestFn.java:137)
>  ** [out :: vm0009] 
>  ** [out :: vm0009] at backtype.storm.command.kill_topology.main(Unknown 
> Source)
>  ** [out :: vm0009]
> I have seen other related issues: 
> https://issues.apache.org/jira/browse/STORM-1578
> I believe this is the same issue (internally using an Integer, but converting 
> the input to a Long) in a different area.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1514) add ability to configure read only option for storm UI

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1514:
---

Assignee: Abhishek Agarwal

> add ability to configure read only option for storm UI
> --
>
> Key: STORM-1514
> URL: https://issues.apache.org/jira/browse/STORM-1514
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-ui
>Reporter: Arpit Gupta
>Assignee: Abhishek Agarwal
> Fix For: 1.0.0
>
>
> Today in a secure env one has to be logged in as storm before they can view 
> the UI. It would be nice if one could configure read only access to specific 
> users and/or groups that way users could view the progress being made by 
> various topologies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1625) Move storm-sql dependencies out of lib folder

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal updated STORM-1625:

Summary: Move storm-sql dependencies out of lib folder  (was: guava 
dependency is being shipped inside storm distribution)

> Move storm-sql dependencies out of lib folder
> -
>
> Key: STORM-1625
> URL: https://issues.apache.org/jira/browse/STORM-1625
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core, storm-sql
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> We shade guava classes inside storm-core so that client can work with any 
> version of guava without any clashes. However, storm-sql-core has a 
> transitive dependency on guava and thus the guava jar is still getting 
> shipped in lib folder. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1640) Provide an option to allow aggregating metrics before being reported

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1640:
---

 Summary: Provide an option to allow aggregating metrics before 
being reported
 Key: STORM-1640
 URL: https://issues.apache.org/jira/browse/STORM-1640
 Project: Apache Storm
  Issue Type: Improvement
  Components: storm-core
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1234) port backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test to java

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1234:
---

Assignee: Abhishek Agarwal

> port  backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test to java
> 
>
> Key: STORM-1234
> URL: https://issues.apache.org/jira/browse/STORM-1234
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> to junit test conversion



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1251) port backtype.storm.serialization.SerializationFactory-test to java

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1251:
---

Assignee: Abhishek Agarwal

> port  backtype.storm.serialization.SerializationFactory-test to java
> 
>
> Key: STORM-1251
> URL: https://issues.apache.org/jira/browse/STORM-1251
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1256) port backtype.storm.utils.ZookeeperServerCnxnFactory-test to java

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1256:
---

Assignee: Abhishek Agarwal

> port backtype.storm.utils.ZookeeperServerCnxnFactory-test to java
> -
>
> Key: STORM-1256
> URL: https://issues.apache.org/jira/browse/STORM-1256
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1240) port backtype.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer-test to java

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1240:
---

Assignee: Abhishek Agarwal

> port  backtype.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer-test to 
> java
> --
>
> Key: STORM-1240
> URL: https://issues.apache.org/jira/browse/STORM-1240
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1235) port backtype.storm.security.auth.ReqContext-test to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199647#comment-15199647
 ] 

Abhishek Agarwal commented on STORM-1235:
-

[~revans2] [~knusbaum] can you take a look at the above PR?

> port  backtype.storm.security.auth.ReqContext-test to java
> --
>
> Key: STORM-1235
> URL: https://issues.apache.org/jira/browse/STORM-1235
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit test migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1637) MongoDB pom.xml fails on JDK8 Travis

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal resolved STORM-1637.
-
Resolution: Fixed

> MongoDB pom.xml fails on JDK8 Travis
> 
>
> Key: STORM-1637
> URL: https://issues.apache.org/jira/browse/STORM-1637
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-mongodb
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>Priority: Critical
>
> https://travis-ci.org/apache/storm/builds/116573238
> JDK7 builds pass JDK8 builds fail with.
> {code}
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]   
> [ERROR]   The project org.apache.storm:storm-mongodb:[unknown-version] 
> (/home/travis/build/apache/storm/external/storm-mongodb/pom.xml) has 1 error
> [ERROR] Non-resolvable parent POM: Could not transfer artifact 
> org.apache.storm:storm:pom:2.0.0-SNAPSHOT from/to codehaus-snapshots 
> (https://nexus.codehaus.org/snapshots/): nexus.codehaus.org and 
> 'parent.relativePath' points at wrong local POM @ line 21, column 13: Unknown 
> host nexus.codehaus.org -> [Help 2]
> [ERROR] 
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR] 
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> [ERROR] [Help 2] 
> http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1637) MongoDB pom.xml fails on JDK8 Travis

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1637:
---

Assignee: Abhishek Agarwal

> MongoDB pom.xml fails on JDK8 Travis
> 
>
> Key: STORM-1637
> URL: https://issues.apache.org/jira/browse/STORM-1637
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-mongodb
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>Priority: Critical
>
> https://travis-ci.org/apache/storm/builds/116573238
> JDK7 builds pass JDK8 builds fail with.
> {code}
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]   
> [ERROR]   The project org.apache.storm:storm-mongodb:[unknown-version] 
> (/home/travis/build/apache/storm/external/storm-mongodb/pom.xml) has 1 error
> [ERROR] Non-resolvable parent POM: Could not transfer artifact 
> org.apache.storm:storm:pom:2.0.0-SNAPSHOT from/to codehaus-snapshots 
> (https://nexus.codehaus.org/snapshots/): nexus.codehaus.org and 
> 'parent.relativePath' points at wrong local POM @ line 21, column 13: Unknown 
> host nexus.codehaus.org -> [Help 2]
> [ERROR] 
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR] 
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> [ERROR] [Help 2] 
> http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1271) port backtype.storm.daemon.task to java

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196948#comment-15196948
 ] 

Abhishek Agarwal commented on STORM-1271:
-

[~Johnbaba] I have already started working on this. Some parts are pending on 
stats and builtin-metrics to be ported. stats have already been ported and 
builtin-metrics is under review. So I shall send a PR soon enough. 

> port backtype.storm.daemon.task to java
> ---
>
> Key: STORM-1271
> URL: https://issues.apache.org/jira/browse/STORM-1271
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> helper functions for task data and sending tuples



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1622) Topology jars referring to shaded classes of older version of storm jar cannot be run in Storm 1.0.0

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193355#comment-15193355
 ] 

Abhishek Agarwal commented on STORM-1622:
-

I think it should be ok to just add new mappings in the StormShadeRequest. 
Depending on the 0.10 doesn't need much changes in topology since the 
store-core packages are still the same. I will add a method for printing 
warning about not using private classes of storm. 

There is another problem related to jar dependency - STORM-1625


> Topology jars referring to shaded classes of older version of storm jar 
> cannot be run in Storm 1.0.0
> 
>
> Key: STORM-1622
> URL: https://issues.apache.org/jira/browse/STORM-1622
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> This commit 
> https://github.com/apache/storm/commit/12fdaa01b20eb144bf18a231a05c92873c90aa09
> changes the package names of shaded classes inside the storm.  These classes 
> are shipped inside the maven release of storm-core jar and can depended upon 
> the topology jar. Jar built with older version of storm-core and using this 
> dependency, wouldn't run on newer version of storm cluster. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1625) guava dependency is being shipped inside storm distribution

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1625:
---

 Summary: guava dependency is being shipped inside storm 
distribution
 Key: STORM-1625
 URL: https://issues.apache.org/jira/browse/STORM-1625
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core, storm-sql
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal


We shade guava classes inside storm-core so that client can work with any 
version of guava without any clashes. However, storm-sql-core has a transitive 
dependency on guava and thus the guava jar is still getting shipped in lib 
folder. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1618) Add the option of passing config directory apart from the file

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal resolved STORM-1618.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

> Add the option of passing config directory apart from the file
> --
>
> Key: STORM-1618
> URL: https://issues.apache.org/jira/browse/STORM-1618
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Trivial
> Fix For: 2.0.0
>
>
> The error message, if configuration file doesn't exist, is also confusing. 
> `Error: Cannot find configuration directory: /etc/storm/conf`
> the binary is actually looking for a file. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-1619) Add the option of passing config directory apart from the file

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal closed STORM-1619.
---
Resolution: Duplicate

> Add the option of passing config directory apart from the file
> --
>
> Key: STORM-1619
> URL: https://issues.apache.org/jira/browse/STORM-1619
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Trivial
>
> The error message, if configuration file doesn't exist, is also confusing. 
> `Error: Cannot find configuration directory: /etc/storm/conf`
> the binary is actually looking for a file. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-971) Storm-Kafka: Emit metric for messages lost due to kafka retention

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal updated STORM-971:
---
Description: In case of TopicOffsetOutOfRange exception, it is useful to 
know just how many unread messages were lost.   (was: {noformat}
if (offset < _emittedToOffset - _spoutConfig.maxOffsetBehind) {
LOG.info(
"Skipping failed tuple at offset=" + offset +
" because it's more than maxOffsetBehind=" + 
_spoutConfig.maxOffsetBehind +
" behind _emittedToOffset=" + _emittedToOffset
);
}
{noformat}
I believe, having a count metric for skipped messages will help configure the 
correct maxOffsetBehind for the topology. Since the skipped messages could 
potentially be lost, number is important for the user)

> Storm-Kafka: Emit metric for messages lost due to kafka retention
> -
>
> Key: STORM-971
> URL: https://issues.apache.org/jira/browse/STORM-971
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> In case of TopicOffsetOutOfRange exception, it is useful to know just how 
> many unread messages were lost. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-971) Storm-Kafka: Emit metric for messages lost due to kafka retention

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal updated STORM-971:
---
Summary: Storm-Kafka: Emit metric for messages lost due to kafka retention  
(was: Storm-Kafka: Emit metric for failed messages which are not retried )

> Storm-Kafka: Emit metric for messages lost due to kafka retention
> -
>
> Key: STORM-971
> URL: https://issues.apache.org/jira/browse/STORM-971
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> {noformat}
> if (offset < _emittedToOffset - _spoutConfig.maxOffsetBehind) {
> LOG.info(
> "Skipping failed tuple at offset=" + offset +
> " because it's more than maxOffsetBehind=" + 
> _spoutConfig.maxOffsetBehind +
> " behind _emittedToOffset=" + _emittedToOffset
> );
> }
> {noformat}
> I believe, having a count metric for skipped messages will help configure the 
> correct maxOffsetBehind for the topology. Since the skipped messages could 
> potentially be lost, number is important for the user



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1618) Add the option of passing config directory apart from the file

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1618:
---

 Summary: Add the option of passing config directory apart from the 
file
 Key: STORM-1618
 URL: https://issues.apache.org/jira/browse/STORM-1618
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal
Priority: Trivial


The error message, if configuration file doesn't exist, is also confusing. 
`Error: Cannot find configuration directory: /etc/storm/conf`

the binary is actually looking for a file. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1619) Add the option of passing config directory apart from the file

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1619:
---

 Summary: Add the option of passing config directory apart from the 
file
 Key: STORM-1619
 URL: https://issues.apache.org/jira/browse/STORM-1619
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal
Priority: Trivial


The error message, if configuration file doesn't exist, is also confusing. 
`Error: Cannot find configuration directory: /etc/storm/conf`

the binary is actually looking for a file. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1249) port backtype.storm.security.serialization.BlowfishTupleSerializer-test to java

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1249:
---

Assignee: Abhishek Agarwal

> port  backtype.storm.security.serialization.BlowfishTupleSerializer-test to 
> java
> 
>
> Key: STORM-1249
> URL: https://issues.apache.org/jira/browse/STORM-1249
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> unit test migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1338) Evaluate/Port JStorm worker classloader

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1338:
---

Assignee: Abhishek Agarwal

> Evaluate/Port JStorm worker classloader
> ---
>
> Key: STORM-1338
> URL: https://issues.apache.org/jira/browse/STORM-1338
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: jstorm-merger
>
> Need to see if we want to continue with shading for classpath isolation, or 
> do we want to adopt the JStorm worker classloader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1237) port backtype.storm.security.auth.ThriftClient-test to java

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1237:
---

Assignee: Abhishek Agarwal

> port  backtype.storm.security.auth.ThriftClient-test to java
> 
>
> Key: STORM-1237
> URL: https://issues.apache.org/jira/browse/STORM-1237
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1235) port backtype.storm.security.auth.ReqContext-test to java

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1235:
---

Assignee: Abhishek Agarwal

> port  backtype.storm.security.auth.ReqContext-test to java
> --
>
> Key: STORM-1235
> URL: https://issues.apache.org/jira/browse/STORM-1235
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit test migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1236) port backtype.storm.security.auth.SaslTransportPlugin-test to java

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1236:
---

Assignee: Abhishek Agarwal

> port  backtype.storm.security.auth.SaslTransportPlugin-test to java
> ---
>
> Key: STORM-1236
> URL: https://issues.apache.org/jira/browse/STORM-1236
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1238) port backtype.storm.security.auth.ThriftServer-test to java

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1238:
---

Assignee: Abhishek Agarwal

> port backtype.storm.security.auth.ThriftServer-test to java
> ---
>
> Key: STORM-1238
> URL: https://issues.apache.org/jira/browse/STORM-1238
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> junit migration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1537) Upgrade to Kryo 3

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185700#comment-15185700
 ] 

Abhishek Agarwal commented on STORM-1537:
-

I tried out Storm with newer versions of kryo and carbonite. All the tests 
passed. 
Here are the changes 
https://github.com/apache/storm/compare/1.x-branch...abhishekagarwal87:kryo3-1.x?expand=1

> Upgrade to Kryo 3
> -
>
> Key: STORM-1537
> URL: https://issues.apache.org/jira/browse/STORM-1537
> Project: Apache Storm
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Oscar Boykin
>Assignee: P. Taylor Goetz
>
> In storm, Kryo (2.21) is used for serialization:
> https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L231
> The user must use the same version storm does, or there will be a java class 
> error at runtime.
> Storm depends on a quasi-abandoned library: carbonite:
> https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L210
> which depends on Kryo 2.21 and Twitter chill 0.3.6:
> https://github.com/sritchie/carbonite/blob/master/project.clj#L8
> Chill, currently on 0.7.3, would like to upgrade to Kryo 3.0.3:
> https://github.com/twitter/chill/pull/245
> because Spark, also depending on chill, would like to upgrade for performance 
> improvements and bugfixes.
> https://issues.apache.org/jira/browse/SPARK-11416
> Unfortunately, summingbird depends on storm:
> https://github.com/twitter/summingbird/blob/develop/build.sbt#L34
> so, if chill is upgraded, and that gets on the classpath, summingbird will 
> break at runtime.
> I propose:
> 1) copy the carbonite code into storm. It is likely the only consumer.
> 2) bump the storm kryo dependency after chill upgrades: recall that storm 
> actually depends on chill-java. A dependency that could possibly be removed 
> after you pull carbonite in.
> 3) once a new version of storm is published, summingbird (and scalding) can 
> upgrade to the latest chill.
> Also, I hope for:
> 4) we as a JVM community get better about classpath isolation and versioning. 
> Diamonds like this in one big classpath make large codebases very fragile.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1595) 'Fail' messages get stuck somewhere

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185450#comment-15185450
 ] 

Abhishek Agarwal commented on STORM-1595:
-

Hi [~knusbaum]  can you add the topology configuration here?

> 'Fail' messages get stuck somewhere 
> 
>
> Key: STORM-1595
> URL: https://issues.apache.org/jira/browse/STORM-1595
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Kyle Nusbaum
> Attachments: screenshot-1.png
>
>
> 'Fail' acks seem to be getting stuck somewhere between the acker and the 
> spout. 
> After a long time - sometimes multiple minutes - the fails show up in the 
> spout.
> I tested this on master and 1.x-branch and it occurs in both places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1600) Do not report errors when the worker shutdown is in progress

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1600:
---

 Summary: Do not report errors when the worker shutdown is in 
progress
 Key: STORM-1600
 URL: https://issues.apache.org/jira/browse/STORM-1600
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal


Usually in a worker, some uncaught exception in an executor threads leads to 
process exit. Process exit is not instantaneous and it first triggers the 
shutdown. The shutdown initiation usually results in network connections 
closing e.g. zookeeper, hdfs in other threads causing other exceptions. These 
threads end up reporting their exceptions as well. It confuses the user who can 
these errors on UI but not the actual root cause of shutdown hidden beneath new 
errors. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1598) Replace metric separator colon (:) with dot (.)

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1598:
---

 Summary: Replace metric separator colon (:) with dot (.)
 Key: STORM-1598
 URL: https://issues.apache.org/jira/browse/STORM-1598
 Project: Apache Storm
  Issue Type: Improvement
  Components: storm-core
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal
Priority: Minor


Metric collection systems such as graphite work well with dot as a path 
separator in metric names. Right now we use : such as 
drpc:num-http-requests etc. This can be replaced with drpc.num-http-requests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1595) 'Fail' messages get stuck somewhere

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177174#comment-15177174
 ] 

Abhishek Agarwal commented on STORM-1595:
-

can you tell us the following configurations in your topology
 - topology.message.timeout.secs
- topology.tick.tuple.freq.secs
- async flag and batch size for kafka bolt

My hunch is that with kafka bolt fails a tuple in a callback which takes some 
time. In this time, init entry is timed out from acker bolt. when fail entry 
comes to the acker bolt, it will also get timed out eventually. Tuple will be 
marked failed after `topology.message.timeout.secs` even though the bolt fails 
it earlier. Though this hypothesis may be invalid if you are not overriding the 
topology.tick.tuple.freq.secs.

> 'Fail' messages get stuck somewhere 
> 
>
> Key: STORM-1595
> URL: https://issues.apache.org/jira/browse/STORM-1595
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Kyle Nusbaum
> Attachments: screenshot-1.png
>
>
> 'Fail' acks seem to be getting stuck somewhere between the acker and the 
> spout. 
> After a long time - sometimes multiple minutes - the fails show up in the 
> spout.
> I tested this on master and 1.x-branch and it occurs in both places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1597) Provide a separate metric for timed out tuples in the spout

[Abhishek Agarwal (JIRA)]

Abhishek Agarwal created STORM-1597:
---

 Summary: Provide a separate metric for timed out tuples in the 
spout
 Key: STORM-1597
 URL: https://issues.apache.org/jira/browse/STORM-1597
 Project: Apache Storm
  Issue Type: Improvement
  Components: storm-core
Reporter: Abhishek Agarwal
Assignee: Abhishek Agarwal


The metric may be useful to distinguish whether the failures are happening in 
bolt or due to delay in ack. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1593) Nimbus indicator for when a Topology finished processing all tuples

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175203#comment-15175203
 ] 

Abhishek Agarwal commented on STORM-1593:
-

It sounds useful. Just to confirm the exact use case, you want to kill the 
topology only after all un-acked messages have been acked. 

> Nimbus indicator for when a Topology finished processing all tuples
> ---
>
> Key: STORM-1593
> URL: https://issues.apache.org/jira/browse/STORM-1593
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Michael Schonfeld
>Priority: Minor
>
> Every time we want to update topologies, we routinely find ourselves waiting 
> aimlessly for topologies to "fully finish" processing. We never truly know 
> when a topology is actually still processing tuples, and when it's really 
> done... Unless of course we wait for a full 10m window showing zeros in 
> Nimbus's topology stats table.
> I think it'd be beneficial to add some sort of a "Green" indicator in Nimbus, 
> showing when a deactivated topology has ~0 tuples ringing through it. Would 
> using the queue send/rcv population metric be correct for this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1588) component page gets divide by 0 if no event loggers configured

[Abhishek Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Agarwal reassigned STORM-1588:
---

Assignee: Abhishek Agarwal

> component page gets divide by 0 if no event loggers configured
> --
>
> Key: STORM-1588
> URL: https://issues.apache.org/jira/browse/STORM-1588
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>Priority: Blocker
>
> If you run a topology with no event loggers configured the component page 
> gets a divide by zero trying to calculate things about the event loggers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1560) Topology stops processing after Netty catches/swallows Throwable

[Abhishek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170539#comment-15170539
 ] 

Abhishek Agarwal commented on STORM-1560:
-

I also checked without any success. One possible cause could be context 
returning a stale value of client. (host-port --> connections) map inside the 
context class is not thread safe. There are synchronized blocks but they do not 
guarantee visibility consistency. 
However, other than shutdown, I could not see any scenario wherein the put and 
get to this map is called from different thread. 

> Topology stops processing after Netty catches/swallows Throwable
> 
>
> Key: STORM-1560
> URL: https://issues.apache.org/jira/browse/STORM-1560
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: P. Taylor Goetz
>Priority: Blocker
>
> In some scenarios, netty connection problems can leave a topology in an 
> unrecoverable state. The likely culprit is the Netty {{HashedWheelTimer}} 
> class that contains the following code:
> {code}
> public void expire() {
> if(this.compareAndSetState(0, 2)) {
> try {
> this.task.run(this);
> } catch (Throwable var2) {
> if(HashedWheelTimer.logger.isWarnEnabled()) {
> HashedWheelTimer.logger.warn("An exception was thrown 
> by " + TimerTask.class.getSimpleName() + '.', var2);
> }
> }
> }
> }
> {code}
> The exception being swallowed can be seen below:
> {code}
> 2016-02-18 08:46:59.116 o.a.s.m.n.Client [INFO] closing Netty Client 
> Netty-Client-/192.168.202.6:6701
> 2016-02-18 08:46:59.173 o.a.s.m.n.Client [INFO] waiting up to 60 ms to 
> send 0 pending messages to Netty-Client-/192.168.202.6:6701
> 2016-02-18 08:46:59.271 STDIO [ERROR] Feb 18, 2016 8:46:59 AM 
> org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer
> WARNING: An exception was thrown by TimerTask.
> java.lang.RuntimeException: Giving up to scheduleConnect to 
> Netty-Client-/192.168.202.6:6701 after 44 failed attempts. 3 messages were 
> lost
>   at org.apache.storm.messaging.netty.Client$Connect.run(Client.java:573)
>   at 
> org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546)
>   at 
> org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446)
>   at 
> org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395)
>   at 
> org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> The netty client then never recovers, and the follows messages repeat forever:
> {code}
> 2016-02-18 09:42:56.251 o.a.s.m.n.Client [ERROR] discarding 1 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> 2016-02-18 09:43:25.248 o.a.s.m.n.Client [ERROR] discarding 1 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> 2016-02-18 09:43:55.248 o.a.s.m.n.Client [ERROR] discarding 1 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> 2016-02-18 09:43:55.752 o.a.s.m.n.Client [ERROR] discarding 2 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> 2016-02-18 09:43:56.252 o.a.s.m.n.Client [ERROR] discarding 1 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> 2016-02-18 09:44:25.249 o.a.s.m.n.Client [ERROR] discarding 1 messages 
> because the Netty client to Netty-Client-/192.168.202.6:6701 is being closed
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)