[GitHub] storm pull request: [STORM-1217] - making small fixes in RAS

[d2r]

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/890#issuecomment-159430061
  
Merging this since comments are address, >= 2 +1's, one by PMC.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1217) making small fixes in RAS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025605#comment-15025605
 ] 

ASF GitHub Bot commented on STORM-1217:
---

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/890#issuecomment-159430061
  
Merging this since comments are address, >= 2 +1's, one by PMC.


> making small fixes in RAS
> -
>
> Key: STORM-1217
> URL: https://issues.apache.org/jira/browse/STORM-1217
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[Roshan Naik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roshan Naik updated STORM-1351:
---
Description: 
A spout can be having a problem generating a  tuple in nextTuple()  because 
 -a) there is no data to generate currently 
 - b) there is some I/O  issues it is experiencing

If the spout returns immediately from the nextTuple() call then the nextTuple() 
will be invoked immediately leading to CPU spike. The CPU spike would last till 
the situation is remedied by new coming data or the i/o issues getting resolved.

Currently to work around this, the spouts will have to implement a exponential 
backoff mechanism internally. There are two problems with this:
 - each spout needs to implement this backoff logic
 - since nextTuple() has an internal sleep and takes longer to return, the 
latency metrics computation gets thrown off


*Thoughts for Solution:*
The spout should be able to indicate a 'no data',  'experiencing error' or 'all 
good' status back to the caller of nextTuple() so that the right backoff logic 
can kick in.

- The most natural way to do this is using the return type of the nextTuple 
method. Currently nextTuple() returns void.  However, this will break source 
and binary compat since the new storm will not be able to invoke the methods on 
the unmodified spouts. This breaking change can only be considered as an option 
only prior to v1.0. 

- Alternatively this can be done by providing an additional method on the 
collector to indicate the condition to the topology runner. The spout can 
invoke this explicitly. the metrics can then also account for 'no data' and 
'error attempts'

- Alternatively - The toplogy  runner may just examine the collector if there 
was new data generated by the nextTuple() call. In this case it cannot 
distinguish between errors v/s no incoming data. 

  was:
A spout can be having a problem generating a  tuple in nextTuple()  because 
 -a) there is no data to generate currently 
 - b) there is some I/O  issues it is experiencing

If the spout returns immediately from the nextTuple() call then the nextTuple() 
will be invoked immediately leading to CPU spike. The CPU spike would last till 
the situation is remedied by new coming data or the i/o issues getting resolved.

Currently to work around this, the spouts will have to implement a exponential 
backoff mechanism internally. There are two problems with this:
 - each spout needs to implement this backoff logic
 - since nextTuple() has an internal sleep and takes longer to return, the 
latency metrics computation gets thrown off


*Thoughts for Solution:*
The spout should be able to indicate a 'backoff',  'experiencing error' or 'all 
good' status back to the caller of nextTuple() so that the right backoff logic 
can kick in.

- The most natural way to do this is using the return type of the nextTuple 
method. Currently nextTuple() returns void.  However, this will break source 
and binary compat since the new storm will not be able to invoke the methods on 
the unmodified spouts. This breaking change can only be considered as an option 
only prior to v1.0. 

- Alternatively this can be done by providing an additional method on the 
collector to indicate the condition to the topology runner. The spout can 
invoke this explicitly. the metrics can then also account for 'no data' and 
'error attempts'

- Alternatively - The toplogy  runner may just examine the collector if there 
was new data generated by the nextTuple() call. In this case it cannot 
distinguish between errors v/s no incoming data. 


> Storm spouts and bolts need a way to communicate problems back to toplogy 
> runner
> 
>
> Key: STORM-1351
> URL: https://issues.apache.org/jira/browse/STORM-1351
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Roshan Naik
>
> A spout can be having a problem generating a  tuple in nextTuple()  because 
>  -a) there is no data to generate currently 
>  - b) there is some I/O  issues it is experiencing
> If the spout returns immediately from the nextTuple() call then the 
> nextTuple() will be invoked immediately leading to CPU spike. The CPU spike 
> would last till the situation is remedied by new coming data or the i/o 
> issues getting resolved.
> Currently to work around this, the spouts will have to implement a 
> exponential backoff mechanism internally. There are two problems with this:
>  - each spout needs to implement this backoff logic
>  - since nextTuple() has an internal sleep and takes longer to return, the 
> latency metrics computation gets thrown off
> *Thoughts for Solution:*
> The spout should be able to indicate a 'no data',  'experiencing error' or 
> 'all good' status back to the caller of nextTuple() so that 

[jira] [Commented] (STORM-1217) making small fixes in RAS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025613#comment-15025613
 ] 

ASF GitHub Bot commented on STORM-1217:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/890


> making small fixes in RAS
> -
>
> Key: STORM-1217
> URL: https://issues.apache.org/jira/browse/STORM-1217
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1217] - making small fixes in RAS

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/890


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-1217) making small fixes in RAS

[Derek Dagit (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Derek Dagit resolved STORM-1217.

   Resolution: Fixed
Fix Version/s: 0.11.0

Thanks [~jerrypeng], I merged this to master.

> making small fixes in RAS
> -
>
> Key: STORM-1217
> URL: https://issues.apache.org/jira/browse/STORM-1217
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1217) making small fixes in RAS

[Derek Dagit (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Derek Dagit updated STORM-1217:
---
Component/s: storm-core

> making small fixes in RAS
> -
>
> Key: STORM-1217
> URL: https://issues.apache.org/jira/browse/STORM-1217
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[Roshan Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025659#comment-15025659
 ] 

Roshan Naik commented on STORM-1351:


Similar problem exists with Bolts too .. for example a Hive Bolt may not be 
able to write to Hive due to some intermittent network issues.  Bolts will not 
have 'no incoming data' case though.

> Storm spouts and bolts need a way to communicate problems back to toplogy 
> runner
> 
>
> Key: STORM-1351
> URL: https://issues.apache.org/jira/browse/STORM-1351
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Roshan Naik
>
> A spout can be having a problem generating a  tuple in nextTuple()  because 
>  -a) there is no data to generate currently 
>  - b) there is some I/O  issues it is experiencing
> If the spout returns immediately from the nextTuple() call then the 
> nextTuple() will be invoked immediately leading to CPU spike. The CPU spike 
> would last till the situation is remedied by new coming data or the i/o 
> issues getting resolved.
> Currently to work around this, the spouts will have to implement a 
> exponential backoff mechanism internally. There are two problems with this:
>  - each spout needs to implement this backoff logic
>  - since nextTuple() has an internal sleep and takes longer to return, the 
> latency metrics computation gets thrown off
> *Thoughts for Solution:*
> The spout should be able to indicate a 'backoff',  'experiencing error' or 
> 'all good' status back to the caller of nextTuple() so that the right backoff 
> logic can kick in.
> - The most natural way to do this is using the return type of the nextTuple 
> method. Currently nextTuple() returns void.  However, this will break source 
> and binary compat since the new storm will not be able to invoke the methods 
> on the unmodified spouts. This breaking change can only be considered as an 
> option only prior to v1.0. 
> - Alternatively this can be done by providing an additional method on the 
> collector to indicate the condition to the topology runner. The spout can 
> invoke this explicitly. the metrics can then also account for 'no data' and 
> 'error attempts'
> - Alternatively - The toplogy  runner may just examine the collector if there 
> was new data generated by the nextTuple() call. In this case it cannot 
> distinguish between errors v/s no incoming data. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[Roshan Naik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roshan Naik reassigned STORM-1351:
--

Assignee: Roshan Naik

> Storm spouts and bolts need a way to communicate problems back to toplogy 
> runner
> 
>
> Key: STORM-1351
> URL: https://issues.apache.org/jira/browse/STORM-1351
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>
> A spout can be having a problem generating a  tuple in nextTuple()  because 
>  -a) there is no data to generate currently 
>  - b) there is some I/O  issues it is experiencing
> If the spout returns immediately from the nextTuple() call then the 
> nextTuple() will be invoked immediately leading to CPU spike. The CPU spike 
> would last till the situation is remedied by new coming data or the i/o 
> issues getting resolved.
> Currently to work around this, the spouts will have to implement a 
> exponential backoff mechanism internally. There are two problems with this:
>  - each spout needs to implement this backoff logic
>  - since nextTuple() has an internal sleep and takes longer to return, the 
> latency metrics computation gets thrown off
> *Thoughts for Solution:*
> The spout should be able to indicate a 'no data',  'experiencing error' or 
> 'all good' status back to the caller of nextTuple() so that the right backoff 
> logic can kick in.
> - The most natural way to do this is using the return type of the nextTuple 
> method. Currently nextTuple() returns void.  However, this will break source 
> and binary compat since the new storm will not be able to invoke the methods 
> on the unmodified spouts. This breaking change can only be considered as an 
> option only prior to v1.0. 
> - Alternatively this can be done by providing an additional method on the 
> collector to indicate the condition to the topology runner. The spout can 
> invoke this explicitly. the metrics can then also account for 'no data' and 
> 'error attempts'
> - Alternatively - The toplogy  runner may just examine the collector if there 
> was new data generated by the nextTuple() call. In this case it cannot 
> distinguish between errors v/s no incoming data. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1218) Use Markdown Syntax for JavaDoc

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025543#comment-15025543
 ] 

ASF GitHub Bot commented on STORM-1218:
---

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/891#issuecomment-159422903
  
**Note:** This pull request originally included the following commit: 
https://github.com/apache/storm/commit/e03b28ca6222f61471b4acb1a6086aab9b597b8a

Earlier today I inadvertently pushed that commit to master. I have since 
reverted it in master. The last commit in this PR re-adds what was reverted, 
but only in my repository.


> Use Markdown Syntax for JavaDoc
> ---
>
> Key: STORM-1218
> URL: https://issues.apache.org/jira/browse/STORM-1218
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: P. Taylor Goetz
>Assignee: P. Taylor Goetz
>
> Currently we use the standard HTML-based markup for our JavaDoc 
> documentation, though many JavaDoc comments are not properly formatted. A 
> good example of this multi-paragraph comments without {{}} tags, which 
> leads to multiple paragraphs being squashed together.
> An easy fix for this is to switch to markdown for JavaDoc comments.
> Markdown format is also much easier to read in the context of code comments, 
> and it's the format we use for our website and documentation.
> Switching to markdown may also encourage developers to better document their 
> code, since it is less cumbersome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1218: Use markdown for JavaDoc

[ptgoetz]

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/891#issuecomment-159422903
  
**Note:** This pull request originally included the following commit: 
https://github.com/apache/storm/commit/e03b28ca6222f61471b4acb1a6086aab9b597b8a

Earlier today I inadvertently pushed that commit to master. I have since 
reverted it in master. The last commit in this PR re-adds what was reverted, 
but only in my repository.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[Roshan Naik (JIRA)]

Roshan Naik created STORM-1351:
--

 Summary: Storm spouts and bolts need a way to communicate problems 
back to toplogy runner
 Key: STORM-1351
 URL: https://issues.apache.org/jira/browse/STORM-1351
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Reporter: Roshan Naik


A spout can be having a problem generating a  tuple in nextTuple()  because 
 -a) there is no data to generate currently 
 - b) there is some I/O  issues it is experiencing

If the spout returns immediately from the nextTuple() call then the nextTuple() 
will be invoked immediately leading to CPU spike. The CPU spike would last till 
the situation is remedied by new coming data or the i/o issues getting resolved.

Currently to work around this, the spouts will have to implement a exponential 
backoff mechanism internally. There are two problems with this:
 - each spout needs to implement this backoff logic
 - since nextTuple() has an internal sleep and takes longer to return, the 
latency metrics computation gets thrown off


*Thoughts for Solution:*
The spout should be able to indicate a 'backoff',  'experiencing error' or 'all 
good' status back to the caller of nextTuple() so that the right backoff logic 
can kick in.

- The most natural way to do this is using the return type of the nextTuple 
method. Currently nextTuple() returns void.  However, this will break source 
and binary compat since the new storm will not be able to invoke the methods on 
the unmodified spouts. This breaking change can only be considered as an option 
only prior to v1.0. 

- Alternatively this can be done by providing an additional method on the 
collector to indicate the condition to the topology runner. The spout can 
invoke this explicitly. the metrics can then also account for 'no data' and 
'error attempts'

- Alternatively - The toplogy  runner may just examine the collector if there 
was new data generated by the nextTuple() call. In this case it cannot 
distinguish between errors v/s no incoming data. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1222. Support Kafka as external tables i...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/896


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1200) Support collations of primary keys

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025952#comment-15025952
 ] 

ASF GitHub Bot commented on STORM-1200:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/901#issuecomment-159457879
  
Thanks @haohui 


> Support collations of primary keys
> --
>
> Key: STORM-1200
> URL: https://issues.apache.org/jira/browse/STORM-1200
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> This jira proposes to add support of specifying collations of primary keys. 
> Collations provide information on uniqueness and monotonicity of columns. The 
> information is essential to implement aggregation functions over streaming 
> data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1200. Support collations of primary keys...

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/901#issuecomment-159457879
  
Thanks @haohui 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: Exclude .idea directory from RAT check

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/898


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1222) Support Kafka as external tables in StormSQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025929#comment-15025929
 ] 

ASF GitHub Bot commented on STORM-1222:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/896#issuecomment-159456248
  
Thanks @haohui merged into STORM-1040


> Support Kafka as external tables in StormSQL
> 
>
> Key: STORM-1222
> URL: https://issues.apache.org/jira/browse/STORM-1222
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> This jira propose to support Kafka as both the data ingress and egress point 
> in StormSQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1320) Port JStorm scheduler to storm

[John Fang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Fang reassigned STORM-1320:


Assignee: John Fang

> Port JStorm scheduler to storm
> --
>
> Key: STORM-1320
> URL: https://issues.apache.org/jira/browse/STORM-1320
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: John Fang
>  Labels: jstorm-merger
>
> Port the JStorm scheduler to storm core.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1200. Support collations of primary keys...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/901


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (STORM-1321) Evaluate/Port JStorm nimbus HA

[John Fang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Fang reassigned STORM-1321:


Assignee: John Fang

> Evaluate/Port JStorm nimbus HA
> --
>
> Key: STORM-1321
> URL: https://issues.apache.org/jira/browse/STORM-1321
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: John Fang
>  Labels: jstorm-merger
>
> The JStorm nimbus HA has been in production longer then the storm nimbus HA 
> has.  we should evaluate the pros and cons of switching all or in part to the 
> JStorm version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1279) port backtype.storm.daemon.supervisor to java

[John Fang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Fang reassigned STORM-1279:


Assignee: John Fang

> port backtype.storm.daemon.supervisor to java
> -
>
> Key: STORM-1279
> URL: https://issues.apache.org/jira/browse/STORM-1279
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: John Fang
>  Labels: java-migration, jstorm-merger
>
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor
>  as an example
> backtype.storm.event usage should be replaced with built-in java threadpools.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1350) Unable to build: Too many files with unapproved licenses (RAT exception)

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025991#comment-15025991
 ] 

Jungtaek Lim commented on STORM-1350:
-

[~rudra.sharma]
It seems resolved in current master branch.
Could you verify and close this issue to "Invalid"?

> Unable to build: Too many files with unapproved licenses (RAT exception)
> 
>
> Key: STORM-1350
> URL: https://issues.apache.org/jira/browse/STORM-1350
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
> Environment: Mac OSX
>Reporter: Rudra Sharma
>Priority: Minor
> Fix For: 0.11.0
>
>
> Issue when making build from the master branch. 
> [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check 
> (default) on project storm: Too many files with unapproved license: 7 See RAT 
> report in: /Users/myusername/Documents/apache-storm/target/rat.txt -> [Help 1]
> Simple workaround is to build with maven flag to skip rat check:
> mvn clean install -Drat.skip=true



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1329) Evaluate/Port JStorm metrics system

[Cody (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cody reassigned STORM-1329:
---

Assignee: Cody

> Evaluate/Port JStorm metrics system
> ---
>
> Key: STORM-1329
> URL: https://issues.apache.org/jira/browse/STORM-1329
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Cody
>  Labels: jstorm-merger
>
> Storm:
> IMetric and IMetricConsumer
> JStorm:
> All levels of metrics, including stream metrics, task metrics, component 
> metrics, topology metrics, even cluster metrics, are sampled & calculated. 
> Some metrics, e.g. ""tuple life cycle"", are very useful for debugging and 
> finding the hotspots of a topology.
> Support full metrics data. Previous metric system can only display mean 
> value of meters/histograms, the new metric system can display m1, m5, m15 of 
> meters, and common percentiles of histograms.
> Use new metrics windows, the mininum metric window is 1 minute, thus we 
> can see the metrics data every single minute.
> Supplies a metric uploader interface, third-party companies can easily 
> build their own metric systems based on the historic metric data.
> Ideally we should have a way to display most/system metrics in the UI.  
> IMetric is too generic to make this happen, but we cannot completely drop 
> support for it.  But perhaps we need to depricate it if the JStorm metrics 
> are much better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1200) Support collations of primary keys

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025950#comment-15025950
 ] 

ASF GitHub Bot commented on STORM-1200:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/901


> Support collations of primary keys
> --
>
> Key: STORM-1200
> URL: https://issues.apache.org/jira/browse/STORM-1200
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> This jira proposes to add support of specifying collations of primary keys. 
> Collations provide information on uniqueness and monotonicity of columns. The 
> information is essential to implement aggregation functions over streaming 
> data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1222. Support Kafka as external tables i...

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/896#issuecomment-159456248
  
Thanks @haohui merged into STORM-1040


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (STORM-1333) Evaluate/port Jstorm metricsMonitor command

[Cody (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cody reassigned STORM-1333:
---

Assignee: Cody

> Evaluate/port Jstorm metricsMonitor command
> ---
>
> Key: STORM-1333
> URL: https://issues.apache.org/jira/browse/STORM-1333
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Cody
>  Labels: jstorm-merger
>
> Allows toggling on/off some metrics which may impact topology performance
> Evaluate the best way to work this into the current storm code.  We may need 
> to do something with TopologyMaster to make it work well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1328) Evaluate/Port JStorm tuple groupings

[John Fang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Fang reassigned STORM-1328:


Assignee: John Fang

> Evaluate/Port JStorm tuple groupings
> 
>
> Key: STORM-1328
> URL: https://issues.apache.org/jira/browse/STORM-1328
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: Robert Joseph Evans
>Assignee: John Fang
>  Labels: jstorm-merger
>
> Storm:
> Load aware balancing in shuffle grouping
> JStorm:
> Has a "localfirst" grouping that causes tuples to be sent to the tasks in 
> the same worker by default. But if the load of all local tasks is high, the 
> tuples will be sent out to remote tasks.
> Improve localOrShuffle grouping from Storm. In Storm's localOrShuffle 
> grouping the definition of "local" is local within the same worker process. 
> i.e., if there is a bolt that the component can send to in the current worker 
> process it will send the tuples there. If there is not one, it will do round 
> robin between all of the instances of that bolt no matter which hosts they 
> are on. JStorm has extended that so that other workers/JVMs on the same host 
> are considered "local" as well, taking into account the load of the network 
> connections on the local worker.
> We should look at combining both of these to have shuffle look at both 
> distance and load to decide where to send a tuple, in addition the the new 
> JStorm localfirst grouping.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1328) Evaluate/Port JStorm tuple groupings

[JIRA]

 [ 
https://issues.apache.org/jira/browse/STORM-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

冯健 reassigned STORM-1328:
-

Assignee: 冯健  (was: John Fang)

> Evaluate/Port JStorm tuple groupings
> 
>
> Key: STORM-1328
> URL: https://issues.apache.org/jira/browse/STORM-1328
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: Robert Joseph Evans
>Assignee: 冯健
>  Labels: jstorm-merger
>
> Storm:
> Load aware balancing in shuffle grouping
> JStorm:
> Has a "localfirst" grouping that causes tuples to be sent to the tasks in 
> the same worker by default. But if the load of all local tasks is high, the 
> tuples will be sent out to remote tasks.
> Improve localOrShuffle grouping from Storm. In Storm's localOrShuffle 
> grouping the definition of "local" is local within the same worker process. 
> i.e., if there is a bolt that the component can send to in the current worker 
> process it will send the tuples there. If there is not one, it will do round 
> robin between all of the instances of that bolt no matter which hosts they 
> are on. JStorm has extended that so that other workers/JVMs on the same host 
> are considered "local" as well, taking into account the load of the network 
> connections on the local worker.
> We should look at combining both of these to have shuffle look at both 
> distance and load to decide where to send a tuple, in addition the the new 
> JStorm localfirst grouping.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Plan for Merging JStorm Code

[Cody Innowhere]

Sorry I've mistaken it with the apache issues account, it's ok now, my id
is: cody.

On Tue, Nov 24, 2015 at 11:54 PM, P. Taylor Goetz  wrote:

> Hi Cody,
>
> I wasn’t able to find your username. Are you sure you have an account on
> cwiki.apache.org?
>
> -Taylor
>
> > On Nov 22, 2015, at 8:46 AM, Cody Innowhere  wrote:
> >
> > Hi Taylor,
> > I'd like to help too, could you add me in? my id is: Cody
> >
> > On Sat, Nov 21, 2015 at 11:51 AM, 刘键(Basti Liu) <
> basti...@alibaba-inc.com>
> > wrote:
> >
> >> Hi Taylor，
> >>
> >> Sorry for the late response.
> >> I'd like to help on this. Could you please help to give me the
> permission?
> >> Thanks.
> >> UserName: basti.lj
> >>
> >> Regards
> >> Basti
> >>
> >> -Original Message-
> >> From: P. Taylor Goetz [mailto:ptgo...@gmail.com]
> >> Sent: Thursday, November 19, 2015 6:24 AM
> >> To: dev@storm.apache.org; Bobby Evans
> >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code
> >>
> >> All I have at this point is a placeholder wiki entry [1], and a lot of
> >> local notes that likely would only make sense to me.
> >>
> >> Let me know your wiki username and I’ll give you permissions. The same
> >> goes for anyone else who wants to help.
> >>
> >> -Taylor
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109
> >>
> >>> On Nov 18, 2015, at 2:08 PM, Bobby Evans 
> >> wrote:
> >>>
> >>> Taylor and others I was hoping to get started filing JIRA and planning
> >>> on how we are going to do the java migration + JStorm merger.  Is
> >>> anyone else starting to do this?  If not would anyone object to me
> >>> starting on it? - Bobby
> >>>
> >>>
> >>>   On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz <
> >> ptgo...@gmail.com> wrote:
> >>>
> >>>
> >>> Thanks for putting this together Basti, that comparison helps a lot.
> >>>
> >>> And thanks Bobby for converting it into markdown. I was going to just
> >> attach the spreadsheet to JIRA, but markdown is a much better solution.
> >>>
> >>> -Taylor
> >>>
>  On Nov 12, 2015, at 12:03 PM, Bobby Evans  >
> >> wrote:
> 
>  I translated the excel spreadsheet into a markdown file and put up a
> >> pull request for it.
>  https://github.com/apache/storm/pull/877
>  I did a few edits to it to make it work with Markdown, and to add in a
> >> few of my own comments.  I also put in a field for JIRAs to be able to
> >> track the migration.
>  Overall I think your evaluation was very good.  We have a fair amount
> >> of work ahead of us to decide what version of various features we want
> to
> >> go forward with.
>   - Bobby
> 
> 
> On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) <
> >> basti...@alibaba-inc.com> wrote:
> 
> 
>  Hi Bobby & Jungtaek,
> 
>  Thanks for your replay.
>  I totally agree that compatibility is the most important thing.
> >> Actually, JStorm has been compatible with the user API of Storm.
>  As you mentioned below, we indeed still have some features different
> >> between Storm and JStorm. I have tried to list them (minor update or
> >> improvements are not included).
>  Please refer to attachment for details. If any missing, please help
>  to point out. (The current working features are probably missing
> here.)
> >> Just have a look at these differences. For the missing features in
> JStorm,
> >> I did not see any obstacle which will block the merge to JStorm.
>  For the features which has different solution between Storm and
> JStorm,
> >> we can evaluate the solution one by one to decision which one is
> >> appropriate.
>  After the finalization of evaluation, I think JStorm team can take the
> >> merging job and publish a stable release in 2 months.
>  But anyway, the detailed implementation for these features with
> >> different solution is transparent to user. So, from user's point of
> view,
> >> there is not any compatibility problem.
> 
>  Besides compatibility, by our experience, stability is also important
> >> and is not an easy job. 4 people in JStorm team took almost one year to
> >> finish the porting from "clojure core"
>  to "java core", and to make it stable. Of course, we have many devs in
> >> community to make the porting job faster. But it still needs a long
> time to
> >> run many online complex topologys to find bugs and fix them. So, that is
> >> the reason why I proposed to do merging and build on a stable "java
> core".
> 
>  -Original Message-
>  From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID]
>  Sent: Wednesday, November 11, 2015 10:51 PM
>  To: dev@storm.apache.org
>  Subject: Re: [DISCUSS] Plan for Merging JStorm Code
> 
>  +1 for doing a 1.0 release based off of the clojure 0.11.x code.
> >> Migrating the APIs to org.apache.storm is a big non-backwards compatible
> >> 

[jira] [Assigned] (STORM-1311) port backtype.storm.ui.core to java

[Jark Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu reassigned STORM-1311:
--

Assignee: Jark Wu

> port backtype.storm.ui.core to java
> ---
>
> Key: STORM-1311
> URL: https://issues.apache.org/jira/browse/STORM-1311
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Jark Wu
>  Labels: java-migration, jstorm-merger
>
> User Interface + REST -> java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1330) Evaluate Port JStorm advanced Rebalancing

[JIRA]

 [ 
https://issues.apache.org/jira/browse/STORM-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

冯健 reassigned STORM-1330:
-

Assignee: 冯健

> Evaluate Port JStorm advanced Rebalancing
> -
>
> Key: STORM-1330
> URL: https://issues.apache.org/jira/browse/STORM-1330
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: 冯健
>  Labels: jstorm-merger
>
> JStorm:
> Besides rebalance, scale-out/in by updating the number of workers, ackers, 
> spouts & bolts dynamically without stopping topology. Routing is updated 
> dynamically within upstream components.
> dynamic routing with some groupings is difficult to get right when there is 
> state, we need to be sure this is well documented, and might want to disallow 
> it for some groupings without a force flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1331) evalute/port jstorm list command

[JIRA]

 [ 
https://issues.apache.org/jira/browse/STORM-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

冯健 reassigned STORM-1331:
-

Assignee: 冯健

> evalute/port jstorm list command
> 
>
> Key: STORM-1331
> URL: https://issues.apache.org/jira/browse/STORM-1331
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: 冯健
>  Labels: jstorm-merger
>
> JStorm:
> List information of all topologies, all supervisors, and JStorm version
> More info is good, but we want it human readable too. Possibly suing multiple 
> commands instead of just one.  Perhaps with a machine readable option too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1335) Evaluate/port JStorm update_topolgy command

[JIRA]

 [ 
https://issues.apache.org/jira/browse/STORM-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

冯健 reassigned STORM-1335:
-

Assignee: 冯健

> Evaluate/port JStorm update_topolgy command
> ---
>
> Key: STORM-1335
> URL: https://issues.apache.org/jira/browse/STORM-1335
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: 冯健
>  Labels: jstorm-merger
>
> Update jars and configuration dynamically for a topology, without stopping 
> the topology. (How is this different from restart command?) 
> This should be evaluated with how this will impact the blob-store feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1332) Evaluate/port jstorm zktool

[JIRA]

 [ 
https://issues.apache.org/jira/browse/STORM-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

冯健 reassigned STORM-1332:
-

Assignee: 冯健

> Evaluate/port jstorm zktool
> ---
>
> Key: STORM-1332
> URL: https://issues.apache.org/jira/browse/STORM-1332
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: 冯健
>  Labels: jstorm-merger
>
> JStorm:
> Supports some ZooKeeper operations, e.g. "list", "read"…
> Will need to be evaluated for security.  Perhaps we can combine this with the 
> heartbeats command to provide a way to explore stored state in a safe 
> debuggable way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1268) port backtype.storm.daemon.builtin-metrics to java

[Cody (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cody reassigned STORM-1268:
---

Assignee: Cody

> port backtype.storm.daemon.builtin-metrics to java
> --
>
> Key: STORM-1268
> URL: https://issues.apache.org/jira/browse/STORM-1268
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Cody
>  Labels: java-migration, jstorm-merger
>
> Built-in metrics



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1352) Trident should support writing to multiple Kafka clusters

[Haohui Mai (JIRA)]

Haohui Mai created STORM-1352:
-

 Summary: Trident should support writing to multiple Kafka clusters
 Key: STORM-1352
 URL: https://issues.apache.org/jira/browse/STORM-1352
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai


Current it is impossible to instantiate two instances of the 
{{TridentKafkaState}} class that write to different Kafka cluster. This is 
because that {{TridentKafkaState}} obtains the the location of the Kafka 
producer from configuration. Multiple instances can only get the same 
configuration in the {{prepare()}} method.

This jira proposes to introduce a configuration class like 
{{TridentKafkaConfig}} to allow multiple instances of {{TridentKafkaState}} to 
write to different Kafka clusters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1252) port backtype.storm.stats to java

[Cody (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cody reassigned STORM-1252:
---

Assignee: Cody

> port backtype.storm.stats to java
> -
>
> Key: STORM-1252
> URL: https://issues.apache.org/jira/browse/STORM-1252
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Cody
>  Labels: java-migration, jstorm-merger
>
> clojure methods for getting/setting built-in statistics.  Mostly wrappers 
> around some java classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1280) port backtype.storm.daemon.logviewer to java

[Jark Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu reassigned STORM-1280:
--

Assignee: Jark Wu

> port backtype.storm.daemon.logviewer to java
> 
>
> Key: STORM-1280
> URL: https://issues.apache.org/jira/browse/STORM-1280
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Jark Wu
>  Labels: java-migration, jstorm-merger
>
> This is providing a UI for accessing and searching logs.  hiccup will need to 
> be replaced, possibly with just hard coded HTML + escaping.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1075 add external module storm-cassandra

[satishd]

Github user satishd commented on the pull request:

https://github.com/apache/storm/pull/827#issuecomment-159475189
  
@fhussonnois You may want to squash/rebase these 14 commits into 1 or 2 
commits.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1075) Storm Cassandra connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026091#comment-15026091
 ] 

ASF GitHub Bot commented on STORM-1075:
---

Github user satishd commented on the pull request:

https://github.com/apache/storm/pull/827#issuecomment-159475189
  
@fhussonnois You may want to squash/rebase these 14 commits into 1 or 2 
commits.


> Storm Cassandra connector
> -
>
> Key: STORM-1075
> URL: https://issues.apache.org/jira/browse/STORM-1075
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Sriharsha Chintalapani
>Assignee: Satish Duggana
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1327) Evaluate/Port JStorm tuple batching

[Basti Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Basti Liu reassigned STORM-1327:


Assignee: Basti Liu

> Evaluate/Port JStorm tuple batching
> ---
>
> Key: STORM-1327
> URL: https://issues.apache.org/jira/browse/STORM-1327
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Basti Liu
>  Labels: jstorm-merger
>
> Storm:
> Batching in DisruptorQueue
> JStorm:
> Do batch before sending tuple to transfer queue and support for adjusting the 
> batch size dynamically according to samples of actual batch size sent out for 
> past intervals.
> Should evaluate both implementations, and see which is better for 
> performance, and possible if we can/should move some of the dynamic batching 
> logic into disruptor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1323) Explore support for JStorm TopologyMaster

[Basti Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Basti Liu reassigned STORM-1323:


Assignee: Basti Liu

> Explore support for JStorm TopologyMaster
> -
>
> Key: STORM-1323
> URL: https://issues.apache.org/jira/browse/STORM-1323
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Basti Liu
>  Labels: jstorm-merger
>
> JStorm uses a TopologyMaster to reduce the load on zookeeper and to provide 
> coordination within the topology.
> Need to evaluate how this impacts storm architecture especially around 
> security, and port it over, if it makes since.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1325) Port jstorm execution thread monitoring to storm

[Basti Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Basti Liu reassigned STORM-1325:


Assignee: Basti Liu

> Port jstorm execution thread monitoring to storm
> 
>
> Key: STORM-1325
> URL: https://issues.apache.org/jira/browse/STORM-1325
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Basti Liu
>  Labels: jstorm-merger
>
> Monitors the status of the execute thread of tasks. It is effective to find 
> the slow bolt in a topology, and potentially uncovers deadlock as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1276) port backtype.storm.daemon.nimbus to java

[Basti Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026178#comment-15026178
 ] 

Basti Liu commented on STORM-1276:
--

The processing structure of JStorm's Nimbus for reference is different from 
Storm's. I am tring to give the demonstration and detailed diff of 
implementation. The doc will be available in one or two days. We can have an 
evaluation first. Anyway, though different structure, it is able to be 
compatible with current "service-handler" and all pluggable schedulers.

> port backtype.storm.daemon.nimbus to java
> -
>
> Key: STORM-1276
> URL: https://issues.apache.org/jira/browse/STORM-1276
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Basti Liu
>  Labels: java-migration, jstorm-merger
>
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/nimbus
> as a possible example



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1341) Better to have multilang subprocess timeout, not relying on supervisor worker timeout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026184#comment-15026184
 ] 

ASF GitHub Bot commented on STORM-1341:
---

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/903

STORM-1341 Let topology have own heartbeat timeout for multilang subprocess

* config name: topology.subprocess.timeout.secs
* if it's not specified, supervisor.worker.timeout.secs will be used

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1341

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/903.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #903


commit fc8c296efc41c1efc6060ba09c07d406ceacc844
Author: Jungtaek Lim 
Date:   2015-11-25T04:46:59Z

STORM-1341 Let topology have own heartbeat timeout for multilang subprocess

* config name: topology.subprocess.timeout.secs
* if it's not specified, supervisor.worker.timeout.secs will be used




> Better to have multilang subprocess timeout, not relying on supervisor worker 
> timeout
> -
>
> Key: STORM-1341
> URL: https://issues.apache.org/jira/browse/STORM-1341
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-multilang
>Reporter: Jungtaek Lim
>Priority: Minor
>
> Currently, ShellBolt determines that subprocess is hang if there're no 
> message from subprocess within SUPERVISOR_WORKER_TIMEOUT_SECS.
> Since ShellBolt treats all kinds of message to heartbeat (STORM-742), we get 
> rid of heartbeat issue that heartbeat tuple cannot processed in time because 
> too many other BoltMsgs waiting.
> But it doesn't cover two scenario, 
> - STORM-758: Subprocess doesn't send anything - emit, ack, fail - so 
> ShellBolt has to rely on heartbeat tuple. It is valid for non-ACK mode. 
> - Each (or some kind of) tuple takes very long time to process. For now it 
> should be shorter than SUPERVISOR_WORKER_TIMEOUT_SECS.
> It could be better to have multilang subprocess timeout for clarity.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Asynchronous acks and fails

[Abhishek Agarwal]

Thanks Bobby for the clarification. Though I would like to point out that
kafka bolt is not backward compatible with versions prior to 0.10.x.
TupleUtils class, which is being referred to by KafkaBolt, is not available
in 0.9.x releases.

If this break was not intentional, I could refactor the bolt to remove the
TupleUtils class. As a workaround, people can also put TupleUtils class in
the application with appropriate package name. This is what I am doing to
make storm-kafka-0.10.0 work with storm-core 0.9.5


On Tue, Nov 24, 2015 at 9:16 PM, Bobby Evans 
wrote:

> Abhishek,
> OutputCollector is thread safe for most methods, and for all methods in
> 0.11.0+ of storm.  In versions prior to 0.11.0 emitting to a shuffle
> grouping was not thread safe because of the grouping code itself was not
> thread safe, hence the synchronization on the collector to support older
> versions of storm.
> If async is false the callback is null, so no acks/fails will happen there
> and the ack will happen right after calling producer.send so I don't think
> acks will happen twice.
>
> As far as performance in concerned we can guess, but without real numbers
> it is hard to tell what will really happen in any situation.  If you have a
> performance benchmark that we can use to profile different kafka bolt
> implementations in terms of both throughput and latency that would be
> great, but without that until someone complains I am not going to worry
> about it too much.  For me personally my biggest concern is the huge amount
> of object creation that is happening.  But I don't see a reason to invest
> more resources unless someone is complaining or there is a need.  If you
> have that need feel free to file a JIRA, write a benchmark and start
> profiling. - Bobby
>
>
> On Tuesday, November 24, 2015 9:11 AM, Abhishek Agarwal <
> abhishc...@gmail.com> wrote:
>
>
>  I was just looking at STORM-826  >and
> was wondering if this is the correct approach for doing acks and fails
> asynchronously.
>
> > Callback will happen on a different thread. Wouldn't it lead to race
> conditions as the disruptor send queues are single producer queues?
> > If async is set to false, wouldn't the acks/fails performed twice? Since
> callback is always executed by the background thread.
>
> Also if bolt operations are high latency and IO bound, will a non-blocking
> implementation without batching lead to increased throughput ?
>
> --
> Regards,
> Abhishek Agarwal
>
>
>
>



-- 
Regards,
Abhishek Agarwal

[GitHub] storm pull request: STORM-1222. Support Kafka as external tables i...

[darionyaphet]

Github user darionyaphet commented on the pull request:

https://github.com/apache/storm/pull/896#issuecomment-159508385
  
@haohui thanks for you great work :+1:  

Storm SQL need more documents to learning how to use it :)

Will you do some document to explain it ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1222) Support Kafka as external tables in StormSQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026292#comment-15026292
 ] 

ASF GitHub Bot commented on STORM-1222:
---

Github user darionyaphet commented on the pull request:

https://github.com/apache/storm/pull/896#issuecomment-159508385
  
@haohui thanks for you great work :+1:  

Storm SQL need more documents to learning how to use it :)

Will you do some document to explain it ?


> Support Kafka as external tables in StormSQL
> 
>
> Key: STORM-1222
> URL: https://issues.apache.org/jira/browse/STORM-1222
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> This jira propose to support Kafka as both the data ingress and egress point 
> in StormSQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1126) Allow a configMethod that takes no arguments (Flux)

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1126.

   Resolution: Fixed
Fix Version/s: 0.11.0

Thanks [~ashnazg], I merged this into master.

> Allow a configMethod that takes no arguments (Flux)
> ---
>
> Key: STORM-1126
> URL: https://issues.apache.org/jira/browse/STORM-1126
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: Flux
>Affects Versions: 0.10.0
>Reporter: Chuck Burgess
>Assignee: Chuck Burgess
>Priority: Minor
> Fix For: 0.11.0
>
>
> Allow the configMethod implementation to handle calling a method that 
> requires no arguments, by omitting the "args" line.
> {noformat}
> configMethods
> - name: "myMethod"
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1126) Allow a configMethod that takes no arguments (Flux)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025227#comment-15025227
 ] 

ASF GitHub Bot commented on STORM-1126:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/833


> Allow a configMethod that takes no arguments (Flux)
> ---
>
> Key: STORM-1126
> URL: https://issues.apache.org/jira/browse/STORM-1126
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: Flux
>Affects Versions: 0.10.0
>Reporter: Chuck Burgess
>Assignee: Chuck Burgess
>Priority: Minor
>
> Allow the configMethod implementation to handle calling a method that 
> requires no arguments, by omitting the "args" line.
> {noformat}
> configMethods
> - name: "myMethod"
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-876] Blobstore API

[d2r]

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45793192
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
+ * indicating the true value of number of updates for a blob. This node 
helps to maintain sanity in case
+ * leadership changes due to crashing.
+ *
+ * 2. Delete does not require to hand over the sequence number.
+ *
+ * 3. Finally, the update has few scenarios.
+ *
+ *  The class implements a TreeSet. The basic idea is if all the nimbodes 
have the same
+ *  sequence number for the blob, then the number of elements in the set 
is 1 which holds
+ *  the latest value of sequence number. If the number of elements are 
greater than 1 then it
+ *  implies that there is sequence mismatch and there is need for syncing 
the blobs across
+ *  nimbodes.
+ *
+ *  The logic for handing over sequence numbers based on the state are 
described as follows
+ *  Here consider Nimbus-1 alias as N1 and Nimbus-2 alias as N2.
+ *  Scenario 1:
+ *  Example: Normal create/update scenario
+ *  Operation Nimbus-1:state Nimbus-2:state Seq-Num-Nimbus-1  
Seq-Num-Nimbus-2  Max-Seq-Num
+ *  Create-Key1   alive - Leader alive  1  
 1
+ *  Sync  alive - Leader alive  1 
1 (callback -> download)  1
+ *  Update-Key1   alive - Leader alive  

[GitHub] storm pull request: [STORM-876] Blobstore API

[d2r]

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/845#issuecomment-159406843
  
backtype.storm.utils.Utils
```
 535 try {
 536 ReadableBlobMeta metadata = cb.getBlobMeta(key);
 537 nimbusBlobVersion = metadata.get_version();
 538 } catch (AuthorizationException | KeyNotFoundException exp) {
 539 throw exp;
 540 }
```

Don't need `try ... catch` if we just throw whatever we catch.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025503#comment-15025503
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45800279
  
--- Diff: storm-core/src/jvm/backtype/storm/blobstore/LocalFsBlobStore.java 
---
@@ -0,0 +1,297 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package backtype.storm.blobstore;
+
+import backtype.storm.Config;
+import backtype.storm.generated.SettableBlobMeta;
+import backtype.storm.generated.AuthorizationException;
+import backtype.storm.generated.KeyAlreadyExistsException;
+import backtype.storm.generated.KeyNotFoundException;
+import backtype.storm.generated.ReadableBlobMeta;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.thrift.TBase;
--- End diff --

Still unused.


> Dist Cache: Basic Functionality
> ---
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and 
> downloading dist cache items.  storm-core.ser, storm-conf.ser and storm.jar 
> should be written into the blob store instead of residing locally. We need a 
> default implementation of the blob store that does essentially what nimbus 
> currently does and does not need anything extra.  But having an HDFS backend 
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and 
> provide a working directory for the worker process with symlinks to the 
> blobs.  It should also allow the blobs to be updated and switch the symlink 
> atomically to point to the new blob once it is downloaded.
> All of this is already done by code internal to Yahoo! we are in the process 
> of getting it ready to push back to open source shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1349) [Flux] Allow constructorArgs to take Maps as arguments

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1349:
---
Fix Version/s: 0.11.0

> [Flux] Allow constructorArgs to take Maps as arguments
> --
>
> Key: STORM-1349
> URL: https://issues.apache.org/jira/browse/STORM-1349
> Project: Apache Storm
>  Issue Type: Bug
>  Components: Flux
>Reporter: P. Taylor Goetz
> Fix For: 0.11.0
>
>
> Creating JIRA for https://github.com/apache/storm/pull/732



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-874) Netty Threads do not handle Errors properly

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz reassigned STORM-874:
-

Assignee: Kishor Patil

> Netty Threads do not handle Errors properly
> ---
>
> Key: STORM-874
> URL: https://issues.apache.org/jira/browse/STORM-874
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.9.2-incubating, 0.10.0
>Reporter: Kishor Patil
>Assignee: Kishor Patil
>
> When low on memory, netty thread could get OOM which if not handled correctly 
> can lead to unexpected behavior such as netty connection leaks.
> {code:java}
> java.lang.OutOfMemoryError: Direct buffer memory
>   at java.nio.Bits.reserveMemory(Bits.java:658) ~[?:1.8.0_25]
>   at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
> ~[?:1.8.0_25]
>   at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_25]
>   at 
> org.jboss.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffers.java:167) 
> ~[netty-3.9.4.Final.jar:?]
>   at 
> org.jboss.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffers.java:151) 
> ~[netty-3.9.4.Final.jar:?]
>   at 
> backtype.storm.messaging.netty.MessageBatch.buffer(MessageBatch.java:101) 
> ~[storm-core-0.9.2-incubating-security.jar:0.9.2-incubating-security]
>   at 
> backtype.storm.messaging.netty.MessageEncoder.encode(MessageEncoder.java:32) 
> ~[storm-core-0.9.2-incubating-security.jar:0.9.2-incubating-security]
>   at 
> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:66)
>  ~[netty-3.9.4.Final.jar:?]
>   at 
> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>  ~[netty-3.9.4.Final.jar:?]
>   at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
>  ~[netty-3.9.4.Final.jar:?]
>   at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
>  ~[netty-3.9.4.Final.jar:?]
>   at org.jboss.netty.channel.Channels.write(Channels.java:704) 
> ~[netty-3.9.4.Final.jar:?]
>   at org.jboss.netty.channel.Channels.write(Channels.java:671) 
> ~[netty-3.9.4.Final.jar:?]
>   at 
> org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248) 
> ~[netty-3.9.4.Final.jar:?]
>   at 
> backtype.storm.messaging.netty.Client.tryDeliverMessages(Client.java:226) 
> ~[storm-core-0.9.2-incubating-security.jar:0.9.2-incubating-security]
>   at backtype.storm.messaging.netty.Client.send(Client.java:173) 
> ~[storm-core-0.9.2-incubating-security.jar:0.9.2-incubating-security]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025395#comment-15025395
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45793991
  
--- Diff: storm-core/src/jvm/backtype/storm/blobstore/LocalFsBlobStore.java 
---
@@ -0,0 +1,297 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package backtype.storm.blobstore;
+
+import backtype.storm.Config;
+import backtype.storm.generated.SettableBlobMeta;
+import backtype.storm.generated.AuthorizationException;
+import backtype.storm.generated.KeyAlreadyExistsException;
+import backtype.storm.generated.KeyNotFoundException;
+import backtype.storm.generated.ReadableBlobMeta;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.thrift.TBase;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.FileNotFoundException;
+import java.io.InputStream;
+
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;;
+
+import static backtype.storm.blobstore.BlobStoreAclHandler.ADMIN;
+import static backtype.storm.blobstore.BlobStoreAclHandler.READ;
+import static backtype.storm.blobstore.BlobStoreAclHandler.WRITE;
+
+/**
+ * Provides a local file system backed blob store implementation for 
Nimbus.
+ */
+public class LocalFsBlobStore extends BlobStore {
+  public static final Logger LOG = 
LoggerFactory.getLogger(LocalFsBlobStore.class);
+  private static final String DATA_PREFIX = "data_";
+  private static final String META_PREFIX = "meta_";
+  protected BlobStoreAclHandler _aclHandler;
+  private final String BLOBSTORE_SUBTREE = "/blobstore/";
+  private NimbusInfo nimbusInfo;
+  private FileBlobStoreImpl fbs;
+  private Map conf;
+
+  @Override
+  public void prepare(Map conf, String overrideBase, NimbusInfo 
nimbusInfo) {
+this.conf = conf;
+this.nimbusInfo = nimbusInfo;
+ if (overrideBase == null) {
+  overrideBase = (String)conf.get(Config.BLOBSTORE_DIR);
+  if (overrideBase == null) {
+overrideBase = (String) conf.get(Config.STORM_LOCAL_DIR);
+  }
+}
+File baseDir = new File(overrideBase, BASE_BLOBS_DIR_NAME);
+try {
+  fbs = new FileBlobStoreImpl(baseDir, conf);
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+_aclHandler = new BlobStoreAclHandler(conf);
+  }
--- End diff --

Must have missed this last time.  Check the indentation for `prepare`, 
looks like a mix of 1-space and 2-space.


> Dist Cache: Basic Functionality
> ---
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and 
> downloading dist cache items.  storm-core.ser, storm-conf.ser and storm.jar 
> should be written into the blob store instead of residing locally. We need a 
> default implementation of the blob store that does essentially what nimbus 
> currently does and does not need anything extra.  But having an HDFS backend 
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and 
> provide a working 

[jira] [Commented] (STORM-1203) worker metadata file creation doesn't use storm.log.dir config

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025221#comment-15025221
 ] 

ASF GitHub Bot commented on STORM-1203:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/878


> worker metadata file creation doesn't use storm.log.dir config
> --
>
> Key: STORM-1203
> URL: https://issues.apache.org/jira/browse/STORM-1203
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-876] Blobstore API

[d2r]

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45789303
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
+ * indicating the true value of number of updates for a blob. This node 
helps to maintain sanity in case
+ * leadership changes due to crashing.
+ *
+ * 2. Delete does not require to hand over the sequence number.
+ *
+ * 3. Finally, the update has few scenarios.
+ *
+ *  The class implements a TreeSet. The basic idea is if all the nimbodes 
have the same
+ *  sequence number for the blob, then the number of elements in the set 
is 1 which holds
+ *  the latest value of sequence number. If the number of elements are 
greater than 1 then it
+ *  implies that there is sequence mismatch and there is need for syncing 
the blobs across
+ *  nimbodes.
+ *
+ *  The logic for handing over sequence numbers based on the state are 
described as follows
+ *  Here consider Nimbus-1 alias as N1 and Nimbus-2 alias as N2.
+ *  Scenario 1:
+ *  Example: Normal create/update scenario
+ *  Operation Nimbus-1:state Nimbus-2:state Seq-Num-Nimbus-1  
Seq-Num-Nimbus-2  Max-Seq-Num
+ *  Create-Key1   alive - Leader alive  1  
 1
+ *  Sync  alive - Leader alive  1 
1 (callback -> download)  1
+ *  Update-Key1   alive - Leader alive  

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025277#comment-15025277
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45789148
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
+ * indicating the true value of number of updates for a blob. This node 
helps to maintain sanity in case
+ * leadership changes due to crashing.
+ *
+ * 2. Delete does not require to hand over the sequence number.
+ *
+ * 3. Finally, the update has few scenarios.
+ *
+ *  The class implements a TreeSet. The basic idea is if all the nimbodes 
have the same
+ *  sequence number for the blob, then the number of elements in the set 
is 1 which holds
+ *  the latest value of sequence number. If the number of elements are 
greater than 1 then it
+ *  implies that there is sequence mismatch and there is need for syncing 
the blobs across
+ *  nimbodes.
+ *
+ *  The logic for handing over sequence numbers based on the state are 
described as follows
+ *  Here consider Nimbus-1 alias as N1 and Nimbus-2 alias as N2.
+ *  Scenario 1:
+ *  Example: Normal create/update scenario
+ *  Operation Nimbus-1:state Nimbus-2:state Seq-Num-Nimbus-1  
Seq-Num-Nimbus-2  Max-Seq-Num
+ *  Create-Key1   alive 

[GitHub] storm pull request: [STORM-876] Blobstore API

[d2r]

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45789148
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
+ * indicating the true value of number of updates for a blob. This node 
helps to maintain sanity in case
+ * leadership changes due to crashing.
+ *
+ * 2. Delete does not require to hand over the sequence number.
+ *
+ * 3. Finally, the update has few scenarios.
+ *
+ *  The class implements a TreeSet. The basic idea is if all the nimbodes 
have the same
+ *  sequence number for the blob, then the number of elements in the set 
is 1 which holds
+ *  the latest value of sequence number. If the number of elements are 
greater than 1 then it
+ *  implies that there is sequence mismatch and there is need for syncing 
the blobs across
+ *  nimbodes.
+ *
+ *  The logic for handing over sequence numbers based on the state are 
described as follows
+ *  Here consider Nimbus-1 alias as N1 and Nimbus-2 alias as N2.
+ *  Scenario 1:
+ *  Example: Normal create/update scenario
+ *  Operation Nimbus-1:state Nimbus-2:state Seq-Num-Nimbus-1  
Seq-Num-Nimbus-2  Max-Seq-Num
+ *  Create-Key1   alive - Leader alive  1  
 1
+ *  Sync  alive - Leader alive  1 
1 (callback -> download)  1
+ *  Update-Key1   alive - Leader alive  

[jira] [Created] (STORM-1350) Unable to build: Too many files with unapproved licenses (RAT exception)

[Rudra Sharma (JIRA)]

Rudra Sharma created STORM-1350:
---

 Summary: Unable to build: Too many files with unapproved licenses 
(RAT exception)
 Key: STORM-1350
 URL: https://issues.apache.org/jira/browse/STORM-1350
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Affects Versions: 0.11.0
 Environment: Mac OSX
Reporter: Rudra Sharma
Priority: Minor


Issue when making build from the master branch. 

[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check 
(default) on project storm: Too many files with unapproved license: 7 See RAT 
report in: /Users/myusername/Documents/apache-storm/target/rat.txt -> [Help 1]

Simple workaround is to build with maven flag to skip rat check:
mvn clean install -Drat.skip=true



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025286#comment-15025286
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45789625
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
--- End diff --

Really nice documentation here!

`later` -> `latter`


> Dist Cache: Basic Functionality
> ---
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and 
> downloading dist cache items.  storm-core.ser, storm-conf.ser and storm.jar 
> should be written into the blob store instead of residing locally. We need a 
> default implementation of the blob store that does essentially what nimbus 
> currently does and does not need anything extra.  But having an HDFS backend 
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and 
> 

[GitHub] storm pull request: [STORM-876] Blobstore API

[redsanket]

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45797735
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
+ * indicating the true value of number of updates for a blob. This node 
helps to maintain sanity in case
+ * leadership changes due to crashing.
+ *
+ * 2. Delete does not require to hand over the sequence number.
+ *
+ * 3. Finally, the update has few scenarios.
+ *
+ *  The class implements a TreeSet. The basic idea is if all the nimbodes 
have the same
+ *  sequence number for the blob, then the number of elements in the set 
is 1 which holds
+ *  the latest value of sequence number. If the number of elements are 
greater than 1 then it
+ *  implies that there is sequence mismatch and there is need for syncing 
the blobs across
+ *  nimbodes.
+ *
+ *  The logic for handing over sequence numbers based on the state are 
described as follows
+ *  Here consider Nimbus-1 alias as N1 and Nimbus-2 alias as N2.
+ *  Scenario 1:
+ *  Example: Normal create/update scenario
+ *  Operation Nimbus-1:state Nimbus-2:state Seq-Num-Nimbus-1  
Seq-Num-Nimbus-2  Max-Seq-Num
+ *  Create-Key1   alive - Leader alive  1  
 1
+ *  Sync  alive - Leader alive  1 
1 (callback -> download)  1
+ *  Update-Key1   alive - Leader 

[GitHub] storm pull request: STORM-1126: allow for configMethods with no ar...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/833


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-876] Blobstore API

[d2r]

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45789625
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
--- End diff --

Really nice documentation here!

`later` -> `latter`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025388#comment-15025388
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45793730
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
+ * indicating the true value of number of updates for a blob. This node 
helps to maintain sanity in case
+ * leadership changes due to crashing.
+ *
+ * 2. Delete does not require to hand over the sequence number.
+ *
+ * 3. Finally, the update has few scenarios.
+ *
+ *  The class implements a TreeSet. The basic idea is if all the nimbodes 
have the same
+ *  sequence number for the blob, then the number of elements in the set 
is 1 which holds
+ *  the latest value of sequence number. If the number of elements are 
greater than 1 then it
+ *  implies that there is sequence mismatch and there is need for syncing 
the blobs across
+ *  nimbodes.
+ *
+ *  The logic for handing over sequence numbers based on the state are 
described as follows
+ *  Here consider Nimbus-1 alias as N1 and Nimbus-2 alias as N2.
+ *  Scenario 1:
+ *  Example: Normal create/update scenario
+ *  Operation Nimbus-1:state Nimbus-2:state Seq-Num-Nimbus-1  
Seq-Num-Nimbus-2  Max-Seq-Num
+ *  Create-Key1   alive 

[GitHub] storm pull request: [STORM-876] Blobstore API

[d2r]

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45793730
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
+ * indicating the true value of number of updates for a blob. This node 
helps to maintain sanity in case
+ * leadership changes due to crashing.
+ *
+ * 2. Delete does not require to hand over the sequence number.
+ *
+ * 3. Finally, the update has few scenarios.
+ *
+ *  The class implements a TreeSet. The basic idea is if all the nimbodes 
have the same
+ *  sequence number for the blob, then the number of elements in the set 
is 1 which holds
+ *  the latest value of sequence number. If the number of elements are 
greater than 1 then it
+ *  implies that there is sequence mismatch and there is need for syncing 
the blobs across
+ *  nimbodes.
+ *
+ *  The logic for handing over sequence numbers based on the state are 
described as follows
+ *  Here consider Nimbus-1 alias as N1 and Nimbus-2 alias as N2.
+ *  Scenario 1:
+ *  Example: Normal create/update scenario
+ *  Operation Nimbus-1:state Nimbus-2:state Seq-Num-Nimbus-1  
Seq-Num-Nimbus-2  Max-Seq-Num
+ *  Create-Key1   alive - Leader alive  1  
 1
+ *  Sync  alive - Leader alive  1 
1 (callback -> download)  1
+ *  Update-Key1   alive - Leader alive  

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025460#comment-15025460
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45797735
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
+ * indicating the true value of number of updates for a blob. This node 
helps to maintain sanity in case
+ * leadership changes due to crashing.
+ *
+ * 2. Delete does not require to hand over the sequence number.
+ *
+ * 3. Finally, the update has few scenarios.
+ *
+ *  The class implements a TreeSet. The basic idea is if all the nimbodes 
have the same
+ *  sequence number for the blob, then the number of elements in the set 
is 1 which holds
+ *  the latest value of sequence number. If the number of elements are 
greater than 1 then it
+ *  implies that there is sequence mismatch and there is need for syncing 
the blobs across
+ *  nimbodes.
+ *
+ *  The logic for handing over sequence numbers based on the state are 
described as follows
+ *  Here consider Nimbus-1 alias as N1 and Nimbus-2 alias as N2.
+ *  Scenario 1:
+ *  Example: Normal create/update scenario
+ *  Operation Nimbus-1:state Nimbus-2:state Seq-Num-Nimbus-1  
Seq-Num-Nimbus-2  Max-Seq-Num
+ *  Create-Key1   

[GitHub] storm pull request: [STORM-876] Blobstore API

[d2r]

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45800279
  
--- Diff: storm-core/src/jvm/backtype/storm/blobstore/LocalFsBlobStore.java 
---
@@ -0,0 +1,297 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package backtype.storm.blobstore;
+
+import backtype.storm.Config;
+import backtype.storm.generated.SettableBlobMeta;
+import backtype.storm.generated.AuthorizationException;
+import backtype.storm.generated.KeyAlreadyExistsException;
+import backtype.storm.generated.KeyNotFoundException;
+import backtype.storm.generated.ReadableBlobMeta;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.thrift.TBase;
--- End diff --

Still unused.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-1349) [Flux] Allow constructorArgs to take Maps as arguments

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1349.

Resolution: Fixed

> [Flux] Allow constructorArgs to take Maps as arguments
> --
>
> Key: STORM-1349
> URL: https://issues.apache.org/jira/browse/STORM-1349
> Project: Apache Storm
>  Issue Type: Bug
>  Components: Flux
>Reporter: P. Taylor Goetz
> Fix For: 0.11.0
>
>
> Creating JIRA for https://github.com/apache/storm/pull/732



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1203. worker metadata file creation does...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/878


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025280#comment-15025280
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45789303
  
--- Diff: 
storm-core/src/jvm/backtype/storm/blobstore/KeySequenceNumber.java ---
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.TreeSet;
+import java.util.Map;
+import java.util.List;
+
+/**
+ * Class hands over the key sequence number which implies the number of 
updates made to a blob.
+ * The information regarding the keys and the sequence number which 
represents the number of updates are
+ * stored within the zookeeper in the following format.
+ * /storm/blobstore/key_name/nimbushostport-sequencenumber
+ * Example:
+ * If there are two nimbodes with nimbus.seeds:leader,non-leader are set,
+ * then the state inside the zookeeper is eventually stored as:
+ * /storm/blobstore/key1/leader:8080-1
+ * /storm/blobstore/key1/non-leader:8080-1
+ * indicates that a new blob with the name key1 has been created on the 
leader
+ * nimbus and the non-leader nimbus syncs after a call back is triggered 
by attempting
+ * to download the blob and finally updates its state inside the zookeeper.
+ *
+ * A watch is placed on the /storm/blobstore/key1 and the znodes 
leader:8080-1 and
+ * non-leader:8080-1 are ephemeral which implies that these nodes exist 
only until the
+ * connection between the corresponding nimbus and the zookeeper persist. 
If in case the
+ * nimbus crashes the node disappears under /storm/blobstore/key1.
+ *
+ * The sequence number for the keys are handed over based on the following 
scenario:
+ * Lets assume there are three nimbodes up and running, one being the 
leader and the other
+ * being the non-leader.
+ *
+ * 1. Create is straight forward.
+ * Check whether the znode -> /storm/blobstore/key1 has been created or 
not. It implies
+ * the blob has not been created yet. If not created, it creates it and 
updates the zookeeper
+ * states under /storm/blobstore/key1 and 
/storm/blobstoremaxkeysequencenumber/key1.
+ * The znodes it creates on these nodes are 
/storm/blobstore/key1/leader:8080-1,
+ * /storm/blobstore/key1/non-leader:8080-1 and 
/storm/blobstoremaxkeysequencenumber/key1/1.
+ * The later holds the global sequence number across all nimbodes more 
like a static variable
+ * indicating the true value of number of updates for a blob. This node 
helps to maintain sanity in case
+ * leadership changes due to crashing.
+ *
+ * 2. Delete does not require to hand over the sequence number.
+ *
+ * 3. Finally, the update has few scenarios.
+ *
+ *  The class implements a TreeSet. The basic idea is if all the nimbodes 
have the same
+ *  sequence number for the blob, then the number of elements in the set 
is 1 which holds
+ *  the latest value of sequence number. If the number of elements are 
greater than 1 then it
+ *  implies that there is sequence mismatch and there is need for syncing 
the blobs across
+ *  nimbodes.
+ *
+ *  The logic for handing over sequence numbers based on the state are 
described as follows
+ *  Here consider Nimbus-1 alias as N1 and Nimbus-2 alias as N2.
+ *  Scenario 1:
+ *  Example: Normal create/update scenario
+ *  Operation Nimbus-1:state Nimbus-2:state Seq-Num-Nimbus-1  
Seq-Num-Nimbus-2  Max-Seq-Num
+ *  Create-Key1   alive 

[GitHub] storm pull request: STORM-1340: Use Travis-CI build matrix to impr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/902


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-1203) worker metadata file creation doesn't use storm.log.dir config

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1203.

Resolution: Fixed

> worker metadata file creation doesn't use storm.log.dir config
> --
>
> Key: STORM-1203
> URL: https://issues.apache.org/jira/browse/STORM-1203
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Storm 2.0 plan

[Derek Dagit]

I am OK with this plan.
 
-- 
Derek



From: Bobby Evans 
To: Dev  
Sent: Monday, November 23, 2015 3:40 PM
Subject: [VOTE] Storm 2.0 plan


Sorry for spaming everyone with all the JIRA creations today.  I have filed all 
of the JIRA corresponding to the plan for JStorm merger listed here.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109
The bylaws 
https://github.com/apache/storm/blob/master/docs/documentation/BYLAWS.md don't 
cover a vote on the direction of the project like this.  They cover the merger 
of each pull request that would be made, but not a direction change. As such 
this vote is more symbolic than anything, and I would love to hear from 
everyone involved.

The current plan is to finish merging in features for a 1.0 release.  

https://cwiki.apache.org/confluence/display/STORM/Storm+Release+1.0
is supposed to cover this, but I think it is missing some features others want, 
so please let us know if you really want to get your feature in before this 
happens.  As such the time frame is a bit flexible but I would like to shoot 
for doing a storm-1.0 release before mid December.
After that we would begin merging in the clojure->java transition JIRA.  Once 
those are complete the feature freeze would be lifted and JStorm features would 
be merged in along with other features.  Hopefully we would have a Storm 2.0 
release by mid February to mid March, depending on how things go. 

I am +1 on this plan (if you couldn't tell)
 - Bobby

[GitHub] storm pull request: Update ObjectDef.java for flux

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/732


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-876] Blobstore API

[d2r]

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/845#issuecomment-159413532
  
backtype.storm.utils.Utils

After an upmerge, it looks like we cleaned up some duplicate method 
declarations.  However, much of the remaining code is duplicated elsewhere.


```
 664 public static  T thriftDeserialize(Class c, byte[] b) {
 665 try {
 666 T ret = (T) c.newInstance();
 667 TDeserializer des = getDes();
 668 des.deserialize((TBase) ret, b);
 669 return ret;
 670 } catch (Exception e) {
 671 throw new RuntimeException(e);
 672 }
 673
 674 }
```
This should just call `Utils.thriftDeserialize(c, b, 0, b.length)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025481#comment-15025481
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/845#issuecomment-159413532
  
backtype.storm.utils.Utils

After an upmerge, it looks like we cleaned up some duplicate method 
declarations.  However, much of the remaining code is duplicated elsewhere.


```
 664 public static  T thriftDeserialize(Class c, byte[] b) {
 665 try {
 666 T ret = (T) c.newInstance();
 667 TDeserializer des = getDes();
 668 des.deserialize((TBase) ret, b);
 669 return ret;
 670 } catch (Exception e) {
 671 throw new RuntimeException(e);
 672 }
 673
 674 }
```
This should just call `Utils.thriftDeserialize(c, b, 0, b.length)`



> Dist Cache: Basic Functionality
> ---
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and 
> downloading dist cache items.  storm-core.ser, storm-conf.ser and storm.jar 
> should be written into the blob store instead of residing locally. We need a 
> default implementation of the blob store that does essentially what nimbus 
> currently does and does not need anything extra.  But having an HDFS backend 
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and 
> provide a working directory for the worker process with symlinks to the 
> blobs.  It should also allow the blobs to be updated and switch the symlink 
> atomically to point to the new blob once it is downloaded.
> All of this is already done by code internal to Yahoo! we are in the process 
> of getting it ready to push back to open source shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Multiple query processing in apache storm

[Renya nath N]

Sir,

 How can I process multile queries in aache storm?
Do I need to create separate topology for each query?







Thank you
Renya Nath N

[jira] [Created] (STORM-1353) Update "jstorm-import" branch to the latest JStorm release 2.1.0

[Basti Liu (JIRA)]

Basti Liu created STORM-1353:


 Summary: Update "jstorm-import" branch to the latest JStorm 
release 2.1.0
 Key: STORM-1353
 URL: https://issues.apache.org/jira/browse/STORM-1353
 Project: Apache Storm
  Issue Type: Request
  Components: storm-core
Reporter: Basti Liu
Assignee: Basti Liu


The jstorm version of "jstorm-import" reference branch is old. Many JStorm 
features listed in JStorm merging plan are not included. This issue was created 
to update JStorm to the latest release 2.1.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1075 add external module storm-cassandra

[satishd]

Github user satishd commented on the pull request:

https://github.com/apache/storm/pull/827#issuecomment-159350084
  
@fhussonnois I already have some of those mappers code and trident related 
code with me. I can work on mappers and integrate with field-tuple/column 
fluent api. Can you refactor your API to remove insert/update builder and push 
those changes? I can take over from there to complete the remaining changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1075 add external module storm-cassandra

[ptgoetz]

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/827#issuecomment-159353605
  
@satishd I think that work can be done in follow-up JIRAs after this PR is 
merged. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1075 add external module storm-cassandra

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/827#issuecomment-159349948
  
@fhussonnois looks good to me as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1075) Storm Cassandra connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024921#comment-15024921
 ] 

ASF GitHub Bot commented on STORM-1075:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/827#issuecomment-159349948
  
@fhussonnois looks good to me as well.


> Storm Cassandra connector
> -
>
> Key: STORM-1075
> URL: https://issues.apache.org/jira/browse/STORM-1075
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Sriharsha Chintalapani
>Assignee: Satish Duggana
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-756 Handle taskids response as soon as p...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/897#issuecomment-159354731
  
Modified ThroughputVsLatency test result
(replace SplitSentence to multilang implementation) 

Parameter: ratePerSecond: 3500, parallelism: 3 (for 3 machines), test 
duration: 5 mins

* master branch

```
uptime:   30 acked:10,060 acked/sec: 335.33 failed:0 99%:   
0 99.9%:   0 min:   0 max:   0 
mean:0.00 stddev:0.00 user:  0 sys:  0 
gc:  0 mem:   0.00
uptime:   61 acked:55,440 acked/sec:   1,788.39 failed:0 99%:  
22,045,261,823 99.9%:  23,286,775,807 min:   1,729,101,824 max:  24,628,953,087 
mean: 11,432,404,508.43 stddev: 4,368,826,946.24 user:198,130 sys: 
29,340 gc:  3,656 mem: 124.56
uptime:   91 acked:   137,980 acked/sec:   4,599.33 failed:0 99%:  
20,703,084,543 99.9%:  22,850,568,191 min:   9,191,424 max:  24,494,735,359 
mean: 7,061,035,402.37 stddev: 6,645,146,118.94 user: 90,500 sys: 
28,790 gc:  3,088 mem: 125.47
uptime:  122 acked:   118,320 acked/sec:   3,816.77 failed:0 99%:   
7,864,319,999 99.9%:   8,640,266,239 min:   8,765,440 max:   8,782,872,575 
mean: 1,387,484,861.22 stddev: 1,946,172,626.31 user: 84,680 sys: 
30,710 gc:  2,080 mem: 125.94
uptime:  152 acked:   104,280 acked/sec:   3,476.00 failed:0 99%:   
9,269,411,839 99.9%:   9,428,795,391 min:   8,765,440 max:   9,487,515,647 
mean: 1,755,618,080.92 stddev: 2,708,794,434.21 user: 83,980 sys: 
32,180 gc:  2,141 mem: 126.13
uptime:  182 acked:   109,440 acked/sec:   3,648.00 failed:0 99%:  
11,391,729,663 99.9%:  12,373,196,799 min:   8,912,896 max:  12,507,414,527 
mean: 2,468,952,333.95 stddev: 3,853,055,496.32 user: 84,560 sys: 
32,350 gc:  2,130 mem: 126.24
uptime:  212 acked:99,780 acked/sec:   3,326.00 failed:0 99%:  
13,967,032,319 99.9%:  14,269,022,207 min:   9,207,808 max:  14,562,623,487 
mean: 4,100,379,004.64 stddev: 4,838,616,992.68 user: 87,170 sys: 
34,340 gc:  2,594 mem: 126.44
uptime:  243 acked:94,280 acked/sec:   3,041.29 failed:0 99%:  
16,886,267,903 99.9%:  17,330,864,127 min:   1,125,122,048 max:  17,465,081,855 
mean: 7,729,820,542.63 stddev: 4,380,190,964.09 user: 91,780 sys: 
36,070 gc:  3,770 mem: 126.53
uptime:  273 acked:   111,800 acked/sec:   3,726.67 failed:0 99%:  
20,283,654,143 99.9%:  20,736,638,975 min:   9,535,488 max:  20,954,742,783 
mean: 8,742,006,020.17 stddev: 6,569,779,263.81 user: 90,150 sys: 
32,600 gc:  3,496 mem: 126.58
uptime:  303 acked:   125,580 acked/sec:   4,186.00 failed:0 99%:  
22,699,573,247 99.9%:  23,051,894,783 min:   9,035,776 max:  23,102,226,431 
mean: 6,929,826,240.64 stddev: 8,474,118,522.65 user: 84,990 sys: 
30,100 gc:  2,583 mem: 126.65
```

* STORM-756

```
uptime:   30 acked:10,100 acked/sec: 336.67 failed:0 99%:  
14,461,960,191 99.9%:  14,814,281,727 min: 442,236,928 max:  14,973,665,279 
mean: 7,939,956,844.68 stddev: 3,841,389,551.14 user:109,710 sys: 
11,110 gc:513 mem: 119.48
uptime:   62 acked:   106,440 acked/sec:   3,326.25 failed:0 99%:  
26,256,343,039 99.9%:  28,387,049,471 min:   2,054,160,384 max:  29,678,895,103 
mean: 13,926,608,768.77 stddev: 5,725,311,173.69 user:108,090 sys: 
24,990 gc:  4,869 mem: 125.12
uptime:   92 acked:   131,340 acked/sec:   4,378.00 failed:0 99%:  
21,760,049,151 99.9%:  23,974,641,663 min:   9,068,544 max:  25,165,823,999 
mean: 7,675,575,835.18 stddev: 6,221,018,932.96 user: 89,030 sys: 
31,020 gc:  3,325 mem: 125.41
uptime:  123 acked:   106,320 acked/sec:   3,429.68 failed:0 99%:  
22,095,593,471 99.9%:  22,884,122,623 min:   9,486,336 max:  23,471,325,183 
mean: 4,663,389,393.35 stddev: 7,215,966,720.18 user: 86,270 sys: 
32,410 gc:  2,577 mem: 125.95
uptime:  153 acked:   130,440 acked/sec:   4,348.00 failed:0 99%:  
22,917,677,055 99.9%:  24,041,750,527 min:   9,453,568 max:  24,108,859,391 
mean: 4,941,133,334.61 stddev: 6,703,138,537.34 user: 84,630 sys: 
29,780 gc:  2,220 mem: 126.18
uptime:  184 acked:   105,800 acked/sec:   3,412.90 failed:0 99%:   
2,885,681,151 99.9%:   3,460,300,799 min:   7,655,424 max:   3,690,987,519 
mean:  613,777,739.77 stddev:  805,471,793.38 user: 80,580 sys: 30,160 
gc:  1,771 mem: 126.37
uptime:  214 acked:   112,520 acked/sec:   3,750.67 failed:0 99%:   
5,205,131,263 99.9%:   5,326,766,079 min:   8,962,048 max:   5,440,012,287 
mean: 1,416,728,576.68 stddev: 1,510,976,383.57 

[jira] [Created] (STORM-1346) upgrade topology CLI tool

[Parth Brahmbhatt (JIRA)]

Parth Brahmbhatt created STORM-1346:
---

 Summary: upgrade topology CLI tool 
 Key: STORM-1346
 URL: https://issues.apache.org/jira/browse/STORM-1346
 Project: Apache Storm
  Issue Type: Sub-task
  Components: storm-core
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1345) Thrift, nimbus ,zookeeper, supervisor and worker changes to support update topology.

[Parth Brahmbhatt (JIRA)]

Parth Brahmbhatt created STORM-1345:
---

 Summary: Thrift, nimbus ,zookeeper, supervisor and worker changes 
to support update topology.
 Key: STORM-1345
 URL: https://issues.apache.org/jira/browse/STORM-1345
 Project: Apache Storm
  Issue Type: Sub-task
  Components: storm-core
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1347) ui changes to display the topology version.

[Parth Brahmbhatt (JIRA)]

Parth Brahmbhatt created STORM-1347:
---

 Summary: ui changes to display the topology version. 
 Key: STORM-1347
 URL: https://issues.apache.org/jira/browse/STORM-1347
 Project: Apache Storm
  Issue Type: Sub-task
  Components: storm-core
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1207: Added flux support for IWindowedBo...

[ptgoetz]

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/899#issuecomment-159365011
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1207) Add flux support for handling windowed bolts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025051#comment-15025051
 ] 

ASF GitHub Bot commented on STORM-1207:
---

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/899#issuecomment-159365011
  
+1


> Add flux support for handling windowed bolts
> 
>
> Key: STORM-1207
> URL: https://issues.apache.org/jira/browse/STORM-1207
> Project: Apache Storm
>  Issue Type: Sub-task
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> Add support to flux for defining and configuring windowed bolts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1189: Maintain wire compatability with 0...

[spicoflorin]

Github user spicoflorin commented on the pull request:

https://github.com/apache/storm/pull/868#issuecomment-159317100
  
Hello!
 I have downloaded the version 0.10.0 and applied the patch from  
https://patch-diff.githubusercontent.com/raw/apache/storm/pull/868.patch, but I 
got compilation errors. I would like to use the 0.10.0 storm version. Can you 
please test again? Or what code version should I use : the one from tags 
(0.10.0) or the one from the branch 0.10.x master?
 ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on 
project storm-core: Compilation failure: Compilation failure:
ERROR] 
/d:/Temp/stom-10/storm-0.10.0/storm-core/src/jvm/backtype/storm/generated/ClusterSummary.java:[321,5]
 duplicate case label
ERROR] 
/d:/Temp/stom-10/storm-0.10.0/storm-core/src/jvm/backtype/storm/generated/ClusterSummary.java:[403,13]
 variable this_present_nimbus_uptime_secs is already defined in method 
equals(backtype.storm.generated.ClusterSummary)
ERROR] 
/d:/Temp/stom-10/storm-0.10.0/storm-core/src/jvm/backtype/storm/generated/ClusterSummary.java:[404,13]
 variable that_present_nimbus_uptime_secs is already defined in method 
equals(backtype.storm.generated.ClusterSummary)
ERROR] 
/d:/Temp/stom-10/storm-0.10.0/storm-core/src/jvm/backtype/storm/generated/ClusterSummary.java:[617,11]
 duplicate case label
ERROR] -> [Help 1]


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024791#comment-15024791
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45759172
  
--- Diff: storm-core/src/jvm/backtype/storm/Config.java ---
@@ -980,6 +980,126 @@
 public static final String SUPERVISOR_SLOTS_PORTS = 
"supervisor.slots.ports";
 
 /**
+ * What blobstore implementation the supervisor should use.
+ */
+@isString
+public static final String SUPERVISOR_BLOBSTORE = 
"supervisor.blobstore.class";
+public static final Object SUPERVISOR_BLOBSTORE_SCHEMA = String.class;
+
+/**
+ * The jvm opts provided to workers launched by this supervisor. All 
"%ID%" substrings are replaced
+ * with an identifier for this worker. Also, "%WORKER-ID%", 
"%STORM-ID%" and "%WORKER-PORT%" are
+ * replaced with appropriate runtime values for this worker.
+ * The distributed cache target size in MB. This is a soft limit to 
the size of the distributed
+ * cache contents.
+ */
+@isPositiveNumber
+@isInteger
+public static final String SUPERVISOR_LOCALIZER_CACHE_TARGET_SIZE_MB = 
"supervisor.localizer.cache.target.size.mb";
+
+/**
+ * The distributed cache cleanup interval. Controls how often it scans 
to attempt to cleanup
+ * anything over the cache target size.
+ */
+@isPositiveNumber
+@isInteger
+public static final String 
SUPERVISOR_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS = 
"supervisor.localizer.cleanup.interval.ms";
+
+/**
+ * What blobstore implementation the storm client should use.
+ */
+@isString
+public static final String CLIENT_BLOBSTORE = "client.blobstore.class";
+
+/**
+ * What blobstore download parallelism the supervisor should use.
+ */
+@isPositiveNumber
+@isInteger
+public static final String SUPERVISOR_BLOBSTORE_DOWNLOAD_THREAD_COUNT 
= "supervisor.blobstore.download.thread.count";
+
+/**
+ * What blobstore download parallelism the supervisor should use.
+ */
+@isPositiveNumber
+@isInteger
+public static final String SUPERVISOR_BLOBSTORE_DOWNLOAD_MAX_RETRIES = 
"supervisor.blobstore.download.max_retries";
+
+/**
+ * The blobstore super user has all read/write/admin permissions to 
all blobs - user running
+ * the blobstore.
+ */
+@isString
+public static final String BLOBSTORE_SUPERUSER = "blobstore.superuser";
+
+/**
+ * What directory to use for the blobstore. The directory is expected 
to be an
+ * absolute path when using HDFS blobstore, for LocalFsBlobStore it 
could be either
+ * absolute or relative.
+ */
+@isString
+public static final String BLOBSTORE_DIR = "blobstore.dir";
+
+/**
+ * What buffer size to use for the blobstore uploads.
+ */
+@isPositiveNumber
+@isInteger
+public static final String 
STORM_BLOBSTORE_INPUTSTREAM_BUFFER_SIZE_BYTES = 
"storm.blobstore.inputstream.buffer.size.bytes";
+
+/**
+ * Enable the blobstore cleaner. Certain blobstores may only want to 
run the cleaner
+ * on one daemon. Currently Nimbus handles setting this.
+ */
+@isBoolean
+public static final String BLOBSTORE_CLEANUP_ENABLE = 
"blobstore.cleanup.enable";
+
+/**
+ * principal for nimbus/supervisor to use to access secure hdfs for 
the blobstore.
+ */
+@isString
+public static final String BLOBSTORE_HDFS_PRINCIPAL = 
"blobstore.hdfs.principal";
+
+/**
+ * keytab for nimbus/supervisor to use to access secure hdfs for the 
blobstore.
+ */
+@isString
+public static final String BLOBSTORE_HDFS_KEYTAB = 
"blobstore.hdfs.keytab";
+
+/**
+ *  Set replication factor for a blob in HDFS Blobstore Implementation
+ */
+@isPositiveNumber
+@isInteger
+public static final String STORM_BLOBSTORE_REPLICATION_FACTOR = 
"storm.blobstore.replication.factor";
--- End diff --

@d2r 
There is also a TOPOLOGY_BLOBSTORE_MIN_REPLICATION_FACTOR which specifies 
the amount of replication a blob has to undergo before the topology is 
launched, but on the other hand for HDFS blobstore we have to set the 
replication factor for the blob on a HDFS level. I could go ahead with 
HDFS_BLOBSTORE* but that would represent HDFS config rather than a STORM 
config. Welcome for suggestions


> Dist Cache: Basic Functionality
> 

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024801#comment-15024801
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45759993
  
--- Diff: storm-core/src/jvm/backtype/storm/blobstore/BlobStore.java ---
@@ -0,0 +1,446 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package backtype.storm.blobstore;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import javax.security.auth.Subject;
+
+import backtype.storm.nimbus.NimbusInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.daemon.Shutdownable;
+import backtype.storm.generated.AuthorizationException;
+import backtype.storm.generated.KeyNotFoundException;
+import backtype.storm.generated.KeyAlreadyExistsException;
+import backtype.storm.generated.ReadableBlobMeta;
+import backtype.storm.generated.SettableBlobMeta;
+
+/**
+ * Provides a way to store blobs that can be downloaded.
+ * Blobs must be able to be uploaded and listed from Nimbus,
+ * and downloaded from the Supervisors. It is a key value based
+ * store. Key being a string and value being the blob data.
+ *
+ * ACL checking must take place against the provided subject.
+ * If the blob store does not support Security it must validate
+ * that all ACLs set are always WORLD, everything.
+ *
+ * The users can upload their blobs through the blob store command
+ * line utility. The command line utilty also allows us to update,
+ * delete.
+ *
+ * Modifying the replication factor only works for HdfsBlobStore
+ * as for the LocalFsBlobStore the replication is dependent on
+ * the number of Nimbodes available.
+ */
+public abstract class BlobStore implements Shutdownable {
+  public static final Logger LOG = 
LoggerFactory.getLogger(BlobStore.class);
+  private static final Pattern KEY_PATTERN = Pattern.compile("^[\\w 
\\t\\.:_-]+$");
+  protected static final String BASE_BLOBS_DIR_NAME = "blobs";
+
+  /**
+   * Allows us to initialize the blob store
+   * @param conf
+   * @param baseDir
+   */
--- End diff --

@d2r addressed


> Dist Cache: Basic Functionality
> ---
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and 
> downloading dist cache items.  storm-core.ser, storm-conf.ser and storm.jar 
> should be written into the blob store instead of residing locally. We need a 
> default implementation of the blob store that does essentially what nimbus 
> currently does and does not need anything extra.  But having an HDFS backend 
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and 
> provide a working directory for the worker process with symlinks to the 
> blobs.  It should also allow the blobs to be updated and switch the symlink 
> atomically to point to the new blob once it is downloaded.
> All of this is already done by code internal to Yahoo! we are in the process 
> of getting it ready to push back to open source shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-876] Blobstore API

[redsanket]

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45760156
  
--- Diff: storm-core/src/jvm/backtype/storm/blobstore/BlobStore.java ---
@@ -0,0 +1,446 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package backtype.storm.blobstore;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import javax.security.auth.Subject;
+
+import backtype.storm.nimbus.NimbusInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.daemon.Shutdownable;
+import backtype.storm.generated.AuthorizationException;
+import backtype.storm.generated.KeyNotFoundException;
+import backtype.storm.generated.KeyAlreadyExistsException;
+import backtype.storm.generated.ReadableBlobMeta;
+import backtype.storm.generated.SettableBlobMeta;
+
+/**
+ * Provides a way to store blobs that can be downloaded.
+ * Blobs must be able to be uploaded and listed from Nimbus,
+ * and downloaded from the Supervisors. It is a key value based
+ * store. Key being a string and value being the blob data.
+ *
+ * ACL checking must take place against the provided subject.
+ * If the blob store does not support Security it must validate
+ * that all ACLs set are always WORLD, everything.
+ *
+ * The users can upload their blobs through the blob store command
+ * line. The command line utilty also allows us to update,
+ * delete.
+ *
+ * Modifying the replication factor only works for HdfsBlobStore
+ * as for the LocalFsBlobStore the replication is dependent on
+ * the number of Nimbodes available.
+ */
+public abstract class BlobStore implements Shutdownable {
+  public static final Logger LOG = 
LoggerFactory.getLogger(BlobStore.class);
+  private static final Pattern KEY_PATTERN = Pattern.compile("^[\\w 
\\t\\.:_-]+$");
+  protected static final String BASE_BLOBS_DIR_NAME = "blobs";
+
+  /**
+   * Allows us to initialize the blob store
+   * @param conf The storm configuration
+   * @param baseDir The directory path to store the blobs
+   * @param nimbusInfo Contains the nimbus host, port and leadership 
information.
+   */
+  public abstract void prepare(Map conf, String baseDir, NimbusInfo 
nimbusInfo);
+
+  /**
+   * Creates the blob.
+   * @param key Key for the blob.
+   * @param meta Metadata which contains the acls information
+   * @param who Is the subject creating the blob.
+   * @return AtomicOutputStream returns a stream into which the data
+   * can be written into.
--- End diff --

@d2r addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024803#comment-15024803
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45760037
  
--- Diff: storm-core/src/jvm/backtype/storm/blobstore/BlobStore.java ---
@@ -0,0 +1,446 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package backtype.storm.blobstore;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import javax.security.auth.Subject;
+
+import backtype.storm.nimbus.NimbusInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.daemon.Shutdownable;
+import backtype.storm.generated.AuthorizationException;
+import backtype.storm.generated.KeyNotFoundException;
+import backtype.storm.generated.KeyAlreadyExistsException;
+import backtype.storm.generated.ReadableBlobMeta;
+import backtype.storm.generated.SettableBlobMeta;
+
+/**
+ * Provides a way to store blobs that can be downloaded.
+ * Blobs must be able to be uploaded and listed from Nimbus,
+ * and downloaded from the Supervisors. It is a key value based
+ * store. Key being a string and value being the blob data.
+ *
+ * ACL checking must take place against the provided subject.
+ * If the blob store does not support Security it must validate
+ * that all ACLs set are always WORLD, everything.
+ *
+ * The users can upload their blobs through the blob store command
+ * line utility. The command line utilty also allows us to update,
+ * delete.
+ *
+ * Modifying the replication factor only works for HdfsBlobStore
+ * as for the LocalFsBlobStore the replication is dependent on
+ * the number of Nimbodes available.
+ */
+public abstract class BlobStore implements Shutdownable {
+  public static final Logger LOG = 
LoggerFactory.getLogger(BlobStore.class);
+  private static final Pattern KEY_PATTERN = Pattern.compile("^[\\w 
\\t\\.:_-]+$");
+  protected static final String BASE_BLOBS_DIR_NAME = "blobs";
+
+  /**
+   * Allows us to initialize the blob store
+   * @param conf
+   * @param baseDir
+   */
+  public abstract void prepare(Map conf, String baseDir, NimbusInfo 
nimbusInfo);
+
+  /**
+   * Creates the blob.
+   * @param key Key for the blob.
+   * @param meta Metadata which contains the acls information
+   * @param who Is the subject creating the blob.
+   * @return AtomicOutputStream returns a stream into which the data
+   * can be written into.
+   * @throws AuthorizationException
+   * @throws KeyAlreadyExistsException
+   */
+  public abstract AtomicOutputStream createBlob(String key, 
SettableBlobMeta meta, Subject who) throws AuthorizationException, 
KeyAlreadyExistsException;
+
+  /**
+   * Updates the blob data.
+   * @param key Key for the blob.
+   * @param who Is the subject creating the blob.
+   * @return AtomicOutputStream returns a stream into which the data
+   * can be written into.
+   * @throws AuthorizationException
+   * @throws KeyNotFoundException
+   */
+  public abstract AtomicOutputStream updateBlob(String key, Subject who) 
throws AuthorizationException, KeyNotFoundException;
+
+  /**
+   * Gets the current version of metadata for a blob
+   * to be viewed by the user or downloaded by the supervisor.
+   * @param key Key for the blob.
+   * @param who Is the subject creating the blob.
+   * @return AtomicOutputStream returns a stream into which the data
+   * can be written into.
  

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024813#comment-15024813
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45760346
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/blobstore/HdfsBlobStore.java
 ---
@@ -0,0 +1,383 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.hdfs.blobstore;
+
+import backtype.storm.Config;
+import backtype.storm.blobstore.AtomicOutputStream;
+import backtype.storm.blobstore.AtomicOutputStream;
--- End diff --

@d2r addressed


> Dist Cache: Basic Functionality
> ---
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and 
> downloading dist cache items.  storm-core.ser, storm-conf.ser and storm.jar 
> should be written into the blob store instead of residing locally. We need a 
> default implementation of the blob store that does essentially what nimbus 
> currently does and does not need anything extra.  But having an HDFS backend 
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and 
> provide a working directory for the worker process with symlinks to the 
> blobs.  It should also allow the blobs to be updated and switch the symlink 
> atomically to point to the new blob once it is downloaded.
> All of this is already done by code internal to Yahoo! we are in the process 
> of getting it ready to push back to open source shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024818#comment-15024818
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45760493
  
--- Diff: storm-core/src/jvm/backtype/storm/blobstore/BlobSynchronizer.java 
---
@@ -0,0 +1,127 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package backtype.storm.blobstore;
+
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.Utils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.thrift.transport.TTransportException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;;
+
+/**
+ * Is called periodically and updates the nimbus with blobs based on the 
state stored inside the zookeeper
+ */
+public class BlobSynchronizer {
+  private static final Logger LOG = 
LoggerFactory.getLogger(BlobSynchronizer.class);
+  private CuratorFramework zkClient;
+  private Map conf;
+  private BlobStore blobStore;
+  private Set blobStoreKeySet = new HashSet();
+  private Set zookeeperKeySet = new HashSet();
+  private NimbusInfo nimbusInfo;
+
+  public BlobSynchronizer(BlobStore blobStore, Map conf) {
+this.blobStore = blobStore;
+this.conf = conf;
+  }
+
+  public void setNimbusInfo(NimbusInfo nimbusInfo) {
+this.nimbusInfo = nimbusInfo;
+  }
+
+  public void setZookeeperKeySet(Set zookeeperKeySet) {
+this.zookeeperKeySet = zookeeperKeySet;
+  }
+
+  public void setBlobStoreKeySet(Set blobStoreKeySet) {
+this.blobStoreKeySet = blobStoreKeySet;
+  }
+
+  public Set getBlobStoreKeySet() {
+Set keySet = new HashSet();
+keySet.addAll(blobStoreKeySet);
+return keySet;
+  }
+
+  public Set getZookeeperKeySet() {
+Set keySet = new HashSet();
+keySet.addAll(zookeeperKeySet);
+return keySet;
+  }
+
+  public synchronized void syncBlobs() {
+try {
+LOG.debug("Sync blobs - blobstore {} keys {} zookeeperkeys {}", 
blobStore, getBlobStoreKeySet(), getZookeeperKeySet());
+zkClient = Utils.createZKClient(conf);
+deleteKeySetFromBlobStoreNotOnZookeeper(getBlobStoreKeySet(), 
getZookeeperKeySet());
+updateKeySetForBlobStore(getBlobStoreKeySet());
+Set keySetToDownload = 
getKeySetToDownload(getBlobStoreKeySet(), getZookeeperKeySet());
+LOG.debug("Key Set Blobstore-> Zookeeper-> DownloadSet {}-> {}-> {}", 
getBlobStoreKeySet(), getZookeeperKeySet(), keySetToDownload);
+
+for (String key : keySetToDownload) {
+  Set nimbusInfoSet = 
Utils.getNimbodesWithLatestVersionOfBlob(zkClient, key);
+  if(Utils.downloadMissingBlob(conf, blobStore, key, nimbusInfoSet)) {
+Utils.createStateInZookeeper(conf, key, nimbusInfo);
+  }
+}
+if (zkClient !=null) {
+  zkClient.close();
+}
+} catch(InterruptedException exp) {
+LOG.error("InterruptedException {}", exp);
+} catch(TTransportException exp) {
+throw new RuntimeException(exp);
+} catch(Exception exp) {
+throw new RuntimeException(exp);
+}
--- End diff --

@d2r addressed


> Dist Cache: Basic Functionality
> ---
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to 

[GitHub] storm pull request: [STORM-876] Blobstore API

[redsanket]

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45760606
  
--- Diff: storm-core/src/jvm/backtype/storm/blobstore/IBlobWatcher.java ---
@@ -0,0 +1,25 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package backtype.storm.blobstore;
+
+/**
+ * Provides a way to know when a blob changes.
+ */
+public interface IBlobWatcher {
+public void blobChanged(String key);
+}
--- End diff --

@d2r removed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024819#comment-15024819
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45760519
  
--- Diff: storm-core/src/jvm/backtype/storm/blobstore/ClientBlobStore.java 
---
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package backtype.storm.blobstore;
+
+import backtype.storm.Config;
+import backtype.storm.daemon.Shutdownable;
+import backtype.storm.generated.AccessControl;
+import backtype.storm.generated.AccessControlType;
+import backtype.storm.generated.AuthorizationException;
+import backtype.storm.generated.ReadableBlobMeta;
+import backtype.storm.generated.SettableBlobMeta;
+import backtype.storm.generated.KeyAlreadyExistsException;
+import backtype.storm.generated.KeyNotFoundException;
+import backtype.storm.utils.NimbusClient;
+import backtype.storm.utils.Utils;
--- End diff --

@d2r addressed


> Dist Cache: Basic Functionality
> ---
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and 
> downloading dist cache items.  storm-core.ser, storm-conf.ser and storm.jar 
> should be written into the blob store instead of residing locally. We need a 
> default implementation of the blob store that does essentially what nimbus 
> currently does and does not need anything extra.  But having an HDFS backend 
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and 
> provide a working directory for the worker process with symlinks to the 
> blobs.  It should also allow the blobs to be updated and switch the symlink 
> atomically to point to the new blob once it is downloaded.
> All of this is already done by code internal to Yahoo! we are in the process 
> of getting it ready to push back to open source shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024817#comment-15024817
 ] 

ASF GitHub Bot commented on STORM-876:
--

Github user redsanket commented on a diff in the pull request:

https://github.com/apache/storm/pull/845#discussion_r45760458
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/blobstore/HdfsBlobStore.java
 ---
@@ -0,0 +1,383 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.hdfs.blobstore;
+
+import backtype.storm.Config;
+import backtype.storm.blobstore.AtomicOutputStream;
+import backtype.storm.blobstore.AtomicOutputStream;
+import backtype.storm.blobstore.BlobStore;
+import backtype.storm.blobstore.BlobStoreAclHandler;
+import backtype.storm.blobstore.BlobStoreFile;
+import backtype.storm.blobstore.InputStreamWithMeta;
+import backtype.storm.generated.AuthorizationException;
+import backtype.storm.generated.KeyNotFoundException;
+import backtype.storm.generated.KeyAlreadyExistsException;
+import backtype.storm.generated.ReadableBlobMeta;
+import backtype.storm.generated.SettableBlobMeta;
+import backtype.storm.nimbus.NimbusInfo;
+import backtype.storm.utils.NimbusClient;
+import backtype.storm.utils.Utils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift7.TBase;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import java.io.ByteArrayOutputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.Iterator;
+import java.util.Map;
+
+import static backtype.storm.blobstore.BlobStoreAclHandler.ADMIN;
+import static backtype.storm.blobstore.BlobStoreAclHandler.READ;
+import static backtype.storm.blobstore.BlobStoreAclHandler.WRITE;
+
+/**
+ * Provides a HDFS file system backed blob store implementation.
+ * Note that this provides an api for having HDFS be the backing store for 
the blobstore,
+ * it is not a service/daemon.
+ */
+public class HdfsBlobStore extends BlobStore {
+  public static final Logger LOG = 
LoggerFactory.getLogger(HdfsBlobStore.class);
+  private static final String DATA_PREFIX = "data_";
+  private static final String META_PREFIX = "meta_";
+  private BlobStoreAclHandler _aclHandler;
+  private HdfsBlobStoreImpl _hbs;
+  private Subject _localSubject;
+  private Map conf;
+
+  /**
+   * Get the subject from Hadoop so we can use it to validate the acls. 
There is no direct
+   * interface from UserGroupInformation to get the subject, so do a doAs 
and get the context.
+   * We could probably run everything in the doAs but for now just grab 
the subject.
+   */
+  private Subject getHadoopUser() {
+Subject subj;
+try {
+  subj = UserGroupInformation.getCurrentUser().doAs(
+  new PrivilegedAction() {
+@Override
+public Subject run() {
+  return Subject.getSubject(AccessController.getContext());
+}
+  });
+} catch (IOException e) {
+  throw new RuntimeException("Error creating subject and logging user 
in!", e);
+}
+return subj;
+  }
+
+  /**
+   * If who is null then we want to use the user hadoop says we are.
+   * Required for the supervisor to call these routines as its not
+   * logged in as anyone.
+   */
+  private Subject checkAndGetSubject(Subject who) {
+if (who == null) {
+  return _localSubject;
+}
+return who;
+  }
+
+  @Override
+  public void prepare(Map conf, String overrideBase, NimbusInfo 
nimbusInfo) {