[jira] [Created] (STORM-1100) TNonblockingServer [ERROR] Unexpected exception while invoking!

[SamMohel (JIRA)]

SamMohel created STORM-1100:
---

 Summary: TNonblockingServer [ERROR] Unexpected exception while 
invoking!
 Key: STORM-1100
 URL: https://issues.apache.org/jira/browse/STORM-1100
 Project: Apache Storm
  Issue Type: Bug
  Components: examples
 Environment: ubuntu 12.04 LTS 
zookeeper-3.4.5 
zeroMQ2.1.7 
storm 0.8.2
Reporter: SamMohel


I'm new to storm and first time to write issue here so Sorry if i made any 
mistake 

when i tried to submit topology spout didn't emit anything (ZERO)
then found in Supervisor log file that hasn't still start 
and in Worker log file found this 
Error on initialization of server mk-worker java.io.IOException: No such file 
or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) at 
java.io.File.createNewFile(Unknown Source) at 
backtype.storm.util$touch.invoke(util.clj:432) at 
backtype.storm.daemon.worker$fn__4348$exec_fn__1228__auto4349.invoke(worker.clj:331)
 at clojure.lang.AFn.applyToHelper(AFn.java:185) at 
clojure.lang.AFn.applyTo(AFn.java:151) at 
clojure.core$apply.invoke(core.clj:601) at 
backtype.storm.daemon.worker$fn__4348$mk_worker__4404.doInvoke(worker.clj:323) 
at clojure.lang.RestFn.invoke(RestFn.java:512) at 
backtype.storm.daemon.worker$_main.invoke(worker.clj:433) at 
clojure.lang.AFn.applyToHelper(AFn.java:172) at 
clojure.lang.AFn.applyTo(AFn.java:151) at 
backtype.storm.daemon.worker.main(Unknown Source) 2015-10-09 02:58:47 util 
[INFO] Halting process: ("Error on initialization")

and after few minutes in Nimbus log file 
TNonblockingServer [ERROR] Unexpected exception while invoking! 
java.io.FileNotFoundException: File 
'storm-local/nimbus/stormdist/fsd-1-1444356616/stormcode.ser' does not exist at 
org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:137) at 
org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1135) at 
backtype.storm.daemon.nimbus$read_storm_topology.invoke(nimbus.clj:305) at 
backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto__$reify__3605.getTopologyInfo(nimbus.clj:1066)
  




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-912: Support SSL on Logviewer

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/717


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-912) Support SSL on Logviewer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950079#comment-14950079
 ] 

ASF GitHub Bot commented on STORM-912:
--

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/717


> Support SSL on Logviewer
> 
>
> Key: STORM-912
> URL: https://issues.apache.org/jira/browse/STORM-912
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
>Priority: Minor
> Fix For: 0.11.0
>
>
> Support SSL on the logviewer like it is in the UI.
> Also detect what method we're using and make sure logviewer links in the UI 
> are pointing to the appropriate Logviewer endpoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1100) TNonblockingServer [ERROR] Unexpected exception while invoking!

[SamMohel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SamMohel reassigned STORM-1100:
---

Assignee: SamMohel

> TNonblockingServer [ERROR] Unexpected exception while invoking!
> ---
>
> Key: STORM-1100
> URL: https://issues.apache.org/jira/browse/STORM-1100
> Project: Apache Storm
>  Issue Type: Bug
>  Components: examples
> Environment: ubuntu 12.04 LTS 
> zookeeper-3.4.5 
> zeroMQ2.1.7 
> storm 0.8.2
>Reporter: SamMohel
>Assignee: SamMohel
>
> I'm new to storm and first time to write issue here so Sorry if i made any 
> mistake 
> when i tried to submit topology spout didn't emit anything (ZERO)
> then found in Supervisor log file that hasn't still start 
> and in Worker log file found this 
> Error on initialization of server mk-worker java.io.IOException: No such file 
> or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) 
> at java.io.File.createNewFile(Unknown Source) at 
> backtype.storm.util$touch.invoke(util.clj:432) at 
> backtype.storm.daemon.worker$fn__4348$exec_fn__1228__auto4349.invoke(worker.clj:331)
>  at clojure.lang.AFn.applyToHelper(AFn.java:185) at 
> clojure.lang.AFn.applyTo(AFn.java:151) at 
> clojure.core$apply.invoke(core.clj:601) at 
> backtype.storm.daemon.worker$fn__4348$mk_worker__4404.doInvoke(worker.clj:323)
>  at clojure.lang.RestFn.invoke(RestFn.java:512) at 
> backtype.storm.daemon.worker$_main.invoke(worker.clj:433) at 
> clojure.lang.AFn.applyToHelper(AFn.java:172) at 
> clojure.lang.AFn.applyTo(AFn.java:151) at 
> backtype.storm.daemon.worker.main(Unknown Source) 2015-10-09 02:58:47 util 
> [INFO] Halting process: ("Error on initialization")
> and after few minutes in Nimbus log file 
> TNonblockingServer [ERROR] Unexpected exception while invoking! 
> java.io.FileNotFoundException: File 
> 'storm-local/nimbus/stormdist/fsd-1-1444356616/stormcode.ser' does not exist 
> at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:137) at 
> org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1135) at 
> backtype.storm.daemon.nimbus$read_storm_topology.invoke(nimbus.clj:305) at 
> backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto__$reify__3605.getTopologyInfo(nimbus.clj:1066)
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-912) Support SSL on Logviewer

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-912:
---
Fix Version/s: (was: 0.11.0)
   0.10.0

> Support SSL on Logviewer
> 
>
> Key: STORM-912
> URL: https://issues.apache.org/jira/browse/STORM-912
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
>Priority: Minor
> Fix For: 0.10.0
>
>
> Support SSL on the logviewer like it is in the UI.
> Also detect what method we're using and make sure logviewer links in the UI 
> are pointing to the appropriate Logviewer endpoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1104] Nimbus HA fails to find newly dow...

[schonfeld]

GitHub user schonfeld opened a pull request:

https://github.com/apache/storm/pull/794

[STORM-1104] Nimbus HA fails to find newly downloaded code files

Nimbus HA using Local File System code distribution is broken. We seem to 
be "caching" the return value of `(code-ids (:conf nimbus))` in the sync-code 
method, by overriding the `code-ids` var from a method 
([nimbus.clj#854](../blob/master/storm-core/src/clj/backtype/storm/daemon/nimbus.clj#L854)
 `defn code-ids`), to a local var 
([nimbus.clj#1669](../blob/master/storm-core/src/clj/backtype/storm/daemon/nimbus.clj#L1669)
 `code-ids (set (code-ids (:conf nimbus)))`).

The problem is, that after downloading code for missing topologies, 
sync-code doesn't realize it has gotten the new missing topology files.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/schonfeld/storm 
nimbus-ha-fails-to-find-code-files

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/794.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #794


commit ba1250993d10ffc523c9f5464371fbeb406d216f
Author: Michael Schonfeld 
Date:   2015-10-09T23:48:41Z

dont override code-ids method




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (STORM-1104) Nimbus HA fails to find newly downloaded code files

[Michael Schonfeld (JIRA)]

Michael Schonfeld created STORM-1104:


 Summary: Nimbus HA fails to find newly downloaded code files
 Key: STORM-1104
 URL: https://issues.apache.org/jira/browse/STORM-1104
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Affects Versions: 0.11.0
Reporter: Michael Schonfeld


We seem to be "caching" the return value of `(code-ids (:conf nimbus))` in the 
sync-code method, by overriding the `code-ids` var from a method (#854 `defn 
code-ids`), to a local var (#1669 `code-ids (set (code-ids (:conf nimbus)))`).

The problem is, that after downloading code for missing topologies, sync-code 
doesn't realize it has gotten the new missing topology files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1104) Nimbus HA fails to find newly downloaded code files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951417#comment-14951417
 ] 

ASF GitHub Bot commented on STORM-1104:
---

GitHub user schonfeld opened a pull request:

https://github.com/apache/storm/pull/794

[STORM-1104] Nimbus HA fails to find newly downloaded code files

Nimbus HA using Local File System code distribution is broken. We seem to 
be "caching" the return value of `(code-ids (:conf nimbus))` in the sync-code 
method, by overriding the `code-ids` var from a method 
([nimbus.clj#854](../blob/master/storm-core/src/clj/backtype/storm/daemon/nimbus.clj#L854)
 `defn code-ids`), to a local var 
([nimbus.clj#1669](../blob/master/storm-core/src/clj/backtype/storm/daemon/nimbus.clj#L1669)
 `code-ids (set (code-ids (:conf nimbus)))`).

The problem is, that after downloading code for missing topologies, 
sync-code doesn't realize it has gotten the new missing topology files.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/schonfeld/storm 
nimbus-ha-fails-to-find-code-files

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/794.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #794


commit ba1250993d10ffc523c9f5464371fbeb406d216f
Author: Michael Schonfeld 
Date:   2015-10-09T23:48:41Z

dont override code-ids method




> Nimbus HA fails to find newly downloaded code files
> ---
>
> Key: STORM-1104
> URL: https://issues.apache.org/jira/browse/STORM-1104
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>
> Nimbus HA using Local File System code distribution is broken. We seem to be 
> "caching" the return value of `(code-ids (:conf nimbus))` in the sync-code 
> method, by overriding the `code-ids` var from a method (#854 `defn 
> code-ids`), to a local var (#1669 `code-ids (set (code-ids (:conf nimbus)))`).
> The problem is, that after downloading code for missing topologies, sync-code 
> doesn't realize it has gotten the new missing topology files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1104) Nimbus HA fails to find newly downloaded code files

[Michael Schonfeld (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Schonfeld updated STORM-1104:
-
Assignee: Michael Schonfeld

> Nimbus HA fails to find newly downloaded code files
> ---
>
> Key: STORM-1104
> URL: https://issues.apache.org/jira/browse/STORM-1104
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>
> We seem to be "caching" the return value of `(code-ids (:conf nimbus))` in 
> the sync-code method, by overriding the `code-ids` var from a method (#854 
> `defn code-ids`), to a local var (#1669 `code-ids (set (code-ids (:conf 
> nimbus)))`).
> The problem is, that after downloading code for missing topologies, sync-code 
> doesn't realize it has gotten the new missing topology files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1104] Nimbus HA fails to find newly dow...

[Parth-Brahmbhatt]

Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/794#issuecomment-147019720
  
+1, good catch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Does Storm work with Spring

[John Reilly]

You can just create the ApplicationConrext using a singleton and make all
fields in the bolt or spout code which reference the app context transient.
On Fri, Oct 9, 2015 at 08:35 Javier Gonzalez  wrote:

> IIRC, only if everything you use in your spouts and bolts is serializable.
> On Oct 6, 2015 11:29 PM, "Ankur Garg"  wrote:
>
>> Hi Ravi ,
>>
>> I was able to make an Integration with Spring but the problem is that I
>> have to autowire for every bolt and spout . That means that even if i
>> parallelize spout and bolt it will get started to each instance  . Is there
>> some way that I only have to do for bolts and spouts once (I mean if I
>> parallelize bolts or spouts individually it can share the conf from
>> somewhere) . IS this possible??
>>
>> Thanks
>> Ankur
>>
>> On Tue, Sep 29, 2015 at 7:57 PM, Ravi Sharma  wrote:
>>
>>> Yes this is for annotation also...
>>>
>>> you can call this method in prepare()  method of bolt and onOpen() method
>>> in every Spout and make sure you don't use any autowire bean before this
>>> call.
>>>
>>>
>>>
>>>
>>> Ravi.
>>>
>>>
>>>
>>>
>>> On Tue, Sep 29, 2015 at 2:22 PM, Ankur Garg 
>>> wrote:
>>>
>>> > Hi Ravi ,
>>> >
>>> > Thanks for your reply . I am using annotation based configuration and
>>> using
>>> > Spring Boot.
>>> >
>>> > Any idea how to do it using annotations ?
>>> >
>>> >
>>> >
>>> > On Tue, Sep 29, 2015 at 6:41 PM, Ravi Sharma 
>>> wrote:
>>> >
>>> > > Bolts and Spouts are created by Storm and not known to Spring
>>> Context.
>>> > You
>>> > > need to manually add them to SpringContext, there are few methods
>>> > available
>>> > > i.e.
>>> > >
>>> > >
>>> > >
>>> >
>>> SpringContext.getContext().getAutowireCapableBeanFactory().autowireBeanProperties(this,
>>> > > AutowireCapableBeanFactory.AUTOWIRE_AUTODETECT, false);
>>> > >
>>> > > SpringContext is my own class where i have injected SpringContext so
>>> > > SpringContext.getContext() returns the actuall Spring Context
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > Ravi.
>>> > >
>>> > >
>>> > > On Tue, Sep 29, 2015 at 1:03 PM, Ankur Garg 
>>> > wrote:
>>> > >
>>> > > > Hi ,
>>> > > >
>>> > > > I am building a Storm topology with set of Spouts and Bolts  and
>>> also
>>> > > using
>>> > > > Spring for Dependency Injection .
>>> > > >
>>> > > > Unfortunately , none of my fields are getting autowired even
>>> though I
>>> > > have
>>> > > > declared all my spouts and Bolts as @Components .
>>> > > >
>>> > > > However the place where I am declaring my topology , Spring is
>>> working
>>> > > fine
>>> > > > .
>>> > > >
>>> > > > Is it because cluster.submitTopology("test", conf,
>>> > > > builder.createTopology())
>>> > > >  submits the topology to a cluster (locally it spawns different
>>> thread
>>> > > for
>>> > > > Spouts and Bolts) that Autowiring is not working?
>>> > > >
>>> > > > Please suggest .
>>> > > >
>>> > >
>>> >
>>>
>>
>>

[jira] [Updated] (STORM-1104) Nimbus HA fails to find newly downloaded code files

[Michael Schonfeld (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Schonfeld updated STORM-1104:
-
Description: 
Nimbus HA using Local File System code distribution is broken. We seem to be 
"caching" the return value of `(code-ids (:conf nimbus))` in the sync-code 
method, by overriding the `code-ids` var from a method (#854 `defn code-ids`), 
to a local var (#1669 `code-ids (set (code-ids (:conf nimbus)))`).

The problem is, that after downloading code for missing topologies, sync-code 
doesn't realize it has gotten the new missing topology files.

  was:
We seem to be "caching" the return value of `(code-ids (:conf nimbus))` in the 
sync-code method, by overriding the `code-ids` var from a method (#854 `defn 
code-ids`), to a local var (#1669 `code-ids (set (code-ids (:conf nimbus)))`).

The problem is, that after downloading code for missing topologies, sync-code 
doesn't realize it has gotten the new missing topology files.


> Nimbus HA fails to find newly downloaded code files
> ---
>
> Key: STORM-1104
> URL: https://issues.apache.org/jira/browse/STORM-1104
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>
> Nimbus HA using Local File System code distribution is broken. We seem to be 
> "caching" the return value of `(code-ids (:conf nimbus))` in the sync-code 
> method, by overriding the `code-ids` var from a method (#854 `defn 
> code-ids`), to a local var (#1669 `code-ids (set (code-ids (:conf nimbus)))`).
> The problem is, that after downloading code for missing topologies, sync-code 
> doesn't realize it has gotten the new missing topology files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1104) Nimbus HA fails to find newly downloaded code files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951470#comment-14951470
 ] 

ASF GitHub Bot commented on STORM-1104:
---

Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/794#issuecomment-147019720
  
+1, good catch.


> Nimbus HA fails to find newly downloaded code files
> ---
>
> Key: STORM-1104
> URL: https://issues.apache.org/jira/browse/STORM-1104
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>
> Nimbus HA using Local File System code distribution is broken. We seem to be 
> "caching" the return value of `(code-ids (:conf nimbus))` in the sync-code 
> method, by overriding the `code-ids` var from a method (#854 `defn 
> code-ids`), to a local var (#1669 `code-ids (set (code-ids (:conf nimbus)))`).
> The problem is, that after downloading code for missing topologies, sync-code 
> doesn't realize it has gotten the new missing topology files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1096: Fix some issues with impersonation...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/787#issuecomment-146762852
  
I found that I made a mistake while backporting. At least bin/storm.py 
should be fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1096) UI tries to impersonate wrong user when getting topology conf for authorization, impersonation is allowed by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949942#comment-14949942
 ] 

ASF GitHub Bot commented on STORM-1096:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/787#issuecomment-146762852
  
I found that I made a mistake while backporting. At least bin/storm.py 
should be fixed.


> UI tries to impersonate wrong user when getting topology conf for 
> authorization, impersonation is allowed by default
> 
>
> Key: STORM-1096
> URL: https://issues.apache.org/jira/browse/STORM-1096
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.10.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> We have started using 0.10.0 under load and found a few issues around the UI 
> and impersonation.
> The UI when trying to connect to nimbus will impersonate other users.  
> Nimbus, by default allows impersonation and just outputs a warning message 
> that it is allowed.  We really should default to not allowing impersonation.  
> having the authorizer configured by default does not hurt when running 
> insecure because impersonation is not possible, but when security is enabled 
> if someone forgets to set this config we are now insecure by default.
> If you do set all of that up correctly the UI now can impersonate the wrong 
> user when connecting to nimbus.
> The UI decides which user to impersonate by pulling it from the request 
> context.  The requestContext is populated from the HttpRequest when 
> assert-authorized-user is called.  assert-authorized-user takes a 
> topology-conf as a parameter.  The only way to get this topology conf is to 
> talk to nimbus, which will get the wrong user because the request context has 
> not been populated yet.
> This just because a huge pain for users who way too often will not be able to 
> see pages on the UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-188: Allow user to specifiy full configu...

[clockfly]

Github user clockfly closed the pull request at:

https://github.com/apache/storm/pull/120


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-188: Allow user to specifiy full configu...

[clockfly]

Github user clockfly commented on the pull request:

https://github.com/apache/storm/pull/120#issuecomment-146773478
  
Sure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1079. Batch Puts to HBase.

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/772#issuecomment-146766892
  
@harshach 
I'm fine to use message.timeout.sec / 2 as tick secs.

But, there's compile failure from Travis CI.

```
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on 
project storm-hbase: Compilation failure
[ERROR] 
/home/travis/build/apache/storm/external/storm-hbase/src/main/java/org/apache/storm/hbase/bolt/HBaseBolt.java:[81,66]
 cannot find symbol
[ERROR] symbol:   variable tickTupleInterval
[ERROR] location: class org.apache.storm.hbase.bolt.HBaseBolt
```

Should change tickTupleInterval to flushIntervalSecs.

If "errored check" has raised, we should check build logs from Travis CI 
cause it could be possible to be compile errors.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: advance kafka offset when deserializer yields ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/758


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1095] Tuple.getSourceGlobalStreamid() h...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/786#discussion_r41599977
  
--- Diff: storm-core/src/jvm/backtype/storm/tuple/Tuple.java ---
@@ -37,9 +37,15 @@
 /**
  * Returns the global stream id (component + stream) of this tuple.
--- End diff --

@mjsax 
Could you append comment to explain 'broken naming convention, use 
getSourceGlobalStreamId() instead'?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1095) Tuple.getSourceGlobalStreamid() has wrong camel-case naming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949943#comment-14949943
 ] 

ASF GitHub Bot commented on STORM-1095:
---

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/786#discussion_r41599977
  
--- Diff: storm-core/src/jvm/backtype/storm/tuple/Tuple.java ---
@@ -37,9 +37,15 @@
 /**
  * Returns the global stream id (component + stream) of this tuple.
--- End diff --

@mjsax 
Could you append comment to explain 'broken naming convention, use 
getSourceGlobalStreamId() instead'?


> Tuple.getSourceGlobalStreamid() has wrong camel-case naming
> ---
>
> Key: STORM-1095
> URL: https://issues.apache.org/jira/browse/STORM-1095
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Trivial
>
> The method Tuple.getSourceGlobalStreamid() should be named 
> Tuple.getSourceGlobalStreamId() to follow camel-case naming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1094) kafka offset does not advance when deserializer yields no object

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-1094:

Affects Version/s: 0.9.6

> kafka offset does not advance when deserializer yields no object 
> -
>
> Key: STORM-1094
> URL: https://issues.apache.org/jira/browse/STORM-1094
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.10.0, 0.11.0, 0.9.6
>Reporter: Pete Prokopowicz
>  Labels: patch
>
> If a custom deserializer returns an empty list of objects as a result of 
> deserializing messages from kafka, the partition manager does not advance the 
> kafka offset, and the same message will continue to be deserialized and 
> return nothing.  The code works if the deserializer returns null, but not if 
> it returns an empty list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: advance kafka offset when deserializer yields ...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/758#issuecomment-146770261
  
Great, I'll merge it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (STORM-1094) kafka offset does not advance when deserializer yields no object

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-1094:

Affects Version/s: 0.10.0

> kafka offset does not advance when deserializer yields no object 
> -
>
> Key: STORM-1094
> URL: https://issues.apache.org/jira/browse/STORM-1094
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.10.0, 0.11.0
>Reporter: Pete Prokopowicz
>  Labels: patch
>
> If a custom deserializer returns an empty list of objects as a result of 
> deserializing messages from kafka, the partition manager does not advance the 
> kafka offset, and the same message will continue to be deserialized and 
> return nothing.  The code works if the deserializer returns null, but not if 
> it returns an empty list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1090) Nimbus HA should support `storm.local.hostname`

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1090.
-
   Resolution: Fixed
 Assignee: Michael Schonfeld
Fix Version/s: 0.11.0

Thanks, [~BaconSeason],
I merged into master.

> Nimbus HA should support `storm.local.hostname`
> ---
>
> Key: STORM-1090
> URL: https://issues.apache.org/jira/browse/STORM-1090
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
> Fix For: 0.11.0
>
>
> Nimbus HA's `NimbusInfo` class relies on each nimbus's hostname for network 
> reachability. This is a show-stopper in situations that utilize 
> Config.STORM_LOCAL_HOSTNAME /  `storm.local.hostname`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1091: Add tick tuple unit tests for hdfs...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/784


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1079) Batch Puts to HBase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949951#comment-14949951
 ] 

ASF GitHub Bot commented on STORM-1079:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/772#issuecomment-146766892
  
@harshach 
I'm fine to use message.timeout.sec / 2 as tick secs.

But, there's compile failure from Travis CI.

```
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on 
project storm-hbase: Compilation failure
[ERROR] 
/home/travis/build/apache/storm/external/storm-hbase/src/main/java/org/apache/storm/hbase/bolt/HBaseBolt.java:[81,66]
 cannot find symbol
[ERROR] symbol:   variable tickTupleInterval
[ERROR] location: class org.apache.storm.hbase.bolt.HBaseBolt
```

Should change tickTupleInterval to flushIntervalSecs.

If "errored check" has raised, we should check build logs from Travis CI 
cause it could be possible to be compile errors.


> Batch Puts to HBase
> ---
>
> Key: STORM-1079
> URL: https://issues.apache.org/jira/browse/STORM-1079
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hbase
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

[rsltrifork]

Github user rsltrifork commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-146779690
  
Thanks for the quick reply, revans2.

> How do you know that the bolt is waiting in a controlled manner?

A Bolt sending to an external system can include a circuit breaker, so it 
can keep waiting while the CB is tripped, while regularly retrying sends to 
check if the recipient is up.

While doing this, the Bolt could also tell Storm that processing of the 
current tuple is currently blocked, and that Storm should reset the tuple 
timeout for it (and subsequent inflight tuples):
`collector.resetTupleTimeout(tuple);`

> Even if you do know that how do you know that if it is waiting in a 
controlled manner that it has not lost track of a tuple?

The bolt is continuously saying that it still has the tuple.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-886) Automatic Back Pressure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950027#comment-14950027
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user rsltrifork commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-146779690
  
Thanks for the quick reply, revans2.

> How do you know that the bolt is waiting in a controlled manner?

A Bolt sending to an external system can include a circuit breaker, so it 
can keep waiting while the CB is tripped, while regularly retrying sends to 
check if the recipient is up.

While doing this, the Bolt could also tell Storm that processing of the 
current tuple is currently blocked, and that Storm should reset the tuple 
timeout for it (and subsequent inflight tuples):
`collector.resetTupleTimeout(tuple);`

> Even if you do know that how do you know that if it is waiting in a 
controlled manner that it has not lost track of a tuple?

The bolt is continuously saying that it still has the tuple.


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Fix For: 0.11.0
>
> Attachments: aSimpleExampleOfBackpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-912: Support SSL on Logviewer

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/717#issuecomment-146767039
  
Great! +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-912) Support SSL on Logviewer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949952#comment-14949952
 ] 

ASF GitHub Bot commented on STORM-912:
--

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/717#issuecomment-146767039
  
Great! +1


> Support SSL on Logviewer
> 
>
> Key: STORM-912
> URL: https://issues.apache.org/jira/browse/STORM-912
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
>Priority: Minor
> Fix For: 0.11.0
>
>
> Support SSL on the logviewer like it is in the UI.
> Also detect what method we're using and make sure logviewer links in the UI 
> are pointing to the appropriate Logviewer endpoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-188) Allow user to specify full configuration path when running storm command

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949995#comment-14949995
 ] 

ASF GitHub Bot commented on STORM-188:
--

Github user clockfly closed the pull request at:

https://github.com/apache/storm/pull/120


> Allow user to specify full configuration path when running storm command
> 
>
> Key: STORM-188
> URL: https://issues.apache.org/jira/browse/STORM-188
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Sean Zhong
>Assignee: Sriharsha Chintalapani
>Priority: Minor
> Fix For: 0.10.0
>
> Attachments: search_local_path_for_config.patch, storm-188.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Currently, storm will only look up configuration path in java classpath. We 
> should also allow user to specify full configuration path. This is very 
> important for a shared cluster environment, like YARN. Multiple storm cluster 
> may runs with different configuration, but share same binary folder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1090] Nimbus HA should support `storm.l...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/783


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1090) Nimbus HA should support `storm.local.hostname`

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950004#comment-14950004
 ] 

ASF GitHub Bot commented on STORM-1090:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/783


> Nimbus HA should support `storm.local.hostname`
> ---
>
> Key: STORM-1090
> URL: https://issues.apache.org/jira/browse/STORM-1090
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Michael Schonfeld
>
> Nimbus HA's `NimbusInfo` class relies on each nimbus's hostname for network 
> reachability. This is a show-stopper in situations that utilize 
> Config.STORM_LOCAL_HOSTNAME /  `storm.local.hostname`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1091) Add unit test for tick tuples to HiveBolt and HdfsBolt

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1091.
-
   Resolution: Fixed
Fix Version/s: 0.11.0

Thanks [~dossett],
I merged into master.


> Add unit test for tick tuples to HiveBolt and HdfsBolt
> --
>
> Key: STORM-1091
> URL: https://issues.apache.org/jira/browse/STORM-1091
> Project: Apache Storm
>  Issue Type: Test
>  Components: storm-hdfs, storm-hive
>Reporter: Aaron Dossett
>Assignee: Aaron Dossett
>Priority: Minor
> Fix For: 0.11.0
>
>
> Add tick tuple unit tests to HiveBolt and HdfsBolt.
> The storm-starter module contains a helpful MockTupleHelpers class.  
> Relocating it to the storm-core test suite will make these tests easier to 
> implement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1091) Add unit test for tick tuples to HiveBolt and HdfsBolt

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950019#comment-14950019
 ] 

ASF GitHub Bot commented on STORM-1091:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/784


> Add unit test for tick tuples to HiveBolt and HdfsBolt
> --
>
> Key: STORM-1091
> URL: https://issues.apache.org/jira/browse/STORM-1091
> Project: Apache Storm
>  Issue Type: Test
>  Components: storm-hdfs, storm-hive
>Reporter: Aaron Dossett
>Assignee: Aaron Dossett
>Priority: Minor
>
> Add tick tuple unit tests to HiveBolt and HdfsBolt.
> The storm-starter module contains a helpful MockTupleHelpers class.  
> Relocating it to the storm-core test suite will make these tests easier to 
> implement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1085) Compile comparison, arithmetic, and field reference expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951316#comment-14951316
 ] 

ASF GitHub Bot commented on STORM-1085:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/789


> Compile comparison, arithmetic, and field reference expressions
> ---
>
> Key: STORM-1085
> URL: https://issues.apache.org/jira/browse/STORM-1085
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> This jira tracks the effort of implementing the comparison, arithmetic and 
> the field reference expressions in the expression compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1085. Compile comparison, arithmetic, an...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/789


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (STORM-1085) Compile comparison, arithmetic, and field reference expressions

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai closed STORM-1085.
-
Resolution: Fixed

Thanks [~sriharsha] for the reviews and commits.

> Compile comparison, arithmetic, and field reference expressions
> ---
>
> Key: STORM-1085
> URL: https://issues.apache.org/jira/browse/STORM-1085
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> This jira tracks the effort of implementing the comparison, arithmetic and 
> the field reference expressions in the expression compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1105) Compile the logical plans into LLVM functions

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated STORM-1105:
--
Component/s: storm-sql
 Issue Type: New Feature  (was: Bug)

> Compile the logical plans into LLVM functions
> -
>
> Key: STORM-1105
> URL: https://issues.apache.org/jira/browse/STORM-1105
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> This jira tracks the effort on compiling the stages of the logical plans into 
> LLVM  functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1105) Compile the logical plans into LLVM functions

[Haohui Mai (JIRA)]

Haohui Mai created STORM-1105:
-

 Summary: Compile the logical plans into LLVM functions
 Key: STORM-1105
 URL: https://issues.apache.org/jira/browse/STORM-1105
 Project: Apache Storm
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira tracks the effort on compiling the stages of the logical plans into 
LLVM  functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-117) Storm should provide client library

[Simon Cooper (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950648#comment-14950648
 ] 

Simon Cooper commented on STORM-117:


I would like to see this also. Any programs/libraries communicating with nimbus 
and deploying topologies currently have to ship with the entire storm runtime, 
which bloats the library quite a bit.

> Storm should provide client library
> ---
>
> Key: STORM-117
> URL: https://issues.apache.org/jira/browse/STORM-117
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Daniel Cruver
>
> Developers that would like to use distributed services such as Storm should 
> have client libraries that allow users to deploy applications (Topologies) 
> and send requests to these application without requiring dependencies only 
> required by the services. 
> Definitions:
> Service Environment - Storm Nimbus, DRPC and, workers
> Client Environment - Standalone JVM, Web Application etc.
> One example of this is hadoop-client. Before it was created user would use 
> hadoop-core and they would be force to include or manually exclude servlet 
> artifacts and other such artifacts that may cause classpath issues in the 
> client's environment. This will cut down on confusion to new storm users that 
> are unfamiliar with the details of storm and logging frameworks or user 
> integrate storm into existing applications.
> One such example is storm now includes Logback-classic, when user adds this 
> to a project that uses slf4j-API it causes their logging system to break. 
> This happen to our project until we manually excluded Logback. In our 
> environment like others uses log4j for logging.  Yes Logback is an 
> improvement over log4j but you can't expect others to change their standard 
> logging framework because of one external dependency.  Also there are newer 
> alternatives like log4j 2 and jboss logging that will compete to be the 
> upgrade path for those that would like to update.
> To start off with I do not expect storm to allow users to change the logging 
> framework used for client code while executing in the context of storm 
> workers just the code run from the requesting application. Note allowing 
> system administrators to change the logging framework would be easy since you 
> are using slf4j.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1093) Launching Workers with resources specified in resource-aware schedulers

[Zhuo Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950671#comment-14950671
 ] 

Zhuo Liu commented on STORM-1093:
-

Thanks [~jerrypeng], nicely STORM-893 (RAS) has just been merged.

> Launching Workers with resources specified in resource-aware schedulers
> ---
>
> Key: STORM-1093
> URL: https://issues.apache.org/jira/browse/STORM-1093
> Project: Apache Storm
>  Issue Type: Story
>  Components: storm-core
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>
> Currently, we have Resource Aware Scheduler (STORM-894) in nimbus, which can 
> allocate different types of resource (CPU, onheap-memory, offheap-memory) to 
> the workers assigned to each topology's tasks.
> However, such resources are not visible to the supervisor, therefore, 
> supervisor will still launch workers with fixed amount of heap size memory 
> (e.g., -Xmx=768M). 
> Therefore, we need a whole set of schemes that allow nimbus to put different 
> types of resources to each worker slot, then push the resources with 
> assignment to ZooKeepers; also, at the supervisor side, such resources in 
> each worker slot should be used by supervisor for launching a worker's JVM 
> (initially, the JVM heap size).
> This scheme can be used not only by RAS scheduler (STORM-893), but also by 
> any customized scheduler for conducting mem/cpu/network resource 
> specified-scheduling.
> In the future, the resources of memory, CPU and network can also be used by 
> supervisor to launch a worker in a resource-segregated container, such as a 
> CGroup or Docker, with isolated Memory/CPU/Network resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1093) Launching Workers with resources specified in resource-aware schedulers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950674#comment-14950674
 ] 

ASF GitHub Bot commented on STORM-1093:
---

Github user zhuoliu commented on the pull request:

https://github.com/apache/storm/pull/790#issuecomment-146924208
  
Thanks Jerry, nicely STORM-893 (RAS) has just been merged.


> Launching Workers with resources specified in resource-aware schedulers
> ---
>
> Key: STORM-1093
> URL: https://issues.apache.org/jira/browse/STORM-1093
> Project: Apache Storm
>  Issue Type: Story
>  Components: storm-core
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>
> Currently, we have Resource Aware Scheduler (STORM-894) in nimbus, which can 
> allocate different types of resource (CPU, onheap-memory, offheap-memory) to 
> the workers assigned to each topology's tasks.
> However, such resources are not visible to the supervisor, therefore, 
> supervisor will still launch workers with fixed amount of heap size memory 
> (e.g., -Xmx=768M). 
> Therefore, we need a whole set of schemes that allow nimbus to put different 
> types of resources to each worker slot, then push the resources with 
> assignment to ZooKeepers; also, at the supervisor side, such resources in 
> each worker slot should be used by supervisor for launching a worker's JVM 
> (initially, the JVM heap size).
> This scheme can be used not only by RAS scheduler (STORM-893), but also by 
> any customized scheduler for conducting mem/cpu/network resource 
> specified-scheduling.
> In the future, the resources of memory, CPU and network can also be used by 
> supervisor to launch a worker in a resource-segregated container, such as a 
> CGroup or Docker, with isolated Memory/CPU/Network resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1093] Launching Workers with resources ...

[zhuoliu]

Github user zhuoliu commented on the pull request:

https://github.com/apache/storm/pull/790#issuecomment-146924208
  
Thanks Jerry, nicely STORM-893 (RAS) has just been merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1084) Improve Storm config validation process to use java annotations instead of *_SCHEMA format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950681#comment-14950681
 ] 

ASF GitHub Bot commented on STORM-1084:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/785#issuecomment-146926754
  
@jerrypeng Perhaps we can do a hybrid approach, where we have validation 
for complex types made up of something that is a bit ugly, but expressive.  And 
a set of simple annotations for common types. for example

```
@Validate(type=Map.class, nullAllowed=false,
key=@Validate(type=String.class),
value=@Validate(type=Number.class, min=0, max=500))
public static final String MY_MAP_CONF="my.map.conf";

@Validate(type=List.class, value=@Validate(type=String.class))
public static final String MY_LIST_OF_STRINGS="my.list.of.strings";

@IntValidator(min=0, nullAllowed=false)
public static final String MY_INT_CONF="my.int.conf";

//And for the really strange types
@Validate(type=List.class, value=@Validate(type=Map.class, 
validator=RegistryValidator.class))
public static final String TOPOLOGY_METRICS_CONSUMER_REGISTER = 
"topology.metrics.consumer.register";
```


> Improve Storm config validation process to use java annotations instead of 
> *_SCHEMA format
> --
>
> Key: STORM-1084
> URL: https://issues.apache.org/jira/browse/STORM-1084
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>
> So currently we specify validators:
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
>  public static final Object STORM_MESSAGING_NETTY_MIN_SLEEP_MS_SCHEMA = 
> ConfigValidation.IntegerValidator;
> A better way to do this is using annotations.  Something like:
> @IntegerValidator
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
> Do this has many advantages. For one you can stack multiple annotations:
> @IntegerValidator
> @NotNull
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
> And we don't have to write another validator for strings that cannot be null
> And we can pass parameters into the annotations: 
> @PositiveIntegerValidator(notNull=true)
>   public static final String DRPC_REQUEST_TIMEOUT_SECS  = 
> "drpc.request.timeout.secs";
> instead of having to write another validator: 
> ConfigValidation.NotNullPosIntegerValidator for checking for not null



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1084] - Improve Storm config validation...

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/785#issuecomment-146926754
  
@jerrypeng Perhaps we can do a hybrid approach, where we have validation 
for complex types made up of something that is a bit ugly, but expressive.  And 
a set of simple annotations for common types. for example

```
@Validate(type=Map.class, nullAllowed=false,
key=@Validate(type=String.class),
value=@Validate(type=Number.class, min=0, max=500))
public static final String MY_MAP_CONF="my.map.conf";

@Validate(type=List.class, value=@Validate(type=String.class))
public static final String MY_LIST_OF_STRINGS="my.list.of.strings";

@IntValidator(min=0, nullAllowed=false)
public static final String MY_INT_CONF="my.int.conf";

//And for the really strange types
@Validate(type=List.class, value=@Validate(type=Map.class, 
validator=RegistryValidator.class))
public static final String TOPOLOGY_METRICS_CONSUMER_REGISTER = 
"topology.metrics.consumer.register";
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1095] Tuple.getSourceGlobalStreamid() h...

[mjsax]

Github user mjsax commented on the pull request:

https://github.com/apache/storm/pull/786#issuecomment-146839646
  
Shall I add failing builds if I encounter them?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (STORM-1101) test-retry-read-assignments in backtype.storm.supervisor-test fails

[Matthias J. Sax (JIRA)]

Matthias J. Sax created STORM-1101:
--

 Summary: test-retry-read-assignments in 
backtype.storm.supervisor-test fails
 Key: STORM-1101
 URL: https://issues.apache.org/jira/browse/STORM-1101
 Project: Apache Storm
  Issue Type: Sub-task
  Components: storm-core
Reporter: Matthias J. Sax


https://travis-ci.org/mjsax/storm/builds/84478972

{noformat}
ava.lang.RuntimeException: Should not have multiple topologies assigned to one 
port
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method) ~[?:1.7.0_76]
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
 ~[?:1.7.0_76]
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 ~[?:1.7.0_76]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 
~[?:1.7.0_76]
at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) 
~[clojure-1.7.0.jar:?]
at backtype.storm.util$throw_runtime.doInvoke(util.clj:845) 
~[classes/:?]
at clojure.lang.RestFn.invoke(RestFn.java:408) ~[clojure-1.7.0.jar:?]
at 
backtype.storm.daemon.supervisor$read_assignments$fn__9770.doInvoke(supervisor.clj:84)
 ~[classes/:?]
at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]
at clojure.core$merge_with$merge_entry__4649.invoke(core.clj:2932) 
~[clojure-1.7.0.jar:?]
at clojure.core$reduce1.invoke(core.clj:909) ~[clojure-1.7.0.jar:?]
at clojure.core$merge_with$merge2__4651.invoke(core.clj:2935) 
~[clojure-1.7.0.jar:?]
at clojure.core$reduce1.invoke(core.clj:909) ~[clojure-1.7.0.jar:?]
at clojure.core$reduce1.invoke(core.clj:900) ~[clojure-1.7.0.jar:?]
at clojure.core$merge_with.doInvoke(core.clj:2936) 
~[clojure-1.7.0.jar:?]
at clojure.lang.RestFn.applyTo(RestFn.java:139) ~[clojure-1.7.0.jar:?]
at clojure.core$apply.invoke(core.clj:632) ~[clojure-1.7.0.jar:?]
at 
backtype.storm.daemon.supervisor$read_assignments.invoke(supervisor.clj:84) 
~[classes/:?]
at 
backtype.storm.daemon.supervisor$read_assignments.invoke(supervisor.clj:86) 
~[classes/:?]
at 
backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__10013.invoke(supervisor.clj:449)
 ~[classes/:?]
at backtype.storm.event$event_manager$fn__9629.invoke(event.clj:40) 
[classes/:?]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1095) Tuple.getSourceGlobalStreamid() has wrong camel-case naming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950282#comment-14950282
 ] 

ASF GitHub Bot commented on STORM-1095:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/786


> Tuple.getSourceGlobalStreamid() has wrong camel-case naming
> ---
>
> Key: STORM-1095
> URL: https://issues.apache.org/jira/browse/STORM-1095
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Trivial
>
> The method Tuple.getSourceGlobalStreamid() should be named 
> Tuple.getSourceGlobalStreamId() to follow camel-case naming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1095] Tuple.getSourceGlobalStreamid() h...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/786


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1095] Tuple.getSourceGlobalStreamid() h...

[mjsax]

Github user mjsax commented on a diff in the pull request:

https://github.com/apache/storm/pull/786#discussion_r41617981
  
--- Diff: storm-core/src/jvm/backtype/storm/tuple/Tuple.java ---
@@ -37,9 +37,15 @@
 /**
  * Returns the global stream id (component + stream) of this tuple.
--- End diff --

Done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1095) Tuple.getSourceGlobalStreamid() has wrong camel-case naming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950190#comment-14950190
 ] 

ASF GitHub Bot commented on STORM-1095:
---

Github user mjsax commented on a diff in the pull request:

https://github.com/apache/storm/pull/786#discussion_r41617981
  
--- Diff: storm-core/src/jvm/backtype/storm/tuple/Tuple.java ---
@@ -37,9 +37,15 @@
 /**
  * Returns the global stream id (component + stream) of this tuple.
--- End diff --

Done.


> Tuple.getSourceGlobalStreamid() has wrong camel-case naming
> ---
>
> Key: STORM-1095
> URL: https://issues.apache.org/jira/browse/STORM-1095
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Trivial
>
> The method Tuple.getSourceGlobalStreamid() should be named 
> Tuple.getSourceGlobalStreamId() to follow camel-case naming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1095) Tuple.getSourceGlobalStreamid() has wrong camel-case naming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950254#comment-14950254
 ] 

ASF GitHub Bot commented on STORM-1095:
---

Github user mjsax commented on the pull request:

https://github.com/apache/storm/pull/786#issuecomment-146839646
  
Shall I add failing builds if I encounter them?


> Tuple.getSourceGlobalStreamid() has wrong camel-case naming
> ---
>
> Key: STORM-1095
> URL: https://issues.apache.org/jira/browse/STORM-1095
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Trivial
>
> The method Tuple.getSourceGlobalStreamid() should be named 
> Tuple.getSourceGlobalStreamId() to follow camel-case naming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1095) Tuple.getSourceGlobalStreamid() has wrong camel-case naming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950260#comment-14950260
 ] 

ASF GitHub Bot commented on STORM-1095:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/786#issuecomment-146840280
  
@mjsax Yeah, it would be great. :)


> Tuple.getSourceGlobalStreamid() has wrong camel-case naming
> ---
>
> Key: STORM-1095
> URL: https://issues.apache.org/jira/browse/STORM-1095
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Trivial
>
> The method Tuple.getSourceGlobalStreamid() should be named 
> Tuple.getSourceGlobalStreamId() to follow camel-case naming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1095] Tuple.getSourceGlobalStreamid() h...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/786#issuecomment-146840280
  
@mjsax Yeah, it would be great. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-1095) Tuple.getSourceGlobalStreamid() has wrong camel-case naming

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1095.
-
   Resolution: Fixed
Fix Version/s: 0.11.0

Thanks [~mjsax],
I merged into master.

> Tuple.getSourceGlobalStreamid() has wrong camel-case naming
> ---
>
> Key: STORM-1095
> URL: https://issues.apache.org/jira/browse/STORM-1095
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Trivial
> Fix For: 0.11.0
>
>
> The method Tuple.getSourceGlobalStreamid() should be named 
> Tuple.getSourceGlobalStreamId() to follow camel-case naming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1095] Tuple.getSourceGlobalStreamid() h...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/786#issuecomment-146835431
  
Thanks for following up, +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1095) Tuple.getSourceGlobalStreamid() has wrong camel-case naming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950244#comment-14950244
 ] 

ASF GitHub Bot commented on STORM-1095:
---

GitHub user mjsax reopened a pull request:

https://github.com/apache/storm/pull/786

[STORM-1095] Tuple.getSourceGlobalStreamid() has wrong camel-case naming



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/storm storm-1095

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/786.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #786


commit 68d81ab092e76e790da4c69fc9215fd1f2633821
Author: mjsax 
Date:   2015-10-07T19:24:24Z

[STORM-1095] Tuple.getSourceGlobalStreamid() has wrong camel-case naming




> Tuple.getSourceGlobalStreamid() has wrong camel-case naming
> ---
>
> Key: STORM-1095
> URL: https://issues.apache.org/jira/browse/STORM-1095
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Trivial
>
> The method Tuple.getSourceGlobalStreamid() should be named 
> Tuple.getSourceGlobalStreamId() to follow camel-case naming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1095] Tuple.getSourceGlobalStreamid() h...

[mjsax]

Github user mjsax closed the pull request at:

https://github.com/apache/storm/pull/786


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1095) Tuple.getSourceGlobalStreamid() has wrong camel-case naming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950242#comment-14950242
 ] 

ASF GitHub Bot commented on STORM-1095:
---

Github user mjsax commented on the pull request:

https://github.com/apache/storm/pull/786#issuecomment-146838634
  
One side question: I linked my own GitHub repo to Travis. However, the 
build fails very often (even if everything is ok). The Travis build of the 
Storm repo seems to be much more stable. 

For example this build is green, but my personal build of this branch 
failed: https://travis-ci.org/mjsax/storm/builds/84478972

Do you have any idea why? It is somewhat annoying that I cannot rely on my 
personal Travis builds.


> Tuple.getSourceGlobalStreamid() has wrong camel-case naming
> ---
>
> Key: STORM-1095
> URL: https://issues.apache.org/jira/browse/STORM-1095
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Trivial
>
> The method Tuple.getSourceGlobalStreamid() should be named 
> Tuple.getSourceGlobalStreamId() to follow camel-case naming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1095] Tuple.getSourceGlobalStreamid() h...

[mjsax]

Github user mjsax commented on the pull request:

https://github.com/apache/storm/pull/786#issuecomment-146838634
  
One side question: I linked my own GitHub repo to Travis. However, the 
build fails very often (even if everything is ok). The Travis build of the 
Storm repo seems to be much more stable. 

For example this build is green, but my personal build of this branch 
failed: https://travis-ci.org/mjsax/storm/builds/84478972

Do you have any idea why? It is somewhat annoying that I cannot rely on my 
personal Travis builds.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1095) Tuple.getSourceGlobalStreamid() has wrong camel-case naming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950243#comment-14950243
 ] 

ASF GitHub Bot commented on STORM-1095:
---

Github user mjsax closed the pull request at:

https://github.com/apache/storm/pull/786


> Tuple.getSourceGlobalStreamid() has wrong camel-case naming
> ---
>
> Key: STORM-1095
> URL: https://issues.apache.org/jira/browse/STORM-1095
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Trivial
>
> The method Tuple.getSourceGlobalStreamid() should be named 
> Tuple.getSourceGlobalStreamId() to follow camel-case naming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1095] Tuple.getSourceGlobalStreamid() h...

[mjsax]

GitHub user mjsax reopened a pull request:

https://github.com/apache/storm/pull/786

[STORM-1095] Tuple.getSourceGlobalStreamid() has wrong camel-case naming



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/storm storm-1095

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/786.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #786


commit 68d81ab092e76e790da4c69fc9215fd1f2633821
Author: mjsax 
Date:   2015-10-07T19:24:24Z

[STORM-1095] Tuple.getSourceGlobalStreamid() has wrong camel-case naming




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-893) Resource Aware Scheduling

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950434#comment-14950434
 ] 

ASF GitHub Bot commented on STORM-893:
--

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/746#issuecomment-146880374
  
@jerrypeng Technically with one +1 from a committer and 24 hours we can 
merge this in.  But this is a large change so I am glad to see more people 
looking at it.  I'll try to go though it again and hopefully merge it in 
shortly.  


> Resource Aware Scheduling
> -
>
> Key: STORM-893
> URL: https://issues.apache.org/jira/browse/STORM-893
> Project: Apache Storm
>  Issue Type: Umbrella
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: resource_aware_scheduler_api.pdf
>
>
> At Yahoo we have been working on resource aware scheduling in storm, based 
> off of some work done in academia.  This rollup ticket is to track the 
> complete project.  With several sub tasks.  Some that are already done and 
> need to be pushed back, and others that we have not started on yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-893] Resource Aware Scheduling

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/746#issuecomment-146880374
  
@jerrypeng Technically with one +1 from a committer and 24 hours we can 
merge this in.  But this is a large change so I am glad to see more people 
looking at it.  I'll try to go though it again and hopefully merge it in 
shortly.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-893) Resource Aware Scheduling

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950436#comment-14950436
 ] 

ASF GitHub Bot commented on STORM-893:
--

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/746#issuecomment-146880546
  
@revans2 Thank you!


> Resource Aware Scheduling
> -
>
> Key: STORM-893
> URL: https://issues.apache.org/jira/browse/STORM-893
> Project: Apache Storm
>  Issue Type: Umbrella
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: resource_aware_scheduler_api.pdf
>
>
> At Yahoo we have been working on resource aware scheduling in storm, based 
> off of some work done in academia.  This rollup ticket is to track the 
> complete project.  With several sub tasks.  Some that are already done and 
> need to be pushed back, and others that we have not started on yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1096) UI tries to impersonate wrong user when getting topology conf for authorization, impersonation is allowed by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950441#comment-14950441
 ] 

ASF GitHub Bot commented on STORM-1096:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/788


> UI tries to impersonate wrong user when getting topology conf for 
> authorization, impersonation is allowed by default
> 
>
> Key: STORM-1096
> URL: https://issues.apache.org/jira/browse/STORM-1096
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.10.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> We have started using 0.10.0 under load and found a few issues around the UI 
> and impersonation.
> The UI when trying to connect to nimbus will impersonate other users.  
> Nimbus, by default allows impersonation and just outputs a warning message 
> that it is allowed.  We really should default to not allowing impersonation.  
> having the authorizer configured by default does not hurt when running 
> insecure because impersonation is not possible, but when security is enabled 
> if someone forgets to set this config we are now insecure by default.
> If you do set all of that up correctly the UI now can impersonate the wrong 
> user when connecting to nimbus.
> The UI decides which user to impersonate by pulling it from the request 
> context.  The requestContext is populated from the HttpRequest when 
> assert-authorized-user is called.  assert-authorized-user takes a 
> topology-conf as a parameter.  The only way to get this topology conf is to 
> talk to nimbus, which will get the wrong user because the request context has 
> not been populated yet.
> This just because a huge pain for users who way too often will not be able to 
> see pages on the UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1096) UI tries to impersonate wrong user when getting topology conf for authorization, impersonation is allowed by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950462#comment-14950462
 ] 

ASF GitHub Bot commented on STORM-1096:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/787


> UI tries to impersonate wrong user when getting topology conf for 
> authorization, impersonation is allowed by default
> 
>
> Key: STORM-1096
> URL: https://issues.apache.org/jira/browse/STORM-1096
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.10.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> We have started using 0.10.0 under load and found a few issues around the UI 
> and impersonation.
> The UI when trying to connect to nimbus will impersonate other users.  
> Nimbus, by default allows impersonation and just outputs a warning message 
> that it is allowed.  We really should default to not allowing impersonation.  
> having the authorizer configured by default does not hurt when running 
> insecure because impersonation is not possible, but when security is enabled 
> if someone forgets to set this config we are now insecure by default.
> If you do set all of that up correctly the UI now can impersonate the wrong 
> user when connecting to nimbus.
> The UI decides which user to impersonate by pulling it from the request 
> context.  The requestContext is populated from the HttpRequest when 
> assert-authorized-user is called.  assert-authorized-user takes a 
> topology-conf as a parameter.  The only way to get this topology conf is to 
> talk to nimbus, which will get the wrong user because the request context has 
> not been populated yet.
> This just because a huge pain for users who way too often will not be able to 
> see pages on the UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1096: Fix some issues with impersonation...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/787


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1095] Tuple.getSourceGlobalStreamid() h...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/786#issuecomment-146839430
  
@mjsax 
Storm has some kinds of random failures now.
I've been filing failures on https://issues.apache.org/jira/browse/STORM-915


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1095) Tuple.getSourceGlobalStreamid() has wrong camel-case naming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950251#comment-14950251
 ] 

ASF GitHub Bot commented on STORM-1095:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/786#issuecomment-146839430
  
@mjsax 
Storm has some kinds of random failures now.
I've been filing failures on https://issues.apache.org/jira/browse/STORM-915


> Tuple.getSourceGlobalStreamid() has wrong camel-case naming
> ---
>
> Key: STORM-1095
> URL: https://issues.apache.org/jira/browse/STORM-1095
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Trivial
>
> The method Tuple.getSourceGlobalStreamid() should be named 
> Tuple.getSourceGlobalStreamId() to follow camel-case naming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-893] Resource Aware Scheduling

[jerrypeng]

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/746#issuecomment-146880546
  
@revans2 Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1096: Fix some issues with impersonation...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/788


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-1096) UI tries to impersonate wrong user when getting topology conf for authorization, impersonation is allowed by default

[Robert Joseph Evans (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans resolved STORM-1096.

   Resolution: Fixed
Fix Version/s: 0.10.0

Thanks for the reviews I merged this into master and 0.10.x-branch

> UI tries to impersonate wrong user when getting topology conf for 
> authorization, impersonation is allowed by default
> 
>
> Key: STORM-1096
> URL: https://issues.apache.org/jira/browse/STORM-1096
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.10.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.10.0
>
>
> We have started using 0.10.0 under load and found a few issues around the UI 
> and impersonation.
> The UI when trying to connect to nimbus will impersonate other users.  
> Nimbus, by default allows impersonation and just outputs a warning message 
> that it is allowed.  We really should default to not allowing impersonation.  
> having the authorizer configured by default does not hurt when running 
> insecure because impersonation is not possible, but when security is enabled 
> if someone forgets to set this config we are now insecure by default.
> If you do set all of that up correctly the UI now can impersonate the wrong 
> user when connecting to nimbus.
> The UI decides which user to impersonate by pulling it from the request 
> context.  The requestContext is populated from the HttpRequest when 
> assert-authorized-user is called.  assert-authorized-user takes a 
> topology-conf as a parameter.  The only way to get this topology conf is to 
> talk to nimbus, which will get the wrong user because the request context has 
> not been populated yet.
> This just because a huge pain for users who way too often will not be able to 
> see pages on the UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1099] Fix worker childopts as arraylist...

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/791#issuecomment-146890241
  
@ppoulosk your new test seems to be failing with an arity issue.  Please 
take a look at it.

```
classname: backtype.storm.supervisor-test / testname: 
test-substitute-childopts-happy-path-arraylist
Uncaught exception, not in assertion.
expected: nil
  actual: clojure.lang.ArityException: Wrong number of args (5) passed to: 
supervisor/substitute-childopts
 at clojure.lang.AFn.throwArity (AFn.java:429)
clojure.lang.AFn.invoke (AFn.java:48)
backtype.storm.supervisor_test/fn (supervisor_test.clj:584)
clojure.test$test_var$fn__7670.invoke (test.clj:704)
clojure.test$test_var.invoke (test.clj:704)
clojure.test$test_vars$fn__7692$fn__7697.invoke (test.clj:722)
clojure.test$default_fixture.invoke (test.clj:674)
clojure.test$test_vars$fn__7692.invoke (test.clj:722)
clojure.test$default_fixture.invoke (test.clj:674)
clojure.test$test_vars.invoke (test.clj:718)
clojure.test$test_all_vars.invoke (test.clj:728)
clojure.test$test_ns.invoke (test.clj:747)
clojure.core$map$fn__4553.invoke (core.clj:2624)
clojure.lang.LazySeq.sval (LazySeq.java:40)
clojure.lang.LazySeq.seq (LazySeq.java:49)
clojure.lang.Cons.next (Cons.java:39)
clojure.lang.RT.boundedLength (RT.java:1735)
clojure.lang.RestFn.applyTo (RestFn.java:130)
clojure.core$apply.invoke (core.clj:632)
clojure.test$run_tests.doInvoke (test.clj:762)
clojure.lang.RestFn.invoke (RestFn.java:408)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1099) worker childopts cannot accept arraylist of strings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950482#comment-14950482
 ] 

ASF GitHub Bot commented on STORM-1099:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/791#issuecomment-146890241
  
@ppoulosk your new test seems to be failing with an arity issue.  Please 
take a look at it.

```
classname: backtype.storm.supervisor-test / testname: 
test-substitute-childopts-happy-path-arraylist
Uncaught exception, not in assertion.
expected: nil
  actual: clojure.lang.ArityException: Wrong number of args (5) passed to: 
supervisor/substitute-childopts
 at clojure.lang.AFn.throwArity (AFn.java:429)
clojure.lang.AFn.invoke (AFn.java:48)
backtype.storm.supervisor_test/fn (supervisor_test.clj:584)
clojure.test$test_var$fn__7670.invoke (test.clj:704)
clojure.test$test_var.invoke (test.clj:704)
clojure.test$test_vars$fn__7692$fn__7697.invoke (test.clj:722)
clojure.test$default_fixture.invoke (test.clj:674)
clojure.test$test_vars$fn__7692.invoke (test.clj:722)
clojure.test$default_fixture.invoke (test.clj:674)
clojure.test$test_vars.invoke (test.clj:718)
clojure.test$test_all_vars.invoke (test.clj:728)
clojure.test$test_ns.invoke (test.clj:747)
clojure.core$map$fn__4553.invoke (core.clj:2624)
clojure.lang.LazySeq.sval (LazySeq.java:40)
clojure.lang.LazySeq.seq (LazySeq.java:49)
clojure.lang.Cons.next (Cons.java:39)
clojure.lang.RT.boundedLength (RT.java:1735)
clojure.lang.RestFn.applyTo (RestFn.java:130)
clojure.core$apply.invoke (core.clj:632)
clojure.test$run_tests.doInvoke (test.clj:762)
clojure.lang.RestFn.invoke (RestFn.java:408)
```


> worker childopts cannot accept arraylist of strings
> ---
>
> Key: STORM-1099
> URL: https://issues.apache.org/jira/browse/STORM-1099
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> This used to work on 0.9, but is broken in 0.10.
> If a customer attempts to add a array of strings as a config for 
> topology.worker.childopts, like the following.
> {noformat}
>   config.put("topology.worker.childopts", Arrays.asList("-Xmx1g", 
> "-Dmy.boxus.parameter=mybogusvalue"));
> {noformat}
> The topology will not launch because "[-Xmx1g" will be an argument to the jvm 
> when the supervisor launches it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-893) Resource Aware Scheduling

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950405#comment-14950405
 ] 

ASF GitHub Bot commented on STORM-893:
--

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/746#issuecomment-146875059
  
I have two +1s now, can we merge?


> Resource Aware Scheduling
> -
>
> Key: STORM-893
> URL: https://issues.apache.org/jira/browse/STORM-893
> Project: Apache Storm
>  Issue Type: Umbrella
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: resource_aware_scheduler_api.pdf
>
>
> At Yahoo we have been working on resource aware scheduling in storm, based 
> off of some work done in academia.  This rollup ticket is to track the 
> complete project.  With several sub tasks.  Some that are already done and 
> need to be pushed back, and others that we have not started on yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-893] Resource Aware Scheduling

[jerrypeng]

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/746#issuecomment-146875059
  
I have two +1s now, can we merge?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-146874511
  
@rsltrifork That does sound like a very interesting approach.  I would love 
to see some numbers on how it would perform.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-886) Automatic Back Pressure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950400#comment-14950400
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-146874511
  
@rsltrifork That does sound like a very interesting approach.  I would love 
to see some numbers on how it would perform.


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Fix For: 0.11.0
>
> Attachments: aSimpleExampleOfBackpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1084) Improve Storm config validation process to use java annotations instead of *_SCHEMA format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950871#comment-14950871
 ] 

ASF GitHub Bot commented on STORM-1084:
---

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/785#issuecomment-146948299
  
@revans2 I don't think you solution will work either.  I don't think we can 
get recursion and java annotations to work,  since fields in annotations cannot 
default to a specific non-null type.  To have annotations that contain fields 
of it self will result in a unending recursive definition:

ComplexValidator key() default  
@ComplexValidator(type=Object.class, key=@ComplexValidator(type=Object.class, 
key=@ComplexValidator(...), value=ComplexValidator(ype=Object.class, 
key=@ComplexValidator(...));



> Improve Storm config validation process to use java annotations instead of 
> *_SCHEMA format
> --
>
> Key: STORM-1084
> URL: https://issues.apache.org/jira/browse/STORM-1084
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>
> So currently we specify validators:
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
>  public static final Object STORM_MESSAGING_NETTY_MIN_SLEEP_MS_SCHEMA = 
> ConfigValidation.IntegerValidator;
> A better way to do this is using annotations.  Something like:
> @IntegerValidator
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
> Do this has many advantages. For one you can stack multiple annotations:
> @IntegerValidator
> @NotNull
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
> And we don't have to write another validator for strings that cannot be null
> And we can pass parameters into the annotations: 
> @PositiveIntegerValidator(notNull=true)
>   public static final String DRPC_REQUEST_TIMEOUT_SECS  = 
> "drpc.request.timeout.secs";
> instead of having to write another validator: 
> ConfigValidation.NotNullPosIntegerValidator for checking for not null



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1084) Improve Storm config validation process to use java annotations instead of *_SCHEMA format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950883#comment-14950883
 ] 

ASF GitHub Bot commented on STORM-1084:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/785#issuecomment-146949145
  
Why can't the default be null?


> Improve Storm config validation process to use java annotations instead of 
> *_SCHEMA format
> --
>
> Key: STORM-1084
> URL: https://issues.apache.org/jira/browse/STORM-1084
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>
> So currently we specify validators:
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
>  public static final Object STORM_MESSAGING_NETTY_MIN_SLEEP_MS_SCHEMA = 
> ConfigValidation.IntegerValidator;
> A better way to do this is using annotations.  Something like:
> @IntegerValidator
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
> Do this has many advantages. For one you can stack multiple annotations:
> @IntegerValidator
> @NotNull
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
> And we don't have to write another validator for strings that cannot be null
> And we can pass parameters into the annotations: 
> @PositiveIntegerValidator(notNull=true)
>   public static final String DRPC_REQUEST_TIMEOUT_SECS  = 
> "drpc.request.timeout.secs";
> instead of having to write another validator: 
> ConfigValidation.NotNullPosIntegerValidator for checking for not null



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1084] - Improve Storm config validation...

[jerrypeng]

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/785#issuecomment-146949595
  
the java annotation spec was written that way


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (STORM-1102) HiveBolt should have a default flush interval

[Aaron Dossett (JIRA)]

Aaron Dossett created STORM-1102:


 Summary: HiveBolt should have a default flush interval
 Key: STORM-1102
 URL: https://issues.apache.org/jira/browse/STORM-1102
 Project: Apache Storm
  Issue Type: Improvement
  Components: storm-hive
Reporter: Aaron Dossett
Assignee: Aaron Dossett
Priority: Minor


HiveBolt uses tick tuples to periodically flush data.  Currently this only 
happens if a flush interval is defined.  HiveBolt should have a default flush 
interval the same way HdfsBolt does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1102: Add a default flush interval for H...

[dossett]

GitHub user dossett opened a pull request:

https://github.com/apache/storm/pull/792

STORM-1102: Add a default flush interval for HiveBolt



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dossett/storm STORM-1102

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/792.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #792


commit 9a25aede1791bba48dbb345fb2523e1484b4db8c
Author: Aaron Dossett 
Date:   2015-10-09T18:27:03Z

STORM-1102: Add a default flush interval for HiveBolt




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1102) HiveBolt should have a default flush interval

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950940#comment-14950940
 ] 

ASF GitHub Bot commented on STORM-1102:
---

GitHub user dossett opened a pull request:

https://github.com/apache/storm/pull/792

STORM-1102: Add a default flush interval for HiveBolt



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dossett/storm STORM-1102

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/792.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #792


commit 9a25aede1791bba48dbb345fb2523e1484b4db8c
Author: Aaron Dossett 
Date:   2015-10-09T18:27:03Z

STORM-1102: Add a default flush interval for HiveBolt




> HiveBolt should have a default flush interval
> -
>
> Key: STORM-1102
> URL: https://issues.apache.org/jira/browse/STORM-1102
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hive
>Reporter: Aaron Dossett
>Assignee: Aaron Dossett
>Priority: Minor
>
> HiveBolt uses tick tuples to periodically flush data.  Currently this only 
> happens if a flush interval is defined.  HiveBolt should have a default flush 
> interval the same way HdfsBolt does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1103) Utils#getConfigFileInputStream can print INFO level logs each time it is called

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950962#comment-14950962
 ] 

ASF GitHub Bot commented on STORM-1103:
---

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/793

[STORM-1103] changes log message to DEBUG from INFO



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm storm-1103-utils-noisy-log

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/793.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #793


commit c917679b6d2c8a4c4609878158c2d6f4debb0ead
Author: Derek Dagit 
Date:   2015-10-09T18:42:02Z

changes log message to DEBUG from INFO




> Utils#getConfigFileInputStream can print INFO level logs each time it is 
> called
> ---
>
> Key: STORM-1103
> URL: https://issues.apache.org/jira/browse/STORM-1103
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> https://github.com/apache/storm/blob/86f2d03c2060f25ccdaf7580cfee5d9f1a9ac2e3/storm-core/src/jvm/backtype/storm/utils/Utils.java#L253
> This can result in fairly noisy logs.  It would be good to move this to DEBUG 
> level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1103] changes log message to DEBUG from...

[d2r]

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/793

[STORM-1103] changes log message to DEBUG from INFO



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm storm-1103-utils-noisy-log

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/793.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #793


commit c917679b6d2c8a4c4609878158c2d6f4debb0ead
Author: Derek Dagit 
Date:   2015-10-09T18:42:02Z

changes log message to DEBUG from INFO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1084] - Improve Storm config validation...

[jerrypeng]

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/785#issuecomment-146948299
  
@revans2 I don't think you solution will work either.  I don't think we can 
get recursion and java annotations to work,  since fields in annotations cannot 
default to a specific non-null type.  To have annotations that contain fields 
of it self will result in a unending recursive definition:

ComplexValidator key() default  
@ComplexValidator(type=Object.class, key=@ComplexValidator(type=Object.class, 
key=@ComplexValidator(...), value=ComplexValidator(ype=Object.class, 
key=@ComplexValidator(...));



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1084] - Improve Storm config validation...

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/785#issuecomment-146949145
  
Why can't the default be null?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1084) Improve Storm config validation process to use java annotations instead of *_SCHEMA format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950888#comment-14950888
 ] 

ASF GitHub Bot commented on STORM-1084:
---

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/785#issuecomment-146949595
  
the java annotation spec was written that way


> Improve Storm config validation process to use java annotations instead of 
> *_SCHEMA format
> --
>
> Key: STORM-1084
> URL: https://issues.apache.org/jira/browse/STORM-1084
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>
> So currently we specify validators:
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
>  public static final Object STORM_MESSAGING_NETTY_MIN_SLEEP_MS_SCHEMA = 
> ConfigValidation.IntegerValidator;
> A better way to do this is using annotations.  Something like:
> @IntegerValidator
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
> Do this has many advantages. For one you can stack multiple annotations:
> @IntegerValidator
> @NotNull
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
> And we don't have to write another validator for strings that cannot be null
> And we can pass parameters into the annotations: 
> @PositiveIntegerValidator(notNull=true)
>   public static final String DRPC_REQUEST_TIMEOUT_SECS  = 
> "drpc.request.timeout.secs";
> instead of having to write another validator: 
> ConfigValidation.NotNullPosIntegerValidator for checking for not null



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1084] - Improve Storm config validation...

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/785#issuecomment-146953738
  
Crap OK.  The JSR has thwarted my plan to take over the universe.  What we 
have now looks like it is as good as it gets.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1084) Improve Storm config validation process to use java annotations instead of *_SCHEMA format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950923#comment-14950923
 ] 

ASF GitHub Bot commented on STORM-1084:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/785#issuecomment-146953738
  
Crap OK.  The JSR has thwarted my plan to take over the universe.  What we 
have now looks like it is as good as it gets.


> Improve Storm config validation process to use java annotations instead of 
> *_SCHEMA format
> --
>
> Key: STORM-1084
> URL: https://issues.apache.org/jira/browse/STORM-1084
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>
> So currently we specify validators:
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
>  public static final Object STORM_MESSAGING_NETTY_MIN_SLEEP_MS_SCHEMA = 
> ConfigValidation.IntegerValidator;
> A better way to do this is using annotations.  Something like:
> @IntegerValidator
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
> Do this has many advantages. For one you can stack multiple annotations:
> @IntegerValidator
> @NotNull
>  public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = 
> "storm.messaging.netty.min_wait_ms";
> And we don't have to write another validator for strings that cannot be null
> And we can pass parameters into the annotations: 
> @PositiveIntegerValidator(notNull=true)
>   public static final String DRPC_REQUEST_TIMEOUT_SECS  = 
> "drpc.request.timeout.secs";
> instead of having to write another validator: 
> ConfigValidation.NotNullPosIntegerValidator for checking for not null



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1103) Utils#getConfigFileInputStream can print INFO level logs each time it is called

[Derek Dagit (JIRA)]

Derek Dagit created STORM-1103:
--

 Summary: Utils#getConfigFileInputStream can print INFO level logs 
each time it is called
 Key: STORM-1103
 URL: https://issues.apache.org/jira/browse/STORM-1103
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Affects Versions: 0.11.0
Reporter: Derek Dagit
Assignee: Derek Dagit
Priority: Minor


https://github.com/apache/storm/blob/86f2d03c2060f25ccdaf7580cfee5d9f1a9ac2e3/storm-core/src/jvm/backtype/storm/utils/Utils.java#L253

This can result in fairly noisy logs.  It would be good to move this to DEBUG 
level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1103) Utils#getConfigFileInputStream can print INFO level logs each time it is called

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950980#comment-14950980
 ] 

ASF GitHub Bot commented on STORM-1103:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/793#issuecomment-146959657
  
+1


> Utils#getConfigFileInputStream can print INFO level logs each time it is 
> called
> ---
>
> Key: STORM-1103
> URL: https://issues.apache.org/jira/browse/STORM-1103
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> https://github.com/apache/storm/blob/86f2d03c2060f25ccdaf7580cfee5d9f1a9ac2e3/storm-core/src/jvm/backtype/storm/utils/Utils.java#L253
> This can result in fairly noisy logs.  It would be good to move this to DEBUG 
> level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1103] changes log message to DEBUG from...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/793#issuecomment-146959657
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1102: Add a default flush interval for H...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/792#issuecomment-146959789
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1102) HiveBolt should have a default flush interval

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950982#comment-14950982
 ] 

ASF GitHub Bot commented on STORM-1102:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/792#issuecomment-146959789
  
+1


> HiveBolt should have a default flush interval
> -
>
> Key: STORM-1102
> URL: https://issues.apache.org/jira/browse/STORM-1102
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hive
>Reporter: Aaron Dossett
>Assignee: Aaron Dossett
>Priority: Minor
>
> HiveBolt uses tick tuples to periodically flush data.  Currently this only 
> happens if a flush interval is defined.  HiveBolt should have a default flush 
> interval the same way HdfsBolt does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1093) Launching Workers with resources specified in resource-aware schedulers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951022#comment-14951022
 ] 

ASF GitHub Bot commented on STORM-1093:
---

Github user zhuoliu commented on the pull request:

https://github.com/apache/storm/pull/790#issuecomment-146962992
  
Just manual test with RAS, it works well, and we can see each worker is 
being launched with different JVM heap sizes (added up by their tasks' 
specified memory usage).
May I have your reviews when you have time @revans2  @d2r ?


> Launching Workers with resources specified in resource-aware schedulers
> ---
>
> Key: STORM-1093
> URL: https://issues.apache.org/jira/browse/STORM-1093
> Project: Apache Storm
>  Issue Type: Story
>  Components: storm-core
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>
> Currently, we have Resource Aware Scheduler (STORM-894) in nimbus, which can 
> allocate different types of resource (CPU, onheap-memory, offheap-memory) to 
> the workers assigned to each topology's tasks.
> However, such resources are not visible to the supervisor, therefore, 
> supervisor will still launch workers with fixed amount of heap size memory 
> (e.g., -Xmx=768M). 
> Therefore, we need a whole set of schemes that allow nimbus to put different 
> types of resources to each worker slot, then push the resources with 
> assignment to ZooKeepers; also, at the supervisor side, such resources in 
> each worker slot should be used by supervisor for launching a worker's JVM 
> (initially, the JVM heap size).
> This scheme can be used not only by RAS scheduler (STORM-893), but also by 
> any customized scheduler for conducting mem/cpu/network resource 
> specified-scheduling.
> In the future, the resources of memory, CPU and network can also be used by 
> supervisor to launch a worker in a resource-segregated container, such as a 
> CGroup or Docker, with isolated Memory/CPU/Network resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)