[jira] [Commented] (STORM-1760) HiveState should retire idle or old writes with flushAndClose

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270124#comment-15270124
 ] 

ASF GitHub Bot commented on STORM-1760:
---

Github user harshach closed the pull request at:

https://github.com/apache/storm/pull/1392


> HiveState should retire idle or old writes with flushAndClose
> -
>
> Key: STORM-1760
> URL: https://issues.apache.org/jira/browse/STORM-1760
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hive
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 1.0.0, 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1760. HiveState should retire idle or ol...

[harshach]

Github user harshach closed the pull request at:

https://github.com/apache/storm/pull/1392


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [storm-elasticsearch]Upgrade elasticsearch ver...

[sweetest]

Github user sweetest commented on the pull request:

https://github.com/apache/storm/pull/1396#issuecomment-216742378
  
I was also doing similar work, code change is almost the same. Looks good 
to me!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [storm-elasticsearch]Upgrade elasticsearch ver...

[kojiisd]

GitHub user kojiisd opened a pull request:

https://github.com/apache/storm/pull/1396

[storm-elasticsearch]Upgrade elasticsearch version from 1.6.0 to 2.3.2



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kojiisd/storm 1.x-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1396.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1396


commit 9f85579120167a9f9d93186ac50134ee3ee802be
Author: kojiisd 
Date:   2016-05-04T04:10:15Z

[storm-elasticsearch]Upgrade elasticsearch version from 1.6.0 to 2.3.2




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: storm-1726: use Put#addColumn to replace the d...

[lujinhong]

Github user lujinhong commented on the pull request:

https://github.com/apache/storm/pull/1388#issuecomment-216717573
  
sorry, please check https://github.com/apache/storm/pull/1395


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: storm-1726: use Put#addColumn to replace the d...

[lujinhong]

Github user lujinhong closed the pull request at:

https://github.com/apache/storm/pull/1388


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: storm-1726: use Put#addColumn to replace the d...

[lujinhong]

GitHub user lujinhong opened a pull request:

https://github.com/apache/storm/pull/1395

storm-1726: use Put#addColumn to replace the deprecated Put#add



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lujinhong/storm 1.x-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1395.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1395


commit de36288c53fba6d50ccbfc49a62e7c51eff974bf
Author: jinhong-lu 
Date:   2016-05-04T01:36:10Z

use Put#addColumn to replace the deprecated Put#add




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1761) Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster Mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269974#comment-15269974
 ] 

ASF GitHub Bot commented on STORM-1761:
---

Github user hustfxj commented on the pull request:

https://github.com/apache/storm/pull/1394#issuecomment-216716374
  
+1


> Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster 
> Mode
> ---
>
> Key: STORM-1761
> URL: https://issues.apache.org/jira/browse/STORM-1761
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Affects Versions: 1.0.1, 0.10.2
>Reporter: Hugo Louro
>Assignee: Hugo Louro
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1761: Storm-Solr Example Throws ArrayInd...

[hustfxj]

Github user hustfxj commented on the pull request:

https://github.com/apache/storm/pull/1394#issuecomment-216716374
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Release Apache Storm 0.10.1 (rc2)

[P. Taylor Goetz]

Harsha is correct.

The 0.10.x branch is in maintenance mode. So we shouldn't be adding to or 
modifying APIs. 

-Taylor


> On May 3, 2016, at 8:55 PM, Harsha  wrote:
> 
> Roshan,
>This is 0.10 maint release . We are not going to include the
>new features that went into 1.0.
> 
> +1 (binding)
> Deployed on a 3-node cluster
> Ran example topologies
> verified signatures of binaries.
> 
> Thanks,
> Harsha
> 
>> On Tue, May 3, 2016, at 05:16 PM, Roshan Naik wrote:
>> This is missing HdfsSpout   (STORM-1199)
>> 
>> -roshan
>> 
>> 
>>> On 5/3/16, 3:39 PM, "Hugo Da Cruz Louro"  wrote:
>>> 
>>> +1 (non binding)
>>> - Found a very minor bug in the storm-solr test topology, for which there
>>> is a very simple workaround and by NO means a blocker for the release.
>>> - unzipped .zip sources and binaries
>>> - mvn clean install in unzipped sources
>>> - created uber jar for storm-solr
>>> - tested storm-solr in local cluster mode and remode cluster mode
>>> - ran test examples
>>> 
>>> On May 3, 2016, at 12:04 AM, Arun Mahadevan
>>> > wrote:
>>> 
>>> +1 (binding)
>>> - Extracted binaries
>>> - Ran sample topologies
>>> - Browsed storm UI
>>> 
>>> Thanks,
>>> Arun
>>> 
>>> 
>>> 
>>> 
>>> On 4/28/16, 12:01 PM, "Jungtaek Lim"
>>> > wrote:
>>> 
>>> +1 (binding)
>>> 
>>> - testing with source distribution : OK
>>> - unzip : OK
>>> - building from source dist : OK
>>> - how to build: running `mvn -P all-tests clean install` on unzipped
>>> source dist.
>>> 
>>> - testing with binary distribution (one machine) : OK
>>> - launch daemons : OK
>>> - run RollingTopWords (local) : OK
>>> - run RollingTopWords (remote) : OK
>>> - activate / deactivate / rebalance / kill : OK
>>> 
>>> Thanks,
>>> Jungtaek Lim
>>> 
>>> 2016년 4월 28일 (목) 오전 3:29, P. Taylor Goetz
>>> >님이 작성:
>>> 
>>> This is a call to vote on releasing Apache Storm 0.10.1 (rc2)
>>> 
>>> Full list of changes in this release:
>>> 
>>> 
>>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGEL
>>> OG.md;hb=8179921a569b6cf1d97798eed8e7b03b131bc495
>>> 
>>> The tag/commit to be voted upon is v0.10.1:
>>> 
>>> 
>>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=ddf051149f3c3
>>> 86342937efdabdaf45694602dd1;hb=8179921a569b6cf1d97798eed8e7b03b131bc495;a=
>>> tree;h=26291835f22474506f0fe90b0459eab0d00bf4a9;hb=f0d3eae7395b3ee036b94b9
>>> 22707f74868ba6190
>>> 
>>> The source archive being voted upon can be found here:
>>> 
>>> 
>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc2/apach
>>> e-storm-0.10.1-src.tar.gz
>>> 
>>> Other release files, signatures and digests can be found here:
>>> 
>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc2/
>>> 
>>> The release artifacts are signed with the following key:
>>> 
>>> 
>>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb
>>> =22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>>> 
>>> The Nexus staging repository for this release is:
>>> 
>>> https://repository.apache.org/content/repositories/orgapachestorm-1031/
>>> 
>>> Please vote on releasing this package as Apache Storm 0.10.1.
>>> 
>>> When voting, please list the actions taken to verify the release.
>>> 
>>> This vote will be open for at least 72 hours.
>>> 
>>> [ ] +1 Release this package as Apache Storm 0.10.1
>>> [ ]  0 No opinion
>>> [ ] -1 Do not release this package because...
>>> 
>>> Thanks to everyone who contributed to this release.
>>> 
>>> -Taylor
>> 

Re: [VOTE] Release Apache Storm 0.10.1 (rc2)

[Harsha]

Roshan,
This is 0.10 maint release . We are not going to include the
new features that went into 1.0.

+1 (binding)
Deployed on a 3-node cluster
Ran example topologies
verified signatures of binaries.

Thanks,
Harsha

On Tue, May 3, 2016, at 05:16 PM, Roshan Naik wrote:
> This is missing HdfsSpout   (STORM-1199)
>   
> -roshan
> 
> 
> On 5/3/16, 3:39 PM, "Hugo Da Cruz Louro"  wrote:
> 
> >+1 (non binding)
> >- Found a very minor bug in the storm-solr test topology, for which there
> >is a very simple workaround and by NO means a blocker for the release.
> >- unzipped .zip sources and binaries
> >- mvn clean install in unzipped sources
> >- created uber jar for storm-solr
> >- tested storm-solr in local cluster mode and remode cluster mode
> >- ran test examples
> >
> >On May 3, 2016, at 12:04 AM, Arun Mahadevan
> >> wrote:
> >
> >+1 (binding)
> >- Extracted binaries
> >- Ran sample topologies
> >- Browsed storm UI
> >
> >Thanks,
> >Arun
> >
> >
> >
> >
> >On 4/28/16, 12:01 PM, "Jungtaek Lim"
> >> wrote:
> >
> >+1 (binding)
> >
> >- testing with source distribution : OK
> >- unzip : OK
> >- building from source dist : OK
> >  - how to build: running `mvn -P all-tests clean install` on unzipped
> >source dist.
> >
> >- testing with binary distribution (one machine) : OK
> >- launch daemons : OK
> >- run RollingTopWords (local) : OK
> >- run RollingTopWords (remote) : OK
> >  - activate / deactivate / rebalance / kill : OK
> >
> >Thanks,
> >Jungtaek Lim
> >
> >2016년 4월 28일 (목) 오전 3:29, P. Taylor Goetz
> >>님이 작성:
> >
> >This is a call to vote on releasing Apache Storm 0.10.1 (rc2)
> >
> >Full list of changes in this release:
> >
> >
> >https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGEL
> >OG.md;hb=8179921a569b6cf1d97798eed8e7b03b131bc495
> >
> >The tag/commit to be voted upon is v0.10.1:
> >
> >
> >https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=ddf051149f3c3
> >86342937efdabdaf45694602dd1;hb=8179921a569b6cf1d97798eed8e7b03b131bc495;a=
> >tree;h=26291835f22474506f0fe90b0459eab0d00bf4a9;hb=f0d3eae7395b3ee036b94b9
> >22707f74868ba6190
> >
> >The source archive being voted upon can be found here:
> >
> >
> >https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc2/apach
> >e-storm-0.10.1-src.tar.gz
> >
> >Other release files, signatures and digests can be found here:
> >
> >https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc2/
> >
> >The release artifacts are signed with the following key:
> >
> >
> >https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb
> >=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> >The Nexus staging repository for this release is:
> >
> >https://repository.apache.org/content/repositories/orgapachestorm-1031/
> >
> >Please vote on releasing this package as Apache Storm 0.10.1.
> >
> >When voting, please list the actions taken to verify the release.
> >
> >This vote will be open for at least 72 hours.
> >
> >[ ] +1 Release this package as Apache Storm 0.10.1
> >[ ]  0 No opinion
> >[ ] -1 Do not release this package because...
> >
> >Thanks to everyone who contributed to this release.
> >
> >-Taylor
> >
> >
> >
> >
> 

[jira] [Updated] (STORM-1761) Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster Mode

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-1761:

Assignee: Hugo Louro  (was: Jungtaek Lim)

> Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster 
> Mode
> ---
>
> Key: STORM-1761
> URL: https://issues.apache.org/jira/browse/STORM-1761
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Affects Versions: 1.0.1, 0.10.2
>Reporter: Hugo Louro
>Assignee: Hugo Louro
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (STORM-1761) Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster Mode

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim reopened STORM-1761:
-

We marked as 'Resolved' when all related pull requests are merged to respective 
branches.
Since patch is submitted, 'In Progress' makes sense. 

> Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster 
> Mode
> ---
>
> Key: STORM-1761
> URL: https://issues.apache.org/jira/browse/STORM-1761
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Affects Versions: 1.0.1, 0.10.2
>Reporter: Hugo Louro
>Assignee: Hugo Louro
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1761) Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster Mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269948#comment-15269948
 ] 

ASF GitHub Bot commented on STORM-1761:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1394#issuecomment-216712023
  
+1 Nice finding.
Btw, I guess it's due to inconsistencies between sample / test topologies, 
one argument for only specifying topology name, or two arguments which one is 
"remote" and next one is topology name.
I don't have strong opinion but it would be better to have unified way to 
run.


> Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster 
> Mode
> ---
>
> Key: STORM-1761
> URL: https://issues.apache.org/jira/browse/STORM-1761
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Affects Versions: 1.0.1, 0.10.2
>Reporter: Hugo Louro
>Assignee: Hugo Louro
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1761: Storm-Solr Example Throws ArrayInd...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1394#issuecomment-216712023
  
+1 Nice finding.
Btw, I guess it's due to inconsistencies between sample / test topologies, 
one argument for only specifying topology name, or two arguments which one is 
"remote" and next one is topology name.
I don't have strong opinion but it would be better to have unified way to 
run.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[P. Taylor Goetz (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269904#comment-15269904
 ] 

P. Taylor Goetz commented on STORM-1757:


My initial thoughts:

I asked on the beam ml early on where this should be developed, and there was a 
lot of support for doing so at beam.

The more I think about, the more I lean toward doing it here, with the 
possibility that it might someday be transferred. We just have a lot more 
collective knowledge of Storm, and are probably better suited to collaborate on 
an implementation.

My other thought is related to timing. It seems prudent to wait until the beam 
API has settled and there is a release to target, but I'm not opposed to 
getting started now.

> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Storm 0.10.1 (rc2)

[Roshan Naik]

This is missing HdfsSpout   (STORM-1199)
  
-roshan


On 5/3/16, 3:39 PM, "Hugo Da Cruz Louro"  wrote:

>+1 (non binding)
>- Found a very minor bug in the storm-solr test topology, for which there
>is a very simple workaround and by NO means a blocker for the release.
>- unzipped .zip sources and binaries
>- mvn clean install in unzipped sources
>- created uber jar for storm-solr
>- tested storm-solr in local cluster mode and remode cluster mode
>- ran test examples
>
>On May 3, 2016, at 12:04 AM, Arun Mahadevan
>> wrote:
>
>+1 (binding)
>- Extracted binaries
>- Ran sample topologies
>- Browsed storm UI
>
>Thanks,
>Arun
>
>
>
>
>On 4/28/16, 12:01 PM, "Jungtaek Lim"
>> wrote:
>
>+1 (binding)
>
>- testing with source distribution : OK
>- unzip : OK
>- building from source dist : OK
>  - how to build: running `mvn -P all-tests clean install` on unzipped
>source dist.
>
>- testing with binary distribution (one machine) : OK
>- launch daemons : OK
>- run RollingTopWords (local) : OK
>- run RollingTopWords (remote) : OK
>  - activate / deactivate / rebalance / kill : OK
>
>Thanks,
>Jungtaek Lim
>
>2016년 4월 28일 (목) 오전 3:29, P. Taylor Goetz
>>님이 작성:
>
>This is a call to vote on releasing Apache Storm 0.10.1 (rc2)
>
>Full list of changes in this release:
>
>
>https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGEL
>OG.md;hb=8179921a569b6cf1d97798eed8e7b03b131bc495
>
>The tag/commit to be voted upon is v0.10.1:
>
>
>https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=ddf051149f3c3
>86342937efdabdaf45694602dd1;hb=8179921a569b6cf1d97798eed8e7b03b131bc495;a=
>tree;h=26291835f22474506f0fe90b0459eab0d00bf4a9;hb=f0d3eae7395b3ee036b94b9
>22707f74868ba6190
>
>The source archive being voted upon can be found here:
>
>
>https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc2/apach
>e-storm-0.10.1-src.tar.gz
>
>Other release files, signatures and digests can be found here:
>
>https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc2/
>
>The release artifacts are signed with the following key:
>
>
>https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb
>=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
>The Nexus staging repository for this release is:
>
>https://repository.apache.org/content/repositories/orgapachestorm-1031/
>
>Please vote on releasing this package as Apache Storm 0.10.1.
>
>When voting, please list the actions taken to verify the release.
>
>This vote will be open for at least 72 hours.
>
>[ ] +1 Release this package as Apache Storm 0.10.1
>[ ]  0 No opinion
>[ ] -1 Do not release this package because...
>
>Thanks to everyone who contributed to this release.
>
>-Taylor
>
>
>
>

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Boyang Jerry Peng (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269888#comment-15269888
 ] 

Boyang Jerry Peng commented on STORM-1757:
--

I think this would be a worth while project.  The google dataflow API  is very 
well thought out.  We could potentially increase the user base of Storm by 
doing this.  There is also demand within Yahoo for this feature

> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Roshan Naik]

Never mind my prev comment… got confused by looking at BEAM-216.

-roshan


On 5/3/16, 4:55 PM, "Roshan Naik"  wrote:

>Shouldn¹t we have a BEAM runner in Storm project instead of  Storm runner
>in Beam project ?
>-roshan
>
>On 5/3/16, 1:15 PM, "Hugo Louro (JIRA)"  wrote:
>
>>
>>[ 
>>https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.
>>p
>>lugin.system.issuetabpanels:comment-tabpanel=15269498#co
>>m
>>ment-15269498 ] 
>>
>>Hugo Louro commented on STORM-1757:
>>---
>>
>>I also would like to work on this project.
>>
>>> Apache Beam Runner for Storm
>>> 
>>>
>>> Key: STORM-1757
>>> URL: https://issues.apache.org/jira/browse/STORM-1757
>>> Project: Apache Storm
>>>  Issue Type: Brainstorming
>>>Reporter: P. Taylor Goetz
>>>Priority: Minor
>>>
>>> This is a call for interested parties to collaborate on an Apache Beam
>>>[1] runner for Storm, and express their thoughts and opinions.
>>> Given the addition of the Windowing API to Apache Storm, we should be
>>>able to map naturally to the Beam API. If not, it may be indicative of
>>>shortcomings of the Storm API that should be addressed.
>>> [1] http://beam.incubator.apache.org
>>
>>
>>
>>--
>>This message was sent by Atlassian JIRA
>>(v6.3.4#6332)
>>
>

Re: [jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Roshan Naik]

Shouldn¹t we have a BEAM runner in Storm project instead of  Storm runner
in Beam project ?
-roshan

On 5/3/16, 1:15 PM, "Hugo Louro (JIRA)"  wrote:

>
>[ 
>https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.p
>lugin.system.issuetabpanels:comment-tabpanel=15269498#com
>ment-15269498 ] 
>
>Hugo Louro commented on STORM-1757:
>---
>
>I also would like to work on this project.
>
>> Apache Beam Runner for Storm
>> 
>>
>> Key: STORM-1757
>> URL: https://issues.apache.org/jira/browse/STORM-1757
>> Project: Apache Storm
>>  Issue Type: Brainstorming
>>Reporter: P. Taylor Goetz
>>Priority: Minor
>>
>> This is a call for interested parties to collaborate on an Apache Beam
>>[1] runner for Storm, and express their thoughts and opinions.
>> Given the addition of the Windowing API to Apache Storm, we should be
>>able to map naturally to the Beam API. If not, it may be indicative of
>>shortcomings of the Storm API that should be addressed.
>> [1] http://beam.incubator.apache.org
>
>
>
>--
>This message was sent by Atlassian JIRA
>(v6.3.4#6332)
>

[jira] [Resolved] (STORM-1761) Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster Mode

[Hugo Louro (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hugo Louro resolved STORM-1761.
---
Resolution: Fixed

> Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster 
> Mode
> ---
>
> Key: STORM-1761
> URL: https://issues.apache.org/jira/browse/STORM-1761
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Affects Versions: 1.0.1, 0.10.2
>Reporter: Hugo Louro
>Assignee: Hugo Louro
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1761) Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster Mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269847#comment-15269847
 ] 

ASF GitHub Bot commented on STORM-1761:
---

GitHub user hmcl opened a pull request:

https://github.com/apache/storm/pull/1394

STORM-1761: Storm-Solr Example Throws ArrayIndexOutOfBoundsException in 
Remote Cluster Mode

Please cherry-pick this patch into 0.10.x-branch and 1.x-branch as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hmcl/storm-apache master_STORM-1761

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1394.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1394


commit 514b932c2f2b643e6865fd82bc6d720acecfa301
Author: Hugo Louro 
Date:   2016-05-03T23:32:45Z

STORM-1761: Storm-Solr Example Throws ArrayIndexOutOfBoundsException in 
Remote Cluster Mode




> Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster 
> Mode
> ---
>
> Key: STORM-1761
> URL: https://issues.apache.org/jira/browse/STORM-1761
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Affects Versions: 1.0.1, 0.10.2
>Reporter: Hugo Louro
>Assignee: Hugo Louro
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1761: Storm-Solr Example Throws ArrayInd...

[hmcl]

GitHub user hmcl opened a pull request:

https://github.com/apache/storm/pull/1394

STORM-1761: Storm-Solr Example Throws ArrayIndexOutOfBoundsException in 
Remote Cluster Mode

Please cherry-pick this patch into 0.10.x-branch and 1.x-branch as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hmcl/storm-apache master_STORM-1761

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1394.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1394


commit 514b932c2f2b643e6865fd82bc6d720acecfa301
Author: Hugo Louro 
Date:   2016-05-03T23:32:45Z

STORM-1761: Storm-Solr Example Throws ArrayIndexOutOfBoundsException in 
Remote Cluster Mode




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (STORM-1761) Storm-Solr Example Throws ArrayIndexOutOfBoundsException in Remote Cluster Mode

[Hugo Louro (JIRA)]

Hugo Louro created STORM-1761:
-

 Summary: Storm-Solr Example Throws ArrayIndexOutOfBoundsException 
in Remote Cluster Mode
 Key: STORM-1761
 URL: https://issues.apache.org/jira/browse/STORM-1761
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-solr
Affects Versions: 1.0.1, 0.10.2
Reporter: Hugo Louro
Assignee: Hugo Louro
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Storm 0.10.1 (rc2)

[Hugo Da Cruz Louro]

+1 (non binding)
- Found a very minor bug in the storm-solr test topology, for which there is a 
very simple workaround and by NO means a blocker for the release.
- unzipped .zip sources and binaries
- mvn clean install in unzipped sources
- created uber jar for storm-solr
- tested storm-solr in local cluster mode and remode cluster mode
- ran test examples

On May 3, 2016, at 12:04 AM, Arun Mahadevan 
> wrote:

+1 (binding)
- Extracted binaries
- Ran sample topologies
- Browsed storm UI

Thanks,
Arun




On 4/28/16, 12:01 PM, "Jungtaek Lim" 
> wrote:

+1 (binding)

- testing with source distribution : OK
- unzip : OK
- building from source dist : OK
  - how to build: running `mvn -P all-tests clean install` on unzipped
source dist.

- testing with binary distribution (one machine) : OK
- launch daemons : OK
- run RollingTopWords (local) : OK
- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK

Thanks,
Jungtaek Lim

2016년 4월 28일 (목) 오전 3:29, P. Taylor Goetz 
>님이 작성:

This is a call to vote on releasing Apache Storm 0.10.1 (rc2)

Full list of changes in this release:


https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=8179921a569b6cf1d97798eed8e7b03b131bc495

The tag/commit to be voted upon is v0.10.1:


https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=ddf051149f3c386342937efdabdaf45694602dd1;hb=8179921a569b6cf1d97798eed8e7b03b131bc495;a=tree;h=26291835f22474506f0fe90b0459eab0d00bf4a9;hb=f0d3eae7395b3ee036b94b922707f74868ba6190

The source archive being voted upon can be found here:


https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc2/apache-storm-0.10.1-src.tar.gz

Other release files, signatures and digests can be found here:

https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc2/

The release artifacts are signed with the following key:


https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd

The Nexus staging repository for this release is:

https://repository.apache.org/content/repositories/orgapachestorm-1031/

Please vote on releasing this package as Apache Storm 0.10.1.

When voting, please list the actions taken to verify the release.

This vote will be open for at least 72 hours.

[ ] +1 Release this package as Apache Storm 0.10.1
[ ]  0 No opinion
[ ] -1 Do not release this package because...

Thanks to everyone who contributed to this release.

-Taylor

Re: [VOTE] Release Apache Storm 1.0.1 (rc3)

[Hugo Da Cruz Louro]

+1 (non binding)
- Found a very minor bug in the storm-solr test topology, for which there is a 
very simple workaround and by NO means a blocker for the release.
- unzipped .zip sources and binaries
- mvn clean install in unzipped sources
- created uber jar for storm-solr and storm-kafka-client
- tested storm-solr and storm-kafka-client in local cluster mode and remode 
cluster mode

On May 2, 2016, at 11:38 PM, Arun Mahadevan 
> wrote:

+1 (binding)
- Extracted all binaries.
- Ran sample topologies.
- Verified Stateful bolt changes added in 1.0.1 works as expected.

Thanks,
Arun





On 5/3/16, 4:52 AM, "Harsha" > wrote:

+1 (binding)
- Deployed 3-node cluster  and example topologies
- Verified the binaries signature.
Thanks,
Harsha

On Sun, May 1, 2016, at 07:11 PM, Jungtaek Lim wrote:
+1 (binding)

- build succeed from source code
- binary extracted well, and supervisor launches worker fine even though
JAVA_HOME is unset (STORM-1741) / also checked when JAVA_HOME is set

Thanks all of efforts you all have been done with this release.

Thanks,
Jungtaek Lim (HeartSaVioR)


2016년 4월 30일 (토) 오전 10:08, Xin Wang 
>님이 작성:

+1 (Non binding)

2016-04-30 5:28 GMT+08:00 P. Taylor Goetz 
>:

If at first you don't succeed... ;)

This is a call to vote on releasing Apache Storm 1.0.1 (rc3)

Full list of changes in this release:



https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=b5c16f919ad4099e6fb25f1095c9af8b64ac9f91

The tag/commit to be voted upon is v1.0.1:



https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=14b2ac0fe33759a50a2392ebdc7ad3821d43b89f;hb=b5c16f919ad4099e6fb25f1095c9af8b64ac9f91

The source archive being voted upon can be found here:



https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.1-rc3/apache-storm-1.0.1-src.tar.gz

Other release files, signatures and digests can be found here:

https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.1-rc3/

The release artifacts are signed with the following key:



https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd

The Nexus staging repository for this release is:

https://repository.apache.org/content/repositories/orgapachestorm-1033/

Please vote on releasing this package as Apache Storm 1.0.1.

When voting, please list the actions taken to verify the release.

This vote will be open for at least 72 hours.

[ ] +1 Release this package as Apache Storm 1.0.1
[ ]  0 No opinion
[ ] -1 Do not release this package because...

Thanks to everyone who contributed to this release.

-Taylor

[jira] [Commented] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269614#comment-15269614
 ] 

ASF GitHub Bot commented on STORM-1674:
---

GitHub user moesol opened a pull request:

https://github.com/apache/storm/pull/1393

(STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

 * Allows minBytes in fetch request to be configured
  from KafkaConfig.fetchMinBytes.
* Defaults new configuration KafkaConfig.fetchMinBytes to 1.
* Defaults fetchMaxWait to 100ms instead of 1ms.

Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
and our kafka servers even though no Kafka messages were moving.
Using the wireshark kafka dissector, we were able to see that
each FetchRequest had maxWait set to 1
and minBytes set to 0. When binBytes is set to 0 the kafka server
responds immediately when there are no messages. In turn the KafkaSpout
polls without any delay causing a constant stream of FetchRequest/
FetchResponse messages. Using a non-KafkaSpout client had a similar
traffic pattern with two key differences
1) minBytes was 1
2) maxWait was 100
With these FetchRequest parameters and no messages flowing,
the kafka server delays the FetchResponse by 100 ms. This reduces
the network traffic from megabits to the low kilobits. It also
reduced the CPU utilization of our kafka server from 140% to 2%.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MoebiusSolutions/storm 1.x-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1393.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1393


commit ff43309e39fb1db2bf2ae5a3d7d0972440880dca
Author: Robert Hastings 
Date:   2016-05-03T20:40:56Z

Addresses network flood from KafkaSpout to kafka server.

* Allows minBytes in fetch request to be configured
  from KafkaConfig.fetchMinBytes.
* Defaults new configuration KafkaConfig.fetchMinBytes to 1.
* Defaults fetchMaxWait to 100ms instead of 1ms.

Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
and our kafka servers even though no Kafka messages were moving.
Using the wireshark kafka dissector, we were able to see that
each FetchRequest had maxWait set to 1
and minBytes set to 0. When binBytes is set to 0 the kafka server
responds immediately when there are no messages. In turn the KafkaSpout
polls without any delay causing a constant stream of FetchRequest/
FetchResponse messages. Using a non-KafkaSpout client had a similar
traffic pattern with two key differences
1) minBytes was 1
2) maxWait was 100
With these FetchRequest parameters and no messages flowing,
the kafka server delays the FetchResponse by 100 ms. This reduces
the network traffic from megabits to the low kilobits. It also
reduced the CPU utilization of our kafka server from 140% to 2%.

Hopefully the risk of this change is low because
the old behavior can be restored using the API by setting
KafkaConfig.fetchMaxWait to 1
KafkaConfig.fetchMinBytes to 0

Conflicts:
external/storm-kafka/src/jvm/storm/kafka/KafkaConfig.java
external/storm-kafka/src/jvm/storm/kafka/KafkaUtils.java

commit d936ad78169283fcd8e0b8923eefc4063b59f074
Author: Robert Hastings 
Date:   2016-05-03T21:19:21Z

Merge remote-tracking branch 'apache/1.x-branch' into 1.x-branch

Conflicts:
external/storm-kafka/src/jvm/org/apache/storm/kafka/KafkaUtils.java




> Idle KafkaSpout consumes more bandwidth than needed
> ---
>
> Key: STORM-1674
> URL: https://issues.apache.org/jira/browse/STORM-1674
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.9.3
>Reporter: Robert Hastings
>
> Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
> and our kafka servers even though no Kafka messages were moving.
> Using the wireshark kafka dissector, we were able to see that
> each FetchRequest had maxWait set to 1
> and minBytes set to 0. When binBytes is set to 0 the kafka server
> responds immediately when there are no messages. In turn the KafkaSpout
> polls without any delay causing a constant stream of FetchRequest/
> FetchResponse messages. Using a non-KafkaSpout client had a similar
> traffic pattern with two key differences
> 1) minBytes was 1
> 2) maxWait was 100
> With these 

[GitHub] storm pull request: (STORM-1674) Idle KafkaSpout consumes more ban...

[moesol]

GitHub user moesol opened a pull request:

https://github.com/apache/storm/pull/1393

(STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

 * Allows minBytes in fetch request to be configured
  from KafkaConfig.fetchMinBytes.
* Defaults new configuration KafkaConfig.fetchMinBytes to 1.
* Defaults fetchMaxWait to 100ms instead of 1ms.

Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
and our kafka servers even though no Kafka messages were moving.
Using the wireshark kafka dissector, we were able to see that
each FetchRequest had maxWait set to 1
and minBytes set to 0. When binBytes is set to 0 the kafka server
responds immediately when there are no messages. In turn the KafkaSpout
polls without any delay causing a constant stream of FetchRequest/
FetchResponse messages. Using a non-KafkaSpout client had a similar
traffic pattern with two key differences
1) minBytes was 1
2) maxWait was 100
With these FetchRequest parameters and no messages flowing,
the kafka server delays the FetchResponse by 100 ms. This reduces
the network traffic from megabits to the low kilobits. It also
reduced the CPU utilization of our kafka server from 140% to 2%.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MoebiusSolutions/storm 1.x-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1393.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1393


commit ff43309e39fb1db2bf2ae5a3d7d0972440880dca
Author: Robert Hastings 
Date:   2016-05-03T20:40:56Z

Addresses network flood from KafkaSpout to kafka server.

* Allows minBytes in fetch request to be configured
  from KafkaConfig.fetchMinBytes.
* Defaults new configuration KafkaConfig.fetchMinBytes to 1.
* Defaults fetchMaxWait to 100ms instead of 1ms.

Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
and our kafka servers even though no Kafka messages were moving.
Using the wireshark kafka dissector, we were able to see that
each FetchRequest had maxWait set to 1
and minBytes set to 0. When binBytes is set to 0 the kafka server
responds immediately when there are no messages. In turn the KafkaSpout
polls without any delay causing a constant stream of FetchRequest/
FetchResponse messages. Using a non-KafkaSpout client had a similar
traffic pattern with two key differences
1) minBytes was 1
2) maxWait was 100
With these FetchRequest parameters and no messages flowing,
the kafka server delays the FetchResponse by 100 ms. This reduces
the network traffic from megabits to the low kilobits. It also
reduced the CPU utilization of our kafka server from 140% to 2%.

Hopefully the risk of this change is low because
the old behavior can be restored using the API by setting
KafkaConfig.fetchMaxWait to 1
KafkaConfig.fetchMinBytes to 0

Conflicts:
external/storm-kafka/src/jvm/storm/kafka/KafkaConfig.java
external/storm-kafka/src/jvm/storm/kafka/KafkaUtils.java

commit d936ad78169283fcd8e0b8923eefc4063b59f074
Author: Robert Hastings 
Date:   2016-05-03T21:19:21Z

Merge remote-tracking branch 'apache/1.x-branch' into 1.x-branch

Conflicts:
external/storm-kafka/src/jvm/org/apache/storm/kafka/KafkaUtils.java




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269608#comment-15269608
 ] 

ASF GitHub Bot commented on STORM-1674:
---

Github user moesol commented on the pull request:

https://github.com/apache/storm/pull/1391#issuecomment-21896
  
Without the MaxWait being set to 100, it is set to 1, which means that 
the bolt will waiting for 10 seconds when no messages are available. On the 1.x 
branch, I was going to use:

+public int fetchMaxWait = FetchRequest.DefaultMaxWait();
+public int fetchMinBytes = FetchRequest.DefaultMinBytes();

to get Kafka default behavior of 100/1.



> Idle KafkaSpout consumes more bandwidth than needed
> ---
>
> Key: STORM-1674
> URL: https://issues.apache.org/jira/browse/STORM-1674
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.9.3
>Reporter: Robert Hastings
>
> Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
> and our kafka servers even though no Kafka messages were moving.
> Using the wireshark kafka dissector, we were able to see that
> each FetchRequest had maxWait set to 1
> and minBytes set to 0. When binBytes is set to 0 the kafka server
> responds immediately when there are no messages. In turn the KafkaSpout
> polls without any delay causing a constant stream of FetchRequest/
> FetchResponse messages. Using a non-KafkaSpout client had a similar
> traffic pattern with two key differences
> 1) minBytes was 1
> 2) maxWait was 100
> With these FetchRequest parameters and no messages flowing,
> the kafka server delays the FetchResponse by 100 ms. This reduces
> the network traffic from megabits to the low kilobits. It also
> reduced the CPU utilization of our kafka server from 140% to 2%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: (STORM-1674) Idle KafkaSpout consumes more ban...

[moesol]

Github user moesol commented on the pull request:

https://github.com/apache/storm/pull/1391#issuecomment-21896
  
Without the MaxWait being set to 100, it is set to 1, which means that 
the bolt will waiting for 10 seconds when no messages are available. On the 1.x 
branch, I was going to use:

+public int fetchMaxWait = FetchRequest.DefaultMaxWait();
+public int fetchMinBytes = FetchRequest.DefaultMinBytes();

to get Kafka default behavior of 100/1.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269582#comment-15269582
 ] 

ASF GitHub Bot commented on STORM-1674:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1391#issuecomment-216663883
  
@moesol there is patch that merged into 1.0 release 
https://github.com/apache/storm/pull/1309/files . Does that changes work for 
you. Any reason for the MaxWait to be at 100


> Idle KafkaSpout consumes more bandwidth than needed
> ---
>
> Key: STORM-1674
> URL: https://issues.apache.org/jira/browse/STORM-1674
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.9.3
>Reporter: Robert Hastings
>
> Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
> and our kafka servers even though no Kafka messages were moving.
> Using the wireshark kafka dissector, we were able to see that
> each FetchRequest had maxWait set to 1
> and minBytes set to 0. When binBytes is set to 0 the kafka server
> responds immediately when there are no messages. In turn the KafkaSpout
> polls without any delay causing a constant stream of FetchRequest/
> FetchResponse messages. Using a non-KafkaSpout client had a similar
> traffic pattern with two key differences
> 1) minBytes was 1
> 2) maxWait was 100
> With these FetchRequest parameters and no messages flowing,
> the kafka server delays the FetchResponse by 100 ms. This reduces
> the network traffic from megabits to the low kilobits. It also
> reduced the CPU utilization of our kafka server from 140% to 2%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: (STORM-1674) Idle KafkaSpout consumes more ban...

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1391#issuecomment-216663883
  
@moesol there is patch that merged into 1.0 release 
https://github.com/apache/storm/pull/1309/files . Does that changes work for 
you. Any reason for the MaxWait to be at 100


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1760) HiveState should retire idle or old writes with flushAndClose

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269572#comment-15269572
 ] 

ASF GitHub Bot commented on STORM-1760:
---

GitHub user harshach opened a pull request:

https://github.com/apache/storm/pull/1392

STORM-1760. HiveState should retire idle or old writes with flushAndClose



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/harshach/incubator-storm STORM-1760

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1392.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1392


commit 90e3d7078f74655765e564be571ab96e9e844e4e
Author: Sriharsha Chintalapani 
Date:   2016-05-03T20:58:46Z

STORM-1760. HiveState should retire idle or old writes with flushAndClose.




> HiveState should retire idle or old writes with flushAndClose
> -
>
> Key: STORM-1760
> URL: https://issues.apache.org/jira/browse/STORM-1760
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hive
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 1.0.0, 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1760. HiveState should retire idle or ol...

[harshach]

GitHub user harshach opened a pull request:

https://github.com/apache/storm/pull/1392

STORM-1760. HiveState should retire idle or old writes with flushAndClose



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/harshach/incubator-storm STORM-1760

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1392.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1392


commit 90e3d7078f74655765e564be571ab96e9e844e4e
Author: Sriharsha Chintalapani 
Date:   2016-05-03T20:58:46Z

STORM-1760. HiveState should retire idle or old writes with flushAndClose.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269570#comment-15269570
 ] 

ASF GitHub Bot commented on STORM-1674:
---

GitHub user moesol opened a pull request:

https://github.com/apache/storm/pull/1391

(STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

* Allows minBytes in fetch request to be configured
  from KafkaConfig.fetchMinBytes.
* Defaults new configuration KafkaConfig.fetchMinBytes to 1.
* Defaults fetchMaxWait to 100ms instead of 1ms.

Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
and our kafka servers even though no Kafka messages were moving.
Using the wireshark kafka dissector, we were able to see that
each FetchRequest had maxWait set to 1
and minBytes set to 0. When binBytes is set to 0 the kafka server
responds immediately when there are no messages. In turn the KafkaSpout
polls without any delay causing a constant stream of FetchRequest/
FetchResponse messages. Using a non-KafkaSpout client had a similar
traffic pattern with two key differences
1) minBytes was 1
2) maxWait was 100
With these FetchRequest parameters and no messages flowing,
the kafka server delays the FetchResponse by 100 ms. This reduces
the network traffic from megabits to the low kilobits. It also
reduced the CPU utilization of our kafka server from 140% to 2%.

Hopefully the risk of this change is low because
the old behavior can be restored using the API by setting
KafkaConfig.fetchMaxWait to 1
KafkaConfig.fetchMinBytes to 0

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MoebiusSolutions/storm 0.10.x-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1391.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1391


commit ea1d3189c8113cc2888366fa5b24579776279886
Author: Robert Hastings 
Date:   2016-03-31T23:14:47Z

Addresses network flood from KafkaSpout to kafka server.

* Allows minBytes in fetch request to be configured
  from KafkaConfig.fetchMinBytes.
* Defaults new configuration KafkaConfig.fetchMinBytes to 1.
* Defaults fetchMaxWait to 100ms instead of 1ms.

Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
and our kafka servers even though no Kafka messages were moving.
Using the wireshark kafka dissector, we were able to see that
each FetchRequest had maxWait set to 1
and minBytes set to 0. When binBytes is set to 0 the kafka server
responds immediately when there are no messages. In turn the KafkaSpout
polls without any delay causing a constant stream of FetchRequest/
FetchResponse messages. Using a non-KafkaSpout client had a similar
traffic pattern with two key differences
1) minBytes was 1
2) maxWait was 100
With these FetchRequest parameters and no messages flowing,
the kafka server delays the FetchResponse by 100 ms. This reduces
the network traffic from megabits to the low kilobits. It also
reduced the CPU utilization of our kafka server from 140% to 2%.

Hopefully the risk of this change is low because
the old behavior can be restored using the API by setting
KafkaConfig.fetchMaxWait to 1
KafkaConfig.fetchMinBytes to 0




> Idle KafkaSpout consumes more bandwidth than needed
> ---
>
> Key: STORM-1674
> URL: https://issues.apache.org/jira/browse/STORM-1674
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.9.3
>Reporter: Robert Hastings
>
> Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
> and our kafka servers even though no Kafka messages were moving.
> Using the wireshark kafka dissector, we were able to see that
> each FetchRequest had maxWait set to 1
> and minBytes set to 0. When binBytes is set to 0 the kafka server
> responds immediately when there are no messages. In turn the KafkaSpout
> polls without any delay causing a constant stream of FetchRequest/
> FetchResponse messages. Using a non-KafkaSpout client had a similar
> traffic pattern with two key differences
> 1) minBytes was 1
> 2) maxWait was 100
> With these FetchRequest parameters and no messages flowing,
> the kafka server delays the FetchResponse by 100 ms. This reduces
> the network traffic from megabits to the low kilobits. It also
> reduced the CPU utilization of our kafka server from 140% to 2%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: (STORM-1674) Idle KafkaSpout consumes more ban...

[moesol]

GitHub user moesol opened a pull request:

https://github.com/apache/storm/pull/1391

(STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

* Allows minBytes in fetch request to be configured
  from KafkaConfig.fetchMinBytes.
* Defaults new configuration KafkaConfig.fetchMinBytes to 1.
* Defaults fetchMaxWait to 100ms instead of 1ms.

Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
and our kafka servers even though no Kafka messages were moving.
Using the wireshark kafka dissector, we were able to see that
each FetchRequest had maxWait set to 1
and minBytes set to 0. When binBytes is set to 0 the kafka server
responds immediately when there are no messages. In turn the KafkaSpout
polls without any delay causing a constant stream of FetchRequest/
FetchResponse messages. Using a non-KafkaSpout client had a similar
traffic pattern with two key differences
1) minBytes was 1
2) maxWait was 100
With these FetchRequest parameters and no messages flowing,
the kafka server delays the FetchResponse by 100 ms. This reduces
the network traffic from megabits to the low kilobits. It also
reduced the CPU utilization of our kafka server from 140% to 2%.

Hopefully the risk of this change is low because
the old behavior can be restored using the API by setting
KafkaConfig.fetchMaxWait to 1
KafkaConfig.fetchMinBytes to 0

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MoebiusSolutions/storm 0.10.x-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1391.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1391


commit ea1d3189c8113cc2888366fa5b24579776279886
Author: Robert Hastings 
Date:   2016-03-31T23:14:47Z

Addresses network flood from KafkaSpout to kafka server.

* Allows minBytes in fetch request to be configured
  from KafkaConfig.fetchMinBytes.
* Defaults new configuration KafkaConfig.fetchMinBytes to 1.
* Defaults fetchMaxWait to 100ms instead of 1ms.

Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
and our kafka servers even though no Kafka messages were moving.
Using the wireshark kafka dissector, we were able to see that
each FetchRequest had maxWait set to 1
and minBytes set to 0. When binBytes is set to 0 the kafka server
responds immediately when there are no messages. In turn the KafkaSpout
polls without any delay causing a constant stream of FetchRequest/
FetchResponse messages. Using a non-KafkaSpout client had a similar
traffic pattern with two key differences
1) minBytes was 1
2) maxWait was 100
With these FetchRequest parameters and no messages flowing,
the kafka server delays the FetchResponse by 100 ms. This reduces
the network traffic from megabits to the low kilobits. It also
reduced the CPU utilization of our kafka server from 140% to 2%.

Hopefully the risk of this change is low because
the old behavior can be restored using the API by setting
KafkaConfig.fetchMaxWait to 1
KafkaConfig.fetchMinBytes to 0




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Hugo Louro (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269498#comment-15269498
 ] 

Hugo Louro commented on STORM-1757:
---

I also would like to work on this project.

> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

[Robert Hastings (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269303#comment-15269303
 ] 

Robert Hastings commented on STORM-1674:


Sure, in progress.

> Idle KafkaSpout consumes more bandwidth than needed
> ---
>
> Key: STORM-1674
> URL: https://issues.apache.org/jira/browse/STORM-1674
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.9.3
>Reporter: Robert Hastings
>
> Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
> and our kafka servers even though no Kafka messages were moving.
> Using the wireshark kafka dissector, we were able to see that
> each FetchRequest had maxWait set to 1
> and minBytes set to 0. When binBytes is set to 0 the kafka server
> responds immediately when there are no messages. In turn the KafkaSpout
> polls without any delay causing a constant stream of FetchRequest/
> FetchResponse messages. Using a non-KafkaSpout client had a similar
> traffic pattern with two key differences
> 1) minBytes was 1
> 2) maxWait was 100
> With these FetchRequest parameters and no messages flowing,
> the kafka server delays the FetchResponse by 100 ms. This reduces
> the network traffic from megabits to the low kilobits. It also
> reduced the CPU utilization of our kafka server from 140% to 2%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1736) Change KafkaTestBroker.buildKafkaConfig to new KafkaConfig api.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269032#comment-15269032
 ] 

ASF GitHub Bot commented on STORM-1736:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1376#issuecomment-216590606
  
@abhishekagarwal87 commented on your PR 
https://github.com/apache/storm/pull/1386 . 


> Change KafkaTestBroker.buildKafkaConfig to new KafkaConfig api.
> ---
>
> Key: STORM-1736
> URL: https://issues.apache.org/jira/browse/STORM-1736
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
>
> 1.x-branch failing with following error
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
> (default-testCompile) on project storm-kafka: Compilation failure
> [ERROR] 
> /Users/harsha/code/harshach/incubator-storm/external/storm-kafka/src/test/org/apache/storm/kafka/KafkaTestBroker.java:[70,16]
>  constructor KafkaConfig in class kafka.server.KafkaConfig cannot be applied 
> to given types;
> [ERROR] required: java.util.Map,boolean
> [ERROR] found: java.util.Properties
> [ERROR] reason: actual and formal argument lists differ in length
> [ERROR] -> [Help 1]
> [ERROR] 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1736. Change KafkaTestBroker.buildKafkaC...

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1376#issuecomment-216590606
  
@abhishekagarwal87 commented on your PR 
https://github.com/apache/storm/pull/1386 . 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1755: Revert the kafka client version to...

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1386#issuecomment-216590463
  
@abhishekagarwal87 what exactly are we trying to do here. Are we talking 
about the producer api not being comptabile with 0.8.x? . There was a suggested 
approach in storm-kafka-client patch where we decided to move producer-api 
under storm-kafka-client and make that version as 0.9. 

If users are intended to run a storm-kafka bolt with 0.8.x cluster they 
should be using previous version and we should note this release notes. Just 
making the verison change here is a temporary fix and also we are removing any 
potential new features in kafka client api out of user hands , example security.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1755) Revert the kafka client version upgrade in storm-kafka module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269030#comment-15269030
 ] 

ASF GitHub Bot commented on STORM-1755:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1386#issuecomment-216590463
  
@abhishekagarwal87 what exactly are we trying to do here. Are we talking 
about the producer api not being comptabile with 0.8.x? . There was a suggested 
approach in storm-kafka-client patch where we decided to move producer-api 
under storm-kafka-client and make that version as 0.9. 

If users are intended to run a storm-kafka bolt with 0.8.x cluster they 
should be using previous version and we should note this release notes. Just 
making the verison change here is a temporary fix and also we are removing any 
potential new features in kafka client api out of user hands , example security.


> Revert the kafka client version upgrade in storm-kafka module
> -
>
> Key: STORM-1755
> URL: https://issues.apache.org/jira/browse/STORM-1755
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Affects Versions: 1.0.1
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> storm-kafka module does not use any feature of new API. The newer kafka 
> client (0.9.x) is not backward compatible with 0.8.x brokers and upgrading 
> storm-kafka version in topology could break the currently running topologies. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1736. Change KafkaTestBroker.buildKafkaC...

[abhishekagarwal87]

Github user abhishekagarwal87 commented on the pull request:

https://github.com/apache/storm/pull/1376#issuecomment-216587066
  
@harshach - with this change, test compilation will fail once the #1386 is 
merged. Since this is the internal class used only in test, do we really need 
to make it compatible with 0.10 branch? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1736) Change KafkaTestBroker.buildKafkaConfig to new KafkaConfig api.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269005#comment-15269005
 ] 

ASF GitHub Bot commented on STORM-1736:
---

Github user abhishekagarwal87 commented on the pull request:

https://github.com/apache/storm/pull/1376#issuecomment-216587066
  
@harshach - with this change, test compilation will fail once the #1386 is 
merged. Since this is the internal class used only in test, do we really need 
to make it compatible with 0.10 branch? 


> Change KafkaTestBroker.buildKafkaConfig to new KafkaConfig api.
> ---
>
> Key: STORM-1736
> URL: https://issues.apache.org/jira/browse/STORM-1736
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
>
> 1.x-branch failing with following error
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
> (default-testCompile) on project storm-kafka: Compilation failure
> [ERROR] 
> /Users/harsha/code/harshach/incubator-storm/external/storm-kafka/src/test/org/apache/storm/kafka/KafkaTestBroker.java:[70,16]
>  constructor KafkaConfig in class kafka.server.KafkaConfig cannot be applied 
> to given types;
> [ERROR] required: java.util.Map,boolean
> [ERROR] found: java.util.Properties
> [ERROR] reason: actual and formal argument lists differ in length
> [ERROR] -> [Help 1]
> [ERROR] 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1760) HiveState should retire idle or old writes with flushAndClose

[Sriharsha Chintalapani (JIRA)]

Sriharsha Chintalapani created STORM-1760:
-

 Summary: HiveState should retire idle or old writes with 
flushAndClose
 Key: STORM-1760
 URL: https://issues.apache.org/jira/browse/STORM-1760
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-hive
Reporter: Sriharsha Chintalapani
Assignee: Sriharsha Chintalapani
 Fix For: 1.0.0, 2.0.0






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1736) Change KafkaTestBroker.buildKafkaConfig to new KafkaConfig api.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268960#comment-15268960
 ] 

ASF GitHub Bot commented on STORM-1736:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1376#issuecomment-216579561
  
@HeartSaVioR @abhishekagarwal87 this issues happens with Kafka 0.10 branch. 
Yes it does break test compatability with 0.8.x . Keep in mind that 
KafkaConfig is an internal kafka class that we use to start a in-process kafka 
broker. We shouldn't be calling it as breaking comptability with client api. 
Also this change will work with Kafka 0.9.x on wards


> Change KafkaTestBroker.buildKafkaConfig to new KafkaConfig api.
> ---
>
> Key: STORM-1736
> URL: https://issues.apache.org/jira/browse/STORM-1736
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
>
> 1.x-branch failing with following error
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
> (default-testCompile) on project storm-kafka: Compilation failure
> [ERROR] 
> /Users/harsha/code/harshach/incubator-storm/external/storm-kafka/src/test/org/apache/storm/kafka/KafkaTestBroker.java:[70,16]
>  constructor KafkaConfig in class kafka.server.KafkaConfig cannot be applied 
> to given types;
> [ERROR] required: java.util.Map,boolean
> [ERROR] found: java.util.Properties
> [ERROR] reason: actual and formal argument lists differ in length
> [ERROR] -> [Help 1]
> [ERROR] 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1736. Change KafkaTestBroker.buildKafkaC...

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1376#issuecomment-216579561
  
@HeartSaVioR @abhishekagarwal87 this issues happens with Kafka 0.10 branch. 
Yes it does break test compatability with 0.8.x . Keep in mind that 
KafkaConfig is an internal kafka class that we use to start a in-process kafka 
broker. We shouldn't be calling it as breaking comptability with client api. 
Also this change will work with Kafka 0.9.x on wards


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1646] Fix Kafka unit tests

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216576871
  
Yeah, I guess Kafka doesn't strictly respect semantic versioning, or their 
major version seems to 2nd number of the version at least before reaching 1.0.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1646) Intermittent test failures in storm-kafka unit tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268946#comment-15268946
 ] 

ASF GitHub Bot commented on STORM-1646:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216576871
  
Yeah, I guess Kafka doesn't strictly respect semantic versioning, or their 
major version seems to 2nd number of the version at least before reaching 1.0.0.


> Intermittent test failures in storm-kafka unit tests
> 
>
> Key: STORM-1646
> URL: https://issues.apache.org/jira/browse/STORM-1646
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> We have been seeing intermittent test failures in KafkaUtilsTest on slow 
> hardware and lightly resourced VMs, as well as an intermittent race condition 
> when running the storm-kafka ExponentialBackoffManager  unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1646) Intermittent test failures in storm-kafka unit tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268930#comment-15268930
 ] 

ASF GitHub Bot commented on STORM-1646:
---

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216573099
  
I might have been a bit quick on the trigger there. The Kafka coding 
guidelines target backward compatibility for one release only. So storm-kafka 
might not support 0.10.0.

(http://kafka.apache.org/coding-guide.html under backwards compatibility)


> Intermittent test failures in storm-kafka unit tests
> 
>
> Key: STORM-1646
> URL: https://issues.apache.org/jira/browse/STORM-1646
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> We have been seeing intermittent test failures in KafkaUtilsTest on slow 
> hardware and lightly resourced VMs, as well as an intermittent race condition 
> when running the storm-kafka ExponentialBackoffManager  unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1646] Fix Kafka unit tests

[srdo]

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216573099
  
I might have been a bit quick on the trigger there. The Kafka coding 
guidelines target backward compatibility for one release only. So storm-kafka 
might not support 0.10.0.

(http://kafka.apache.org/coding-guide.html under backwards compatibility)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1646] Fix Kafka unit tests

[srdo]

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216564619
  
storm-kafka will probably support 0.10.0, I believe the Kafka developers 
version their APIs so new clusters can talk to old clients. That shouldn't 
affect the choice of configuration parameters though, since any 0.8.x support 
in 0.10.0 would expect 0.8.x configuration parameters.

See 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala
 and how it's used in 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala
 if you're interested. The way I understand it is the client indicates which 
version of the API it wants to use, and then the broker uses that version of 
the API for requests/responses.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1700 Introduce 'whitelist' / 'blacklist'...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1324#discussion_r61901102
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/MetricsConsumerBolt.java ---
@@ -47,17 +68,71 @@ public void prepare(Map stormConf, TopologyContext 
context, OutputCollector coll
 }
 _metricsConsumer.prepare(stormConf, _registrationArgument, 
context, collector);
 _collector = collector;
+_taskExecuteThread = new Thread(new MetricsHandlerRunnable());
+_taskExecuteThread.setDaemon(true);
+_taskExecuteThread.start();
 }
 
 @Override
 public void execute(Tuple input) {
-
_metricsConsumer.handleDataPoints((IMetricsConsumer.TaskInfo)input.getValue(0), 
(Collection)input.getValue(1));
+// remove older tasks if task queue exceeds the max size
+if (_taskQueue.size() > _maxRetainMetricTuples) {
+while (_taskQueue.size() - 1 > _maxRetainMetricTuples) {
--- End diff --

I guess maybe it could drop one more item but it's not a big deal anyway. 
Ideally, taskQueue exceeding the max retain metric tuples should not occur.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1646) Intermittent test failures in storm-kafka unit tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268887#comment-15268887
 ] 

ASF GitHub Bot commented on STORM-1646:
---

Github user ppoulosk commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216565610
  
Thanks for the explanation.  Will revert that change.


> Intermittent test failures in storm-kafka unit tests
> 
>
> Key: STORM-1646
> URL: https://issues.apache.org/jira/browse/STORM-1646
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> We have been seeing intermittent test failures in KafkaUtilsTest on slow 
> hardware and lightly resourced VMs, as well as an intermittent race condition 
> when running the storm-kafka ExponentialBackoffManager  unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1646] Fix Kafka unit tests

[ppoulosk]

Github user ppoulosk commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216565610
  
Thanks for the explanation.  Will revert that change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1646) Intermittent test failures in storm-kafka unit tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268881#comment-15268881
 ] 

ASF GitHub Bot commented on STORM-1646:
---

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216564619
  
storm-kafka will probably support 0.10.0, I believe the Kafka developers 
version their APIs so new clusters can talk to old clients. That shouldn't 
affect the choice of configuration parameters though, since any 0.8.x support 
in 0.10.0 would expect 0.8.x configuration parameters.

See 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala
 and how it's used in 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala
 if you're interested. The way I understand it is the client indicates which 
version of the API it wants to use, and then the broker uses that version of 
the API for requests/responses.


> Intermittent test failures in storm-kafka unit tests
> 
>
> Key: STORM-1646
> URL: https://issues.apache.org/jira/browse/STORM-1646
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> We have been seeing intermittent test failures in KafkaUtilsTest on slow 
> hardware and lightly resourced VMs, as well as an intermittent race condition 
> when running the storm-kafka ExponentialBackoffManager  unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1700) Introduce 'whitelist' / 'blacklist' option to MetricsConsumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268883#comment-15268883
 ] 

ASF GitHub Bot commented on STORM-1700:
---

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1324#discussion_r61901102
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/MetricsConsumerBolt.java ---
@@ -47,17 +68,71 @@ public void prepare(Map stormConf, TopologyContext 
context, OutputCollector coll
 }
 _metricsConsumer.prepare(stormConf, _registrationArgument, 
context, collector);
 _collector = collector;
+_taskExecuteThread = new Thread(new MetricsHandlerRunnable());
+_taskExecuteThread.setDaemon(true);
+_taskExecuteThread.start();
 }
 
 @Override
 public void execute(Tuple input) {
-
_metricsConsumer.handleDataPoints((IMetricsConsumer.TaskInfo)input.getValue(0), 
(Collection)input.getValue(1));
+// remove older tasks if task queue exceeds the max size
+if (_taskQueue.size() > _maxRetainMetricTuples) {
+while (_taskQueue.size() - 1 > _maxRetainMetricTuples) {
--- End diff --

I guess maybe it could drop one more item but it's not a big deal anyway. 
Ideally, taskQueue exceeding the max retain metric tuples should not occur.


> Introduce 'whitelist' / 'blacklist' option to MetricsConsumer
> -
>
> Key: STORM-1700
> URL: https://issues.apache.org/jira/browse/STORM-1700
> Project: Apache Storm
>  Issue Type: Sub-task
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> Storm provides various metrics by default, and so on some external modules 
> (storm-kafka).
> When we register MetricsConsumer, MetricsConsumer should handle all of 
> metrics. If MetricsConsumer cannot keep up with these metrics, only way to 
> keep up is increasing parallelism, which seems limited. Furthermore, some 
> users don't want to care about some metrics since unintended metrics will 
> fill external storage.
> Though MetricsConsumer itself can filter metrics by name, it would be better 
> to support filter by Storm side. It will reduce the redundant works for Storm 
> community.
> If we provide filter options, it would be great.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1700 Introduce 'whitelist' / 'blacklist'...

[unsleepy22]

Github user unsleepy22 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1324#discussion_r61899624
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/MetricsConsumerBolt.java ---
@@ -47,17 +68,71 @@ public void prepare(Map stormConf, TopologyContext 
context, OutputCollector coll
 }
 _metricsConsumer.prepare(stormConf, _registrationArgument, 
context, collector);
 _collector = collector;
+_taskExecuteThread = new Thread(new MetricsHandlerRunnable());
+_taskExecuteThread.setDaemon(true);
+_taskExecuteThread.start();
 }
 
 @Override
 public void execute(Tuple input) {
-
_metricsConsumer.handleDataPoints((IMetricsConsumer.TaskInfo)input.getValue(0), 
(Collection)input.getValue(1));
+// remove older tasks if task queue exceeds the max size
+if (_taskQueue.size() > _maxRetainMetricTuples) {
+while (_taskQueue.size() - 1 > _maxRetainMetricTuples) {
--- End diff --

I guess we may run into a concurrent issue here, if while statement 
suffices, it poll one item, while at the same time, metrics consumer consumers 
one item, anyway, I don't think this matters too much so I'm ok to keep it as 
is.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1700) Introduce 'whitelist' / 'blacklist' option to MetricsConsumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268864#comment-15268864
 ] 

ASF GitHub Bot commented on STORM-1700:
---

Github user unsleepy22 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1324#discussion_r61899624
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/MetricsConsumerBolt.java ---
@@ -47,17 +68,71 @@ public void prepare(Map stormConf, TopologyContext 
context, OutputCollector coll
 }
 _metricsConsumer.prepare(stormConf, _registrationArgument, 
context, collector);
 _collector = collector;
+_taskExecuteThread = new Thread(new MetricsHandlerRunnable());
+_taskExecuteThread.setDaemon(true);
+_taskExecuteThread.start();
 }
 
 @Override
 public void execute(Tuple input) {
-
_metricsConsumer.handleDataPoints((IMetricsConsumer.TaskInfo)input.getValue(0), 
(Collection)input.getValue(1));
+// remove older tasks if task queue exceeds the max size
+if (_taskQueue.size() > _maxRetainMetricTuples) {
+while (_taskQueue.size() - 1 > _maxRetainMetricTuples) {
--- End diff --

I guess we may run into a concurrent issue here, if while statement 
suffices, it poll one item, while at the same time, metrics consumer consumers 
one item, anyway, I don't think this matters too much so I'm ok to keep it as 
is.


> Introduce 'whitelist' / 'blacklist' option to MetricsConsumer
> -
>
> Key: STORM-1700
> URL: https://issues.apache.org/jira/browse/STORM-1700
> Project: Apache Storm
>  Issue Type: Sub-task
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> Storm provides various metrics by default, and so on some external modules 
> (storm-kafka).
> When we register MetricsConsumer, MetricsConsumer should handle all of 
> metrics. If MetricsConsumer cannot keep up with these metrics, only way to 
> keep up is increasing parallelism, which seems limited. Furthermore, some 
> users don't want to care about some metrics since unintended metrics will 
> fill external storage.
> Though MetricsConsumer itself can filter metrics by name, it would be better 
> to support filter by Storm side. It will reduce the redundant works for Storm 
> community.
> If we provide filter options, it would be great.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (STORM-1758) Distributed log search doesn't work in dockerized environment

[Elisey Zanko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elisey Zanko updated STORM-1758:

Comment: was deleted

(was: It works. Thanks!)

> Distributed log search doesn't work in dockerized environment
> -
>
> Key: STORM-1758
> URL: https://issues.apache.org/jira/browse/STORM-1758
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-ui
>Affects Versions: 1.0.0
>Reporter: Elisey Zanko
>
> I run the Storm using the following docker-compose.yml
> {code}
> version: '2'
> services:
> zookeeper:
> image: jplock/zookeeper:3.4.8
> restart: always
> nimbus:
> image: 31z4/storm:1.0.0
> command: nimbus -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - zookeeper
> restart: always
> ports:
> - 6627:6627
> volumes:
> - logs:/logs
> supervisor:
> image: 31z4/storm:1.0.0
> command: supervisor -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> restart: always
> volumes:
> - logs:/logs
> logviewer:
> image: 31z4/storm:1.0.0
> command: logviewer -c storm.log.dir="/logs"
> restart: always
> ports:
> - 8000:8000
> volumes:
> - logs:/logs
> ui:
> image: 31z4/storm:1.0.0
> command: ui -c storm.log.dir="/logs" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> - logviewer
> restart: always
> ports:
> - 8080:8080
> volumes:
> - logs:/log
> volumes:
> logs: {}
> {code}
> And distributed log search doesn't work because the Storm UI tries to access 
> the logviewer by supervisor's container id as a host.
> Here is the list of running containers
> {code}
> $ docker ps
> 7ae118eef55c31z4/storm:1.0.0 "bin/storm ui -c stor"   5 
> minutes ago   Up 5 minutes   0.0.0.0:8080->8080/tcp 
> stormdocker_ui_1
> 5a9101dc251031z4/storm:1.0.0 "bin/storm supervisor"   5 
> minutes ago   Up 5 minutes  
> stormdocker_supervisor_1
> 4d954096cf1831z4/storm:1.0.0 "bin/storm nimbus -c "   5 
> minutes ago   Up 5 minutes   0.0.0.0:6627->6627/tcp 
> stormdocker_nimbus_1
> 070080342c4f31z4/storm:1.0.0 "bin/storm logviewer "   5 
> minutes ago   Up 5 minutes   0.0.0.0:8000->8000/tcp 
> stormdocker_logviewer_1
> 8650786a13ccjplock/zookeeper:3.4.8   "/opt/zookeeper/bin/z"   5 
> minutes ago   Up 5 minutes   2181/tcp, 2888/tcp, 3888/tcp   
> stormdocker_zookeeper_1
> {code}
> And here is what the Storm UI requests
> {code}
> curl 
> 'http://5a9101dc2510:8000/search/topology-1-1462284216%2F6701%2Fworker.log?search-string=split=1'
>  -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: 
> http://192.168.99.100:8080/search_result.html?search=split=topology-1-1462284216=1'
>  -H 'Origin: http://192.168.99.100:8080' -H 'User-Agent: Mozilla/5.0 
> (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) 
> Chrome/50.0.2661.86 Safari/537.36' --compressed
> {code}
> I guess adding an ability to explicitly specify the logviewer host in the 
> storm.yaml would solve the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1736) Change KafkaTestBroker.buildKafkaConfig to new KafkaConfig api.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268861#comment-15268861
 ] 

ASF GitHub Bot commented on STORM-1736:
---

Github user abhishekagarwal87 commented on the pull request:

https://github.com/apache/storm/pull/1376#issuecomment-216560931
  
This would also break the compatibility of storm 1.0.0 with 0.8.x kafka 
client API. 


> Change KafkaTestBroker.buildKafkaConfig to new KafkaConfig api.
> ---
>
> Key: STORM-1736
> URL: https://issues.apache.org/jira/browse/STORM-1736
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
>
> 1.x-branch failing with following error
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
> (default-testCompile) on project storm-kafka: Compilation failure
> [ERROR] 
> /Users/harsha/code/harshach/incubator-storm/external/storm-kafka/src/test/org/apache/storm/kafka/KafkaTestBroker.java:[70,16]
>  constructor KafkaConfig in class kafka.server.KafkaConfig cannot be applied 
> to given types;
> [ERROR] required: java.util.Map,boolean
> [ERROR] found: java.util.Properties
> [ERROR] reason: actual and formal argument lists differ in length
> [ERROR] -> [Help 1]
> [ERROR] 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1736. Change KafkaTestBroker.buildKafkaC...

[abhishekagarwal87]

Github user abhishekagarwal87 commented on the pull request:

https://github.com/apache/storm/pull/1376#issuecomment-216560931
  
This would also break the compatibility of storm 1.0.0 with 0.8.x kafka 
client API. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1700) Introduce 'whitelist' / 'blacklist' option to MetricsConsumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268857#comment-15268857
 ] 

ASF GitHub Bot commented on STORM-1700:
---

Github user unsleepy22 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1324#discussion_r61898673
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/filter/FilterByMetricName.java ---
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.metric.filter;
+
+import com.google.common.base.Function;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import org.apache.storm.metric.api.IMetricsConsumer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Pattern;
+
+public class FilterByMetricName implements MetricsFilter {
+
+private List whitelistPattern;
+private List blacklistPattern;
+private boolean noneSpecified = false;
+
+public FilterByMetricName(List whitelistPattern, List 
blacklistPattern) {
+// guard NPE
+if (whitelistPattern == null) {
+this.whitelistPattern = Collections.emptyList();
+} else {
+this.whitelistPattern = 
convertPatternStringsToPatternInstances(whitelistPattern);
+}
+
+// guard NPE
+if (blacklistPattern == null) {
+this.blacklistPattern = Collections.emptyList();
+} else {
+this.blacklistPattern = 
convertPatternStringsToPatternInstances(blacklistPattern);
+}
+
+if (this.whitelistPattern.isEmpty() && 
this.blacklistPattern.isEmpty()) {
+noneSpecified = true;
+} else if (!this.whitelistPattern.isEmpty() && 
!this.blacklistPattern.isEmpty()) {
+throw new IllegalArgumentException("You have to specify either 
includes or excludes, or none.");
+}
+}
+
+private ArrayList 
convertPatternStringsToPatternInstances(List patterns) {
+return Lists.newArrayList(Iterators.transform(patterns.iterator(), 
new Function() {
+@Override
+public Pattern apply(String s) {
+return Pattern.compile(s);
+}
+}));
+}
+
+@Override
+public boolean apply(IMetricsConsumer.DataPoint dataPoint) {
+if (noneSpecified) {
+return true;
+}
+
+String metricName = dataPoint.name;
+
+if (!whitelistPattern.isEmpty()) {
--- End diff --

OK, I personally prefer the latter approach.


> Introduce 'whitelist' / 'blacklist' option to MetricsConsumer
> -
>
> Key: STORM-1700
> URL: https://issues.apache.org/jira/browse/STORM-1700
> Project: Apache Storm
>  Issue Type: Sub-task
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> Storm provides various metrics by default, and so on some external modules 
> (storm-kafka).
> When we register MetricsConsumer, MetricsConsumer should handle all of 
> metrics. If MetricsConsumer cannot keep up with these metrics, only way to 
> keep up is increasing parallelism, which seems limited. Furthermore, some 
> users don't want to care about some metrics since unintended metrics will 
> fill external storage.
> Though MetricsConsumer itself can filter metrics by name, it would be better 
> to support filter by Storm side. It will reduce the redundant works for Storm 
> community.
> If we provide filter options, it would be great.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1700 Introduce 'whitelist' / 'blacklist'...

[unsleepy22]

Github user unsleepy22 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1324#discussion_r61898673
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/filter/FilterByMetricName.java ---
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.metric.filter;
+
+import com.google.common.base.Function;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import org.apache.storm.metric.api.IMetricsConsumer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Pattern;
+
+public class FilterByMetricName implements MetricsFilter {
+
+private List whitelistPattern;
+private List blacklistPattern;
+private boolean noneSpecified = false;
+
+public FilterByMetricName(List whitelistPattern, List 
blacklistPattern) {
+// guard NPE
+if (whitelistPattern == null) {
+this.whitelistPattern = Collections.emptyList();
+} else {
+this.whitelistPattern = 
convertPatternStringsToPatternInstances(whitelistPattern);
+}
+
+// guard NPE
+if (blacklistPattern == null) {
+this.blacklistPattern = Collections.emptyList();
+} else {
+this.blacklistPattern = 
convertPatternStringsToPatternInstances(blacklistPattern);
+}
+
+if (this.whitelistPattern.isEmpty() && 
this.blacklistPattern.isEmpty()) {
+noneSpecified = true;
+} else if (!this.whitelistPattern.isEmpty() && 
!this.blacklistPattern.isEmpty()) {
+throw new IllegalArgumentException("You have to specify either 
includes or excludes, or none.");
+}
+}
+
+private ArrayList 
convertPatternStringsToPatternInstances(List patterns) {
+return Lists.newArrayList(Iterators.transform(patterns.iterator(), 
new Function() {
+@Override
+public Pattern apply(String s) {
+return Pattern.compile(s);
+}
+}));
+}
+
+@Override
+public boolean apply(IMetricsConsumer.DataPoint dataPoint) {
+if (noneSpecified) {
+return true;
+}
+
+String metricName = dataPoint.name;
+
+if (!whitelistPattern.isEmpty()) {
--- End diff --

OK, I personally prefer the latter approach.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1758) Distributed log search doesn't work in dockerized environment

[Elisey Zanko (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268852#comment-15268852
 ] 

Elisey Zanko commented on STORM-1758:
-

Actually it's only necessary to add this into the nimbus and supervisor 
configuration.

> Distributed log search doesn't work in dockerized environment
> -
>
> Key: STORM-1758
> URL: https://issues.apache.org/jira/browse/STORM-1758
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-ui
>Affects Versions: 1.0.0
>Reporter: Elisey Zanko
>
> I run the Storm using the following docker-compose.yml
> {code}
> version: '2'
> services:
> zookeeper:
> image: jplock/zookeeper:3.4.8
> restart: always
> nimbus:
> image: 31z4/storm:1.0.0
> command: nimbus -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - zookeeper
> restart: always
> ports:
> - 6627:6627
> volumes:
> - logs:/logs
> supervisor:
> image: 31z4/storm:1.0.0
> command: supervisor -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> restart: always
> volumes:
> - logs:/logs
> logviewer:
> image: 31z4/storm:1.0.0
> command: logviewer -c storm.log.dir="/logs"
> restart: always
> ports:
> - 8000:8000
> volumes:
> - logs:/logs
> ui:
> image: 31z4/storm:1.0.0
> command: ui -c storm.log.dir="/logs" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> - logviewer
> restart: always
> ports:
> - 8080:8080
> volumes:
> - logs:/log
> volumes:
> logs: {}
> {code}
> And distributed log search doesn't work because the Storm UI tries to access 
> the logviewer by supervisor's container id as a host.
> Here is the list of running containers
> {code}
> $ docker ps
> 7ae118eef55c31z4/storm:1.0.0 "bin/storm ui -c stor"   5 
> minutes ago   Up 5 minutes   0.0.0.0:8080->8080/tcp 
> stormdocker_ui_1
> 5a9101dc251031z4/storm:1.0.0 "bin/storm supervisor"   5 
> minutes ago   Up 5 minutes  
> stormdocker_supervisor_1
> 4d954096cf1831z4/storm:1.0.0 "bin/storm nimbus -c "   5 
> minutes ago   Up 5 minutes   0.0.0.0:6627->6627/tcp 
> stormdocker_nimbus_1
> 070080342c4f31z4/storm:1.0.0 "bin/storm logviewer "   5 
> minutes ago   Up 5 minutes   0.0.0.0:8000->8000/tcp 
> stormdocker_logviewer_1
> 8650786a13ccjplock/zookeeper:3.4.8   "/opt/zookeeper/bin/z"   5 
> minutes ago   Up 5 minutes   2181/tcp, 2888/tcp, 3888/tcp   
> stormdocker_zookeeper_1
> {code}
> And here is what the Storm UI requests
> {code}
> curl 
> 'http://5a9101dc2510:8000/search/topology-1-1462284216%2F6701%2Fworker.log?search-string=split=1'
>  -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: 
> http://192.168.99.100:8080/search_result.html?search=split=topology-1-1462284216=1'
>  -H 'Origin: http://192.168.99.100:8080' -H 'User-Agent: Mozilla/5.0 
> (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) 
> Chrome/50.0.2661.86 Safari/537.36' --compressed
> {code}
> I guess adding an ability to explicitly specify the logviewer host in the 
> storm.yaml would solve the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1755) Revert the kafka client version upgrade in storm-kafka module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268850#comment-15268850
 ] 

ASF GitHub Bot commented on STORM-1755:
---

Github user abhishekagarwal87 commented on the pull request:

https://github.com/apache/storm/pull/1386#issuecomment-216558668
  
Yes. Good thing is storm-kafka doesn't use any api only available in kafka 
0.9.x and thus can still run with kafka 0.8.x by simply excluding dependency. 


> Revert the kafka client version upgrade in storm-kafka module
> -
>
> Key: STORM-1755
> URL: https://issues.apache.org/jira/browse/STORM-1755
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Affects Versions: 1.0.1
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> storm-kafka module does not use any feature of new API. The newer kafka 
> client (0.9.x) is not backward compatible with 0.8.x brokers and upgrading 
> storm-kafka version in topology could break the currently running topologies. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1755: Revert the kafka client version to...

[abhishekagarwal87]

Github user abhishekagarwal87 commented on the pull request:

https://github.com/apache/storm/pull/1386#issuecomment-216558668
  
Yes. Good thing is storm-kafka doesn't use any api only available in kafka 
0.9.x and thus can still run with kafka 0.8.x by simply excluding dependency. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1758) Distributed log search doesn't work in dockerized environment

[Elisey Zanko (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268836#comment-15268836
 ] 

Elisey Zanko commented on STORM-1758:
-

It works. Thanks!

> Distributed log search doesn't work in dockerized environment
> -
>
> Key: STORM-1758
> URL: https://issues.apache.org/jira/browse/STORM-1758
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-ui
>Affects Versions: 1.0.0
>Reporter: Elisey Zanko
>
> I run the Storm using the following docker-compose.yml
> {code}
> version: '2'
> services:
> zookeeper:
> image: jplock/zookeeper:3.4.8
> restart: always
> nimbus:
> image: 31z4/storm:1.0.0
> command: nimbus -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - zookeeper
> restart: always
> ports:
> - 6627:6627
> volumes:
> - logs:/logs
> supervisor:
> image: 31z4/storm:1.0.0
> command: supervisor -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> restart: always
> volumes:
> - logs:/logs
> logviewer:
> image: 31z4/storm:1.0.0
> command: logviewer -c storm.log.dir="/logs"
> restart: always
> ports:
> - 8000:8000
> volumes:
> - logs:/logs
> ui:
> image: 31z4/storm:1.0.0
> command: ui -c storm.log.dir="/logs" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> - logviewer
> restart: always
> ports:
> - 8080:8080
> volumes:
> - logs:/log
> volumes:
> logs: {}
> {code}
> And distributed log search doesn't work because the Storm UI tries to access 
> the logviewer by supervisor's container id as a host.
> Here is the list of running containers
> {code}
> $ docker ps
> 7ae118eef55c31z4/storm:1.0.0 "bin/storm ui -c stor"   5 
> minutes ago   Up 5 minutes   0.0.0.0:8080->8080/tcp 
> stormdocker_ui_1
> 5a9101dc251031z4/storm:1.0.0 "bin/storm supervisor"   5 
> minutes ago   Up 5 minutes  
> stormdocker_supervisor_1
> 4d954096cf1831z4/storm:1.0.0 "bin/storm nimbus -c "   5 
> minutes ago   Up 5 minutes   0.0.0.0:6627->6627/tcp 
> stormdocker_nimbus_1
> 070080342c4f31z4/storm:1.0.0 "bin/storm logviewer "   5 
> minutes ago   Up 5 minutes   0.0.0.0:8000->8000/tcp 
> stormdocker_logviewer_1
> 8650786a13ccjplock/zookeeper:3.4.8   "/opt/zookeeper/bin/z"   5 
> minutes ago   Up 5 minutes   2181/tcp, 2888/tcp, 3888/tcp   
> stormdocker_zookeeper_1
> {code}
> And here is what the Storm UI requests
> {code}
> curl 
> 'http://5a9101dc2510:8000/search/topology-1-1462284216%2F6701%2Fworker.log?search-string=split=1'
>  -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: 
> http://192.168.99.100:8080/search_result.html?search=split=topology-1-1462284216=1'
>  -H 'Origin: http://192.168.99.100:8080' -H 'User-Agent: Mozilla/5.0 
> (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) 
> Chrome/50.0.2661.86 Safari/537.36' --compressed
> {code}
> I guess adding an ability to explicitly specify the logviewer host in the 
> storm.yaml would solve the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1758) Distributed log search doesn't work in dockerized environment

[Elisey Zanko (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268834#comment-15268834
 ] 

Elisey Zanko commented on STORM-1758:
-

It works. Thanks!

> Distributed log search doesn't work in dockerized environment
> -
>
> Key: STORM-1758
> URL: https://issues.apache.org/jira/browse/STORM-1758
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-ui
>Affects Versions: 1.0.0
>Reporter: Elisey Zanko
>
> I run the Storm using the following docker-compose.yml
> {code}
> version: '2'
> services:
> zookeeper:
> image: jplock/zookeeper:3.4.8
> restart: always
> nimbus:
> image: 31z4/storm:1.0.0
> command: nimbus -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - zookeeper
> restart: always
> ports:
> - 6627:6627
> volumes:
> - logs:/logs
> supervisor:
> image: 31z4/storm:1.0.0
> command: supervisor -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> restart: always
> volumes:
> - logs:/logs
> logviewer:
> image: 31z4/storm:1.0.0
> command: logviewer -c storm.log.dir="/logs"
> restart: always
> ports:
> - 8000:8000
> volumes:
> - logs:/logs
> ui:
> image: 31z4/storm:1.0.0
> command: ui -c storm.log.dir="/logs" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> - logviewer
> restart: always
> ports:
> - 8080:8080
> volumes:
> - logs:/log
> volumes:
> logs: {}
> {code}
> And distributed log search doesn't work because the Storm UI tries to access 
> the logviewer by supervisor's container id as a host.
> Here is the list of running containers
> {code}
> $ docker ps
> 7ae118eef55c31z4/storm:1.0.0 "bin/storm ui -c stor"   5 
> minutes ago   Up 5 minutes   0.0.0.0:8080->8080/tcp 
> stormdocker_ui_1
> 5a9101dc251031z4/storm:1.0.0 "bin/storm supervisor"   5 
> minutes ago   Up 5 minutes  
> stormdocker_supervisor_1
> 4d954096cf1831z4/storm:1.0.0 "bin/storm nimbus -c "   5 
> minutes ago   Up 5 minutes   0.0.0.0:6627->6627/tcp 
> stormdocker_nimbus_1
> 070080342c4f31z4/storm:1.0.0 "bin/storm logviewer "   5 
> minutes ago   Up 5 minutes   0.0.0.0:8000->8000/tcp 
> stormdocker_logviewer_1
> 8650786a13ccjplock/zookeeper:3.4.8   "/opt/zookeeper/bin/z"   5 
> minutes ago   Up 5 minutes   2181/tcp, 2888/tcp, 3888/tcp   
> stormdocker_zookeeper_1
> {code}
> And here is what the Storm UI requests
> {code}
> curl 
> 'http://5a9101dc2510:8000/search/topology-1-1462284216%2F6701%2Fworker.log?search-string=split=1'
>  -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: 
> http://192.168.99.100:8080/search_result.html?search=split=topology-1-1462284216=1'
>  -H 'Origin: http://192.168.99.100:8080' -H 'User-Agent: Mozilla/5.0 
> (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) 
> Chrome/50.0.2661.86 Safari/537.36' --compressed
> {code}
> I guess adding an ability to explicitly specify the logviewer host in the 
> storm.yaml would solve the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1758) Distributed log search doesn't work in dockerized environment

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268793#comment-15268793
 ] 

Jungtaek Lim commented on STORM-1758:
-

Could you add 'storm.local.hostname: ' to storm.yaml for each 
node and see it helps?

> Distributed log search doesn't work in dockerized environment
> -
>
> Key: STORM-1758
> URL: https://issues.apache.org/jira/browse/STORM-1758
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-ui
>Affects Versions: 1.0.0
>Reporter: Elisey Zanko
>
> I run the Storm using the following docker-compose.yml
> {code}
> version: '2'
> services:
> zookeeper:
> image: jplock/zookeeper:3.4.8
> restart: always
> nimbus:
> image: 31z4/storm:1.0.0
> command: nimbus -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - zookeeper
> restart: always
> ports:
> - 6627:6627
> volumes:
> - logs:/logs
> supervisor:
> image: 31z4/storm:1.0.0
> command: supervisor -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> restart: always
> volumes:
> - logs:/logs
> logviewer:
> image: 31z4/storm:1.0.0
> command: logviewer -c storm.log.dir="/logs"
> restart: always
> ports:
> - 8000:8000
> volumes:
> - logs:/logs
> ui:
> image: 31z4/storm:1.0.0
> command: ui -c storm.log.dir="/logs" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> - logviewer
> restart: always
> ports:
> - 8080:8080
> volumes:
> - logs:/log
> volumes:
> logs: {}
> {code}
> And distributed log search doesn't work because the Storm UI tries to access 
> the logviewer by supervisor's container id as a host.
> Here is the list of running containers
> {code}
> $ docker ps
> 7ae118eef55c31z4/storm:1.0.0 "bin/storm ui -c stor"   5 
> minutes ago   Up 5 minutes   0.0.0.0:8080->8080/tcp 
> stormdocker_ui_1
> 5a9101dc251031z4/storm:1.0.0 "bin/storm supervisor"   5 
> minutes ago   Up 5 minutes  
> stormdocker_supervisor_1
> 4d954096cf1831z4/storm:1.0.0 "bin/storm nimbus -c "   5 
> minutes ago   Up 5 minutes   0.0.0.0:6627->6627/tcp 
> stormdocker_nimbus_1
> 070080342c4f31z4/storm:1.0.0 "bin/storm logviewer "   5 
> minutes ago   Up 5 minutes   0.0.0.0:8000->8000/tcp 
> stormdocker_logviewer_1
> 8650786a13ccjplock/zookeeper:3.4.8   "/opt/zookeeper/bin/z"   5 
> minutes ago   Up 5 minutes   2181/tcp, 2888/tcp, 3888/tcp   
> stormdocker_zookeeper_1
> {code}
> And here is what the Storm UI requests
> {code}
> curl 
> 'http://5a9101dc2510:8000/search/topology-1-1462284216%2F6701%2Fworker.log?search-string=split=1'
>  -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: 
> http://192.168.99.100:8080/search_result.html?search=split=topology-1-1462284216=1'
>  -H 'Origin: http://192.168.99.100:8080' -H 'User-Agent: Mozilla/5.0 
> (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) 
> Chrome/50.0.2661.86 Safari/537.36' --compressed
> {code}
> I guess adding an ability to explicitly specify the logviewer host in the 
> storm.yaml would solve the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1759) Viewing logs from the Storm UI doesn't work in dockerized environment

[Elisey Zanko (JIRA)]

Elisey Zanko created STORM-1759:
---

 Summary: Viewing logs from the Storm UI doesn't work in dockerized 
environment
 Key: STORM-1759
 URL: https://issues.apache.org/jira/browse/STORM-1759
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-ui
Affects Versions: 0.10.0, 1.0.0, 0.9.6
Reporter: Elisey Zanko


I run the Storm using the following docker-compose.yml

{code}
version: '2'

services:
zookeeper:
image: jplock/zookeeper:3.4.8
restart: always

nimbus:
image: 31z4/storm:1.0.0
command: nimbus -c storm.log.dir="/logs" -c 
storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
depends_on:
- zookeeper
restart: always
ports:
- 6627:6627
volumes:
- logs:/logs

supervisor:
image: 31z4/storm:1.0.0
command: supervisor -c storm.log.dir="/logs" -c 
storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
depends_on:
- nimbus
restart: always
volumes:
- logs:/logs

logviewer:
image: 31z4/storm:1.0.0
command: logviewer -c storm.log.dir="/logs"
restart: always
ports:
- 8000:8000
volumes:
- logs:/logs

ui:
image: 31z4/storm:1.0.0
command: ui -c storm.log.dir="/logs" -c nimbus.host="nimbus"
depends_on:
- nimbus
- logviewer
restart: always
ports:
- 8080:8080
volumes:
- logs:/log

volumes:
logs: {}
{code}

And opening the logs from the Storm UI doesn't work because all links are 
pointing to different container ids as hosts.

I guess adding an ability to explicitly specify the logviewer host in the 
storm.yaml would solve the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (STORM-1758) Distributed log search doesn't work in dockerized environment

[Elisey Zanko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elisey Zanko updated STORM-1758:

Comment: was deleted

(was: That would work.)

> Distributed log search doesn't work in dockerized environment
> -
>
> Key: STORM-1758
> URL: https://issues.apache.org/jira/browse/STORM-1758
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-ui
>Affects Versions: 1.0.0
>Reporter: Elisey Zanko
>
> I run the Storm using the following docker-compose.yml
> {code}
> version: '2'
> services:
> zookeeper:
> image: jplock/zookeeper:3.4.8
> restart: always
> nimbus:
> image: 31z4/storm:1.0.0
> command: nimbus -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - zookeeper
> restart: always
> ports:
> - 6627:6627
> volumes:
> - logs:/logs
> supervisor:
> image: 31z4/storm:1.0.0
> command: supervisor -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> restart: always
> volumes:
> - logs:/logs
> logviewer:
> image: 31z4/storm:1.0.0
> command: logviewer -c storm.log.dir="/logs"
> restart: always
> ports:
> - 8000:8000
> volumes:
> - logs:/logs
> ui:
> image: 31z4/storm:1.0.0
> command: ui -c storm.log.dir="/logs" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> - logviewer
> restart: always
> ports:
> - 8080:8080
> volumes:
> - logs:/log
> volumes:
> logs: {}
> {code}
> And distributed log search doesn't work because the Storm UI tries to access 
> the logviewer by supervisor's container id as a host.
> Here is the list of running containers
> {code}
> $ docker ps
> 7ae118eef55c31z4/storm:1.0.0 "bin/storm ui -c stor"   5 
> minutes ago   Up 5 minutes   0.0.0.0:8080->8080/tcp 
> stormdocker_ui_1
> 5a9101dc251031z4/storm:1.0.0 "bin/storm supervisor"   5 
> minutes ago   Up 5 minutes  
> stormdocker_supervisor_1
> 4d954096cf1831z4/storm:1.0.0 "bin/storm nimbus -c "   5 
> minutes ago   Up 5 minutes   0.0.0.0:6627->6627/tcp 
> stormdocker_nimbus_1
> 070080342c4f31z4/storm:1.0.0 "bin/storm logviewer "   5 
> minutes ago   Up 5 minutes   0.0.0.0:8000->8000/tcp 
> stormdocker_logviewer_1
> 8650786a13ccjplock/zookeeper:3.4.8   "/opt/zookeeper/bin/z"   5 
> minutes ago   Up 5 minutes   2181/tcp, 2888/tcp, 3888/tcp   
> stormdocker_zookeeper_1
> {code}
> And here is what the Storm UI requests
> {code}
> curl 
> 'http://5a9101dc2510:8000/search/topology-1-1462284216%2F6701%2Fworker.log?search-string=split=1'
>  -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: 
> http://192.168.99.100:8080/search_result.html?search=split=topology-1-1462284216=1'
>  -H 'Origin: http://192.168.99.100:8080' -H 'User-Agent: Mozilla/5.0 
> (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) 
> Chrome/50.0.2661.86 Safari/537.36' --compressed
> {code}
> I guess adding an ability to explicitly specify the logviewer host in the 
> storm.yaml would solve the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1758) Distributed log search doesn't work in dockerized environment

[Elisey Zanko (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268773#comment-15268773
 ] 

Elisey Zanko commented on STORM-1758:
-

That would work.

> Distributed log search doesn't work in dockerized environment
> -
>
> Key: STORM-1758
> URL: https://issues.apache.org/jira/browse/STORM-1758
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-ui
>Affects Versions: 1.0.0
>Reporter: Elisey Zanko
>
> I run the Storm using the following docker-compose.yml
> {code}
> version: '2'
> services:
> zookeeper:
> image: jplock/zookeeper:3.4.8
> restart: always
> nimbus:
> image: 31z4/storm:1.0.0
> command: nimbus -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - zookeeper
> restart: always
> ports:
> - 6627:6627
> volumes:
> - logs:/logs
> supervisor:
> image: 31z4/storm:1.0.0
> command: supervisor -c storm.log.dir="/logs" -c 
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> restart: always
> volumes:
> - logs:/logs
> logviewer:
> image: 31z4/storm:1.0.0
> command: logviewer -c storm.log.dir="/logs"
> restart: always
> ports:
> - 8000:8000
> volumes:
> - logs:/logs
> ui:
> image: 31z4/storm:1.0.0
> command: ui -c storm.log.dir="/logs" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> - logviewer
> restart: always
> ports:
> - 8080:8080
> volumes:
> - logs:/log
> volumes:
> logs: {}
> {code}
> And distributed log search doesn't work because the Storm UI tries to access 
> the logviewer by supervisor's container id as a host.
> Here is the list of running containers
> {code}
> $ docker ps
> 7ae118eef55c31z4/storm:1.0.0 "bin/storm ui -c stor"   5 
> minutes ago   Up 5 minutes   0.0.0.0:8080->8080/tcp 
> stormdocker_ui_1
> 5a9101dc251031z4/storm:1.0.0 "bin/storm supervisor"   5 
> minutes ago   Up 5 minutes  
> stormdocker_supervisor_1
> 4d954096cf1831z4/storm:1.0.0 "bin/storm nimbus -c "   5 
> minutes ago   Up 5 minutes   0.0.0.0:6627->6627/tcp 
> stormdocker_nimbus_1
> 070080342c4f31z4/storm:1.0.0 "bin/storm logviewer "   5 
> minutes ago   Up 5 minutes   0.0.0.0:8000->8000/tcp 
> stormdocker_logviewer_1
> 8650786a13ccjplock/zookeeper:3.4.8   "/opt/zookeeper/bin/z"   5 
> minutes ago   Up 5 minutes   2181/tcp, 2888/tcp, 3888/tcp   
> stormdocker_zookeeper_1
> {code}
> And here is what the Storm UI requests
> {code}
> curl 
> 'http://5a9101dc2510:8000/search/topology-1-1462284216%2F6701%2Fworker.log?search-string=split=1'
>  -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: 
> http://192.168.99.100:8080/search_result.html?search=split=topology-1-1462284216=1'
>  -H 'Origin: http://192.168.99.100:8080' -H 'User-Agent: Mozilla/5.0 
> (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) 
> Chrome/50.0.2661.86 Safari/537.36' --compressed
> {code}
> I guess adding an ability to explicitly specify the logviewer host in the 
> storm.yaml would solve the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1758) Distributed log search doesn't work in dockerized environment

[Elisey Zanko (JIRA)]

Elisey Zanko created STORM-1758:
---

 Summary: Distributed log search doesn't work in dockerized 
environment
 Key: STORM-1758
 URL: https://issues.apache.org/jira/browse/STORM-1758
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-ui
Affects Versions: 1.0.0
Reporter: Elisey Zanko


I run the Storm using the following docker-compose.yml

{code}
version: '2'

services:
zookeeper:
image: jplock/zookeeper:3.4.8
restart: always

nimbus:
image: 31z4/storm:1.0.0
command: nimbus -c storm.log.dir="/logs" -c 
storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
depends_on:
- zookeeper
restart: always
ports:
- 6627:6627
volumes:
- logs:/logs

supervisor:
image: 31z4/storm:1.0.0
command: supervisor -c storm.log.dir="/logs" -c 
storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
depends_on:
- nimbus
restart: always
volumes:
- logs:/logs

logviewer:
image: 31z4/storm:1.0.0
command: logviewer -c storm.log.dir="/logs"
restart: always
ports:
- 8000:8000
volumes:
- logs:/logs

ui:
image: 31z4/storm:1.0.0
command: ui -c storm.log.dir="/logs" -c nimbus.host="nimbus"
depends_on:
- nimbus
- logviewer
restart: always
ports:
- 8080:8080
volumes:
- logs:/log

volumes:
logs: {}
{code}

And distributed log search doesn't work because the Storm UI tries to access 
the logviewer by supervisor's container id as a host.

Here is the list of running containers
{code}
$ docker ps
7ae118eef55c31z4/storm:1.0.0 "bin/storm ui -c stor"   5 minutes 
ago   Up 5 minutes   0.0.0.0:8080->8080/tcp 
stormdocker_ui_1
5a9101dc251031z4/storm:1.0.0 "bin/storm supervisor"   5 minutes 
ago   Up 5 minutes  
stormdocker_supervisor_1
4d954096cf1831z4/storm:1.0.0 "bin/storm nimbus -c "   5 minutes 
ago   Up 5 minutes   0.0.0.0:6627->6627/tcp 
stormdocker_nimbus_1
070080342c4f31z4/storm:1.0.0 "bin/storm logviewer "   5 minutes 
ago   Up 5 minutes   0.0.0.0:8000->8000/tcp 
stormdocker_logviewer_1
8650786a13ccjplock/zookeeper:3.4.8   "/opt/zookeeper/bin/z"   5 minutes 
ago   Up 5 minutes   2181/tcp, 2888/tcp, 3888/tcp   
stormdocker_zookeeper_1
{code}

And here is what the Storm UI requests
{code}
curl 
'http://5a9101dc2510:8000/search/topology-1-1462284216%2F6701%2Fworker.log?search-string=split=1'
 -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: 
http://192.168.99.100:8080/search_result.html?search=split=topology-1-1462284216=1'
 -H 'Origin: http://192.168.99.100:8080' -H 'User-Agent: Mozilla/5.0 
(Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/50.0.2661.86 Safari/537.36' --compressed
{code}

I guess adding an ability to explicitly specify the logviewer host in the 
storm.yaml would solve the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1646) Intermittent test failures in storm-kafka unit tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268756#comment-15268756
 ] 

ASF GitHub Bot commented on STORM-1646:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216535715
  
Issue is that some users report that storm-kafka 1.0.0 with 0.9 is not 
working properly.

Please refer https://issues.apache.org/jira/browse/STORM-1755 and #1386.
When we adjust the version into 0.8.x, that option becomes useless for test 
codes, which could incur failures which seems hard to find.

But using `metadata.fetch.timeout.ms` wouldn't make any issues for Kafka 
0.8 and 0.9. And I don't think storm-kafka can support Kafka same or upper than 
0.10.0 clearly so no issue for using deprecated property. But this is just my 2 
cents and I'd like to hear about other opinions, too.


> Intermittent test failures in storm-kafka unit tests
> 
>
> Key: STORM-1646
> URL: https://issues.apache.org/jira/browse/STORM-1646
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> We have been seeing intermittent test failures in KafkaUtilsTest on slow 
> hardware and lightly resourced VMs, as well as an intermittent race condition 
> when running the storm-kafka ExponentialBackoffManager  unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1646] Fix Kafka unit tests

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216535715
  
Issue is that some users report that storm-kafka 1.0.0 with 0.9 is not 
working properly.

Please refer https://issues.apache.org/jira/browse/STORM-1755 and #1386.
When we adjust the version into 0.8.x, that option becomes useless for test 
codes, which could incur failures which seems hard to find.

But using `metadata.fetch.timeout.ms` wouldn't make any issues for Kafka 
0.8 and 0.9. And I don't think storm-kafka can support Kafka same or upper than 
0.10.0 clearly so no issue for using deprecated property. But this is just my 2 
cents and I'd like to hear about other opinions, too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1646) Intermittent test failures in storm-kafka unit tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268721#comment-15268721
 ] 

ASF GitHub Bot commented on STORM-1646:
---

Github user ppoulosk commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216529805
  
@HeartSaVioR,

I assumed that because storm-kafka already builds against kafka 0.9 and 
that the changes to max.block.ms were only in unit test code it would not 
affect any user compatibility.  Is that wrong?


> Intermittent test failures in storm-kafka unit tests
> 
>
> Key: STORM-1646
> URL: https://issues.apache.org/jira/browse/STORM-1646
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> We have been seeing intermittent test failures in KafkaUtilsTest on slow 
> hardware and lightly resourced VMs, as well as an intermittent race condition 
> when running the storm-kafka ExponentialBackoffManager  unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1646] Fix Kafka unit tests

[ppoulosk]

Github user ppoulosk commented on the pull request:

https://github.com/apache/storm/pull/1335#issuecomment-216529805
  
@HeartSaVioR,

I assumed that because storm-kafka already builds against kafka 0.9 and 
that the changes to max.block.ms were only in unit test code it would not 
affect any user compatibility.  Is that wrong?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1707) Improve supervisor latency by removing 2-min wait

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268709#comment-15268709
 ] 

ASF GitHub Bot commented on STORM-1707:
---

Github user ppoulosk commented on the pull request:

https://github.com/apache/storm/pull/1370#issuecomment-216527257
  
Will do.  Thanks.


> Improve supervisor latency by removing 2-min wait
> -
>
> Key: STORM-1707
> URL: https://issues.apache.org/jira/browse/STORM-1707
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> After launching workers, the supervisor waits up to 2 minutes synchronously 
> for the workers to be "launched".
> We should remove this, and instead keep track of launch time, making the 
> "killer" function smart enough to determine the difference between a worker 
> that's still launching, one that's timed out, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1707] Remove two minute timeout after w...

[ppoulosk]

Github user ppoulosk commented on the pull request:

https://github.com/apache/storm/pull/1370#issuecomment-216527257
  
Will do.  Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1750 (0.10.x): Ensure worker dies when r...

[srdo]

GitHub user srdo opened a pull request:

https://github.com/apache/storm/pull/1390

STORM-1750 (0.10.x): Ensure worker dies when report-error-and-die is …

…called. Make cluster set-data try setting data if node creation fails 
because the node exists.

Backport of STORM-1750

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1750-0.10.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1390.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1390


commit aaf3abf1132db1f22789bd58875367ac75216e37
Author: Stig Rohde Døssing 
Date:   2016-05-03T13:04:27Z

STORM-1750 (0.10.x): Ensure worker dies when report-error-and-die is 
called. Make cluster set-data try setting data if node creation fails because 
the node exists.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1756) Storm-kafka tests leak resources due to retained references to KafkaServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268593#comment-15268593
 ] 

ASF GitHub Bot commented on STORM-1756:
---

GitHub user srdo reopened a pull request:

https://github.com/apache/storm/pull/1387

STORM-1756: Explicitly null KafkaServer reference in KafkaTestBroker …

…to prevent out of memory on large test classes.

I'm not clear on whether JUnit keeps the reference for the lifetime of the 
module test, or whether they're released after all tests in a class have run. 
Either way, a test class with a KafkaTestBroker field and many tests would 
likely still run out of memory.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1756

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1387.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1387


commit 0f760c20f2a5d1376d20f1dcb259bdb000cc1d35
Author: Stig Rohde Døssing 
Date:   2016-05-02T13:07:07Z

STORM-1756: Explicitly null KafkaServer reference in KafkaTestBroker to 
prevent out of memory on large test classes




> Storm-kafka tests leak resources due to retained references to KafkaServer
> --
>
> Key: STORM-1756
> URL: https://issues.apache.org/jira/browse/STORM-1756
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>
> Since JUnit keeps a reference to each test class until all tests for a module 
> have run, the KafkaTestBroker fields are not eligible for garbage collection 
> until all tests have run. KafkaTestBroker keeps a reference to 
> KafkaServerStartable/KafkaServer, which allocates resources that are also not 
> eligible for collection. This very quickly causes surefire to run out of 
> memory if this class is used more than a few places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1756) Storm-kafka tests leak resources due to retained references to KafkaServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268592#comment-15268592
 ] 

ASF GitHub Bot commented on STORM-1756:
---

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1387#issuecomment-216508276
  
I did see OOME due to KafkaServer resources. 8x 130MB byte arrays caused 
the VM to run out of memory. As I said, I can't reproduce it. It's probably not 
an issue on master when storm-kafka gets downgraded to 0.8.2.1 (here 
https://github.com/apache/storm/pull/1386), since one of the changes in Kafka 
0.9.0.1 was to enable the log cleaner thread by default (and thus allocate some 
very large byte arrays when starting a server).




> Storm-kafka tests leak resources due to retained references to KafkaServer
> --
>
> Key: STORM-1756
> URL: https://issues.apache.org/jira/browse/STORM-1756
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>
> Since JUnit keeps a reference to each test class until all tests for a module 
> have run, the KafkaTestBroker fields are not eligible for garbage collection 
> until all tests have run. KafkaTestBroker keeps a reference to 
> KafkaServerStartable/KafkaServer, which allocates resources that are also not 
> eligible for collection. This very quickly causes surefire to run out of 
> memory if this class is used more than a few places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1756: Explicitly null KafkaServer refere...

[srdo]

GitHub user srdo reopened a pull request:

https://github.com/apache/storm/pull/1387

STORM-1756: Explicitly null KafkaServer reference in KafkaTestBroker …

…to prevent out of memory on large test classes.

I'm not clear on whether JUnit keeps the reference for the lifetime of the 
module test, or whether they're released after all tests in a class have run. 
Either way, a test class with a KafkaTestBroker field and many tests would 
likely still run out of memory.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1756

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1387.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1387


commit 0f760c20f2a5d1376d20f1dcb259bdb000cc1d35
Author: Stig Rohde Døssing 
Date:   2016-05-02T13:07:07Z

STORM-1756: Explicitly null KafkaServer reference in KafkaTestBroker to 
prevent out of memory on large test classes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1756: Explicitly null KafkaServer refere...

[srdo]

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1387#issuecomment-216508276
  
I did see OOME due to KafkaServer resources. 8x 130MB byte arrays caused 
the VM to run out of memory. As I said, I can't reproduce it. It's probably not 
an issue on master when storm-kafka gets downgraded to 0.8.2.1 (here 
https://github.com/apache/storm/pull/1386), since one of the changes in Kafka 
0.9.0.1 was to enable the log cleaner thread by default (and thus allocate some 
very large byte arrays when starting a server).




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1750) Report-error-and-die may not kill the worker

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268575#comment-15268575
 ] 

ASF GitHub Bot commented on STORM-1750:
---

GitHub user srdo opened a pull request:

https://github.com/apache/storm/pull/1389

STORM-1750: Ensure worker dies when report-error-and-die is called. M…

…ake zookeeper_state_factory set-data try setting data if node creation 
fails because the node exists.

Backport of STORM-1750

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1750-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1389.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1389


commit 6e4bdbc8cacd7621c570126a5d9e8da9f599d037
Author: Stig Rohde Døssing 
Date:   2016-05-03T11:51:13Z

STORM-1750: Ensure worker dies when report-error-and-die is called. Make 
zookeeper_state_factory set-data try setting data if node creation fails 
because the node exists




> Report-error-and-die may not kill the worker
> 
>
> Key: STORM-1750
> URL: https://issues.apache.org/jira/browse/STORM-1750
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Critical
>
> The report-error-and-die function in executor.clj calls report-error, which 
> can throw exceptions if Curator runs into any kind of trouble while 
> registering the error. I suspect this may happen with network errors, but it 
> can also happen if two executors for the same component throw exceptions at 
> the same time and no errors have been registered for the component 
> previously. This is because both calls to report-error-and-die update the 
> lastErrorPath, and ZkStateStorage set_data doesn't catch the potential 
> NodeExistsException that may be thrown from the create call.
> If an exception is thrown from report-error, the suicide-fn is never called, 
> and the worker keeps running sans the crashed executor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1750: Ensure worker dies when report-err...

[srdo]

GitHub user srdo opened a pull request:

https://github.com/apache/storm/pull/1389

STORM-1750: Ensure worker dies when report-error-and-die is called. M…

…ake zookeeper_state_factory set-data try setting data if node creation 
fails because the node exists.

Backport of STORM-1750

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1750-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1389.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1389


commit 6e4bdbc8cacd7621c570126a5d9e8da9f599d037
Author: Stig Rohde Døssing 
Date:   2016-05-03T11:51:13Z

STORM-1750: Ensure worker dies when report-error-and-die is called. Make 
zookeeper_state_factory set-data try setting data if node creation fails 
because the node exists




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1756) Storm-kafka tests leak resources due to retained references to KafkaServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268574#comment-15268574
 ] 

ASF GitHub Bot commented on STORM-1756:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1387#issuecomment-216505586
  
Please reopen this issue if you saw intermittent test failure. What I asked 
is that it's a theory or it occurred some issues.


> Storm-kafka tests leak resources due to retained references to KafkaServer
> --
>
> Key: STORM-1756
> URL: https://issues.apache.org/jira/browse/STORM-1756
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>
> Since JUnit keeps a reference to each test class until all tests for a module 
> have run, the KafkaTestBroker fields are not eligible for garbage collection 
> until all tests have run. KafkaTestBroker keeps a reference to 
> KafkaServerStartable/KafkaServer, which allocates resources that are also not 
> eligible for collection. This very quickly causes surefire to run out of 
> memory if this class is used more than a few places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1756: Explicitly null KafkaServer refere...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1387#issuecomment-216505586
  
Please reopen this issue if you saw intermittent test failure. What I asked 
is that it's a theory or it occurred some issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1745) Add partition to PartitionManager logs where it's missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268528#comment-15268528
 ] 

ASF GitHub Bot commented on STORM-1745:
---

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1380#issuecomment-216496822
  
@HeartSaVioR Sure. The current logs look like this "Starting Kafka 
10.4.5.1:0 from offset 15520", which doesn't tell you which topic it's 
affecting. The logs in L259 and L269 don't tell you which partition/topic 
they're affecting at all. This PR changes the log to use the toString on 
Partition for logging, which looks like this: "Starting Kafka 10.4.5.3 
Partition{host=10.4.5.3:9092, topic=dce-data, partition=26} from offset 1612424"


> Add partition to PartitionManager logs where it's missing
> -
>
> Key: STORM-1745
> URL: https://issues.apache.org/jira/browse/STORM-1745
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1745: Add partition to log output in Par...

[srdo]

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1380#issuecomment-216496822
  
@HeartSaVioR Sure. The current logs look like this "Starting Kafka 
10.4.5.1:0 from offset 15520", which doesn't tell you which topic it's 
affecting. The logs in L259 and L269 don't tell you which partition/topic 
they're affecting at all. This PR changes the log to use the toString on 
Partition for logging, which looks like this: "Starting Kafka 10.4.5.3 
Partition{host=10.4.5.3:9092, topic=dce-data, partition=26} from offset 1612424"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-1756) Storm-kafka tests leak resources due to retained references to KafkaServer

[JIRA]

 [ 
https://issues.apache.org/jira/browse/STORM-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stig Rohde Døssing resolved STORM-1756.
---
Resolution: Cannot Reproduce

Tried setting up some test classes with KafkaTestBroker fields and initializing 
them without nulling them in tearDown. Could not reproduce.

> Storm-kafka tests leak resources due to retained references to KafkaServer
> --
>
> Key: STORM-1756
> URL: https://issues.apache.org/jira/browse/STORM-1756
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>
> Since JUnit keeps a reference to each test class until all tests for a module 
> have run, the KafkaTestBroker fields are not eligible for garbage collection 
> until all tests have run. KafkaTestBroker keeps a reference to 
> KafkaServerStartable/KafkaServer, which allocates resources that are also not 
> eligible for collection. This very quickly causes surefire to run out of 
> memory if this class is used more than a few places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1735) Nimbus logs that replication was not reached when min-replication-count was reached exactly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268514#comment-15268514
 ] 

ASF GitHub Bot commented on STORM-1735:
---

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1369#issuecomment-216493707
  
@HeartSaVioR Sure, please do :)


> Nimbus logs that replication was not reached when min-replication-count was 
> reached exactly
> ---
>
> Key: STORM-1735
> URL: https://issues.apache.org/jira/browse/STORM-1735
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Minor
>
> When waiting for replication during topology submission, Nimbus logs whether 
> replication succeeded within the timeout or not. The check for whether to log 
> that it timed out or succeeded is off by one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1735: Nimbus should log that replication...

[srdo]

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1369#issuecomment-216493707
  
@HeartSaVioR Sure, please do :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1756) Storm-kafka tests leak resources due to retained references to KafkaServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268491#comment-15268491
 ] 

ASF GitHub Bot commented on STORM-1756:
---

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1387#issuecomment-216490426
  
If this turns out to be a problem later, disabling the log cleaner thread 
during tests is probably a better fix. The tests don't make logs long enough to 
be compacted anyway.


> Storm-kafka tests leak resources due to retained references to KafkaServer
> --
>
> Key: STORM-1756
> URL: https://issues.apache.org/jira/browse/STORM-1756
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>
> Since JUnit keeps a reference to each test class until all tests for a module 
> have run, the KafkaTestBroker fields are not eligible for garbage collection 
> until all tests have run. KafkaTestBroker keeps a reference to 
> KafkaServerStartable/KafkaServer, which allocates resources that are also not 
> eligible for collection. This very quickly causes surefire to run out of 
> memory if this class is used more than a few places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1756: Explicitly null KafkaServer refere...

[srdo]

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1387#issuecomment-216490426
  
If this turns out to be a problem later, disabling the log cleaner thread 
during tests is probably a better fix. The tests don't make logs long enough to 
be compacted anyway.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1745) Add partition to PartitionManager logs where it's missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268480#comment-15268480
 ] 

ASF GitHub Bot commented on STORM-1745:
---

GitHub user srdo reopened a pull request:

https://github.com/apache/storm/pull/1380

STORM-1745: Add partition to log output in PartitionManager

Adding topic name and partition id to the log output makes it easier to 
debug when an issue arises.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1745

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1380.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1380


commit 39ff8a3cdbecf9b6a20ebf2dd86c53f5a4064478
Author: Stig Rohde Døssing 
Date:   2016-04-29T09:05:55Z

STORM-1745: Add partition to log output in PartitionManager




> Add partition to PartitionManager logs where it's missing
> -
>
> Key: STORM-1745
> URL: https://issues.apache.org/jira/browse/STORM-1745
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1745) Add partition to PartitionManager logs where it's missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268479#comment-15268479
 ] 

ASF GitHub Bot commented on STORM-1745:
---

Github user srdo closed the pull request at:

https://github.com/apache/storm/pull/1380


> Add partition to PartitionManager logs where it's missing
> -
>
> Key: STORM-1745
> URL: https://issues.apache.org/jira/browse/STORM-1745
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1745: Add partition to log output in Par...

[srdo]

GitHub user srdo reopened a pull request:

https://github.com/apache/storm/pull/1380

STORM-1745: Add partition to log output in PartitionManager

Adding topic name and partition id to the log output makes it easier to 
debug when an issue arises.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1745

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1380.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1380


commit 39ff8a3cdbecf9b6a20ebf2dd86c53f5a4064478
Author: Stig Rohde Døssing 
Date:   2016-04-29T09:05:55Z

STORM-1745: Add partition to log output in PartitionManager




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1745: Add partition to log output in Par...

[srdo]

Github user srdo closed the pull request at:

https://github.com/apache/storm/pull/1380


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---