[GitHub] storm issue #1565: STORM-1970: external project examples refator

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/1565
  
hi @darionyaphet  sorry for the less information.
this PR address @abhishekagarwal87 's topic from dev@ mailing list.

>Right now the example topologies/classes for some external modules are 
being put up in the test folder itself. The problem I see is that,
-> the example code isn't really test code so test folder isn't the right 
fit.
-> people, who are looking for example code, may not find the example code.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1536: Storm 1890 ensure we refetch static resources after packa...

[abellina]

Github user abellina commented on the issue:

https://github.com/apache/storm/pull/1536
  
No problem. Thanks @HeartSaVioR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1536: Storm 1890 ensure we refetch static resources after packa...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1536
  
@abellina I don't have enough knowledge to front-end tech. But at a glance, 
I think concept is good.
@knusbaum @kishorvpatil Please go ahead reviewing & merging.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1536: Storm 1890 ensure we refetch static resources after packa...

[abellina]

Github user abellina commented on the issue:

https://github.com/apache/storm/pull/1536
  
@HeartSaVioR, could you take a look at this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378670#comment-15378670
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1560
  
I also addressed review comments and also RAT issue to 1.x PR. 
Please take a look at #1561 as well. Thanks!


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1560: STORM-1966 Expand metric having Map type as value into mu...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1560
  
I also addressed review comments and also RAT issue to 1.x PR. 
Please take a look at #1561 as well. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378621#comment-15378621
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1560
  
Thanks @HeartSaVioR . +1.


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1560: STORM-1966 Expand metric having Map type as value into mu...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1560
  
Thanks @HeartSaVioR . +1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1561: STORM-1966 Expand metric having Map type as value into mu...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1561
  
@abhishekagarwal87 @harshach @ptgoetz Please review this as well. Thanks in 
advance!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378609#comment-15378609
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1561
  
@abhishekagarwal87 @harshach @ptgoetz Please review this as well. Thanks in 
advance!


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1560: STORM-1966 Expand metric having Map type as value into mu...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1560
  
@harshach Addressed.

I found RAT issue again (my bad) and fixed it.

Btw, I didn't address review comments to PR for 1.x-branch. Once I'm done 
I'll mention reviewers  to review from PR for 1.x-branch as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378583#comment-15378583
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1560
  
@harshach Addressed.

I found RAT issue again (my bad) and fixed it.

Btw, I didn't address review comments to PR for 1.x-branch. Once I'm done 
I'll mention reviewers  to review from PR for 1.x-branch as well.


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1910) One topology can't use hdfs spout to read from two locations

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-1910:

Issue Type: Improvement  (was: Bug)

> One topology can't use hdfs spout to read from two locations
> 
>
> Key: STORM-1910
> URL: https://issues.apache.org/jira/browse/STORM-1910
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hdfs
>Affects Versions: 1.0.1
>Reporter: Raghav Kumar Gautam
>Assignee: Roshan Naik
> Fix For: 2.0.0, 1.1.0
>
>
> The hdfs uri is passed using config:
> {code}
> conf.put(Configs.HDFS_URI, hdfsUri);
> {code}
> I see two problems with this approach:
> 1. If someone wants to used two hdfsUri in same or different spouts - then 
> that does not seem feasible.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/examples/storm-starter/src/jvm/storm/starter/HdfsSpoutTopology.java#L117-L117
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L331-L331
> {code}
> if ( !conf.containsKey(Configs.SOURCE_DIR) ) {
>   LOG.error(Configs.SOURCE_DIR + " setting is required");
>   throw new RuntimeException(Configs.SOURCE_DIR + " setting is required");
> }
> this.sourceDirPath = new Path( conf.get(Configs.SOURCE_DIR).toString() );
> {code}
> 2. It does not fail fast i.e. at the time of topology submissing. We can fail 
> fast if the hdfs path is invalid or credentials/permissions are not ok. Such 
> errors at this time can only be detected at runtime by looking at the worker 
> logs.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L297-L297



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Request for reviewing

[Jungtaek Lim]

I thought STORM-1910  is
likely an improvement, not bugfix. That is why I included this to Storm
1.1.0 epic.
I'd like to see others' opinions as well. If we think it's a bugfix we can
include it to 1.0.2.

What do all of you think about this?

2016년 7월 15일 (금) 오전 4:43, Roshan Naik 님이 작성:

> STORM-1910
> -roshan
>
>
> On 7/14/16, 8:59 AM, "Jungtaek Lim"  wrote:
>
> >We seems close to make all issues for 1.0.2 epic resolved soon.
> >
> >Epic link: https://issues.apache.org/jira/browse/STORM-1855
> >All of open issues are in progress of reviewing. Most of them got +1 and
> >is
> >expected to be merged in soon.
> >
> >Is anyone having other critical or blocker issues which need to be
> >included
> >to 1.0.2?
> >
> >2016년 7월 9일 (토) 오전 12:34, P. Taylor Goetz 님이 작성:
> >
> >> Thanks Juntaek, I finally got some time to review it.
> >>
> >> Regarding the 1.0.2 release, there are a few open/in-progress issues
> >> remaining, but I think we should be in a position to cut a new RC
> >>sometime
> >> next week.
> >>
> >> -Taylor
> >>
> >>
> >> > On Jul 8, 2016, at 7:43 AM, Jungtaek Lim  wrote:
> >> >
> >> > Hi devs,
> >> >
> >> > I've a pull request for a critical issue, and the patch modifies the
> >> heart
> >> > of supervisor code, so I'd like to make my patch get reviewed by more
> >> > people in community.
> >> >
> >> > STORM-1934: Race condition between sync-supervisor and sync-processes
> >> > raises several strange issues
> >> > 
> >> > Discussion is on this PR: https://github.com/apache/storm/pull/1528
> >> >
> >> > Btw, since Storm 1.0.2 has lots of bugfix marked as 'Critical' and
> >> 'Major'
> >> > which are reported by users several times, I'd like to see 1.0.2
> >> > official release
> >> > sooner.
> >> >
> >> >
> >>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20Storm%20AND%20
> >>%22epic%20link%22%20%3D%20STORM-1855%20order%20by%20priority
> >> >
> >> > There're 4 unresolved issues but pull requests are available for
> >> 'Critical'
> >> > issues so once we resolve them I guess we could get other issues out
> >>of
> >> > 1.0.2 and start working for releasing.
> >> >
> >> > Given that STORM-1934 has two binding +1 votes, I'd like to wait for
> >> > several days, and merge if no further discussion is in progress.
> >> >
> >> > Thanks in advance for taking your time.
> >> >
> >> > Best regards,
> >> > Jungtaek Lim (HeartSaVioR)
> >>
> >>
>
>

[jira] [Commented] (STORM-1910) One topology can't use hdfs spout to read from two locations

[Roshan Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378435#comment-15378435
 ] 

Roshan Naik commented on STORM-1910:


[~ptgoetz] should this be marked for 1.0.2 as well ?

> One topology can't use hdfs spout to read from two locations
> 
>
> Key: STORM-1910
> URL: https://issues.apache.org/jira/browse/STORM-1910
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hdfs
>Affects Versions: 1.0.1
>Reporter: Raghav Kumar Gautam
>Assignee: Roshan Naik
> Fix For: 2.0.0, 1.1.0
>
>
> The hdfs uri is passed using config:
> {code}
> conf.put(Configs.HDFS_URI, hdfsUri);
> {code}
> I see two problems with this approach:
> 1. If someone wants to used two hdfsUri in same or different spouts - then 
> that does not seem feasible.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/examples/storm-starter/src/jvm/storm/starter/HdfsSpoutTopology.java#L117-L117
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L331-L331
> {code}
> if ( !conf.containsKey(Configs.SOURCE_DIR) ) {
>   LOG.error(Configs.SOURCE_DIR + " setting is required");
>   throw new RuntimeException(Configs.SOURCE_DIR + " setting is required");
> }
> this.sourceDirPath = new Path( conf.get(Configs.SOURCE_DIR).toString() );
> {code}
> 2. It does not fail fast i.e. at the time of topology submissing. We can fail 
> fast if the hdfs path is invalid or credentials/permissions are not ok. Such 
> errors at this time can only be detected at runtime by looking at the worker 
> logs.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L297-L297



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1910) One topology can't use hdfs spout to read from two locations

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1910.

   Resolution: Fixed
Fix Version/s: 2.0.0

Merged to master and 1.x-branch

> One topology can't use hdfs spout to read from two locations
> 
>
> Key: STORM-1910
> URL: https://issues.apache.org/jira/browse/STORM-1910
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hdfs
>Affects Versions: 1.0.1
>Reporter: Raghav Kumar Gautam
>Assignee: Roshan Naik
> Fix For: 2.0.0, 1.1.0
>
>
> The hdfs uri is passed using config:
> {code}
> conf.put(Configs.HDFS_URI, hdfsUri);
> {code}
> I see two problems with this approach:
> 1. If someone wants to used two hdfsUri in same or different spouts - then 
> that does not seem feasible.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/examples/storm-starter/src/jvm/storm/starter/HdfsSpoutTopology.java#L117-L117
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L331-L331
> {code}
> if ( !conf.containsKey(Configs.SOURCE_DIR) ) {
>   LOG.error(Configs.SOURCE_DIR + " setting is required");
>   throw new RuntimeException(Configs.SOURCE_DIR + " setting is required");
> }
> this.sourceDirPath = new Path( conf.get(Configs.SOURCE_DIR).toString() );
> {code}
> 2. It does not fail fast i.e. at the time of topology submissing. We can fail 
> fast if the hdfs path is invalid or credentials/permissions are not ok. Such 
> errors at this time can only be detected at runtime by looking at the worker 
> logs.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L297-L297



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1558: STORM-1910 One topology cannot use hdfs spout to r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1558


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1910) One topology can't use hdfs spout to read from two locations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378311#comment-15378311
 ] 

ASF GitHub Bot commented on STORM-1910:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1558


> One topology can't use hdfs spout to read from two locations
> 
>
> Key: STORM-1910
> URL: https://issues.apache.org/jira/browse/STORM-1910
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hdfs
>Affects Versions: 1.0.1
>Reporter: Raghav Kumar Gautam
>Assignee: Roshan Naik
> Fix For: 1.1.0
>
>
> The hdfs uri is passed using config:
> {code}
> conf.put(Configs.HDFS_URI, hdfsUri);
> {code}
> I see two problems with this approach:
> 1. If someone wants to used two hdfsUri in same or different spouts - then 
> that does not seem feasible.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/examples/storm-starter/src/jvm/storm/starter/HdfsSpoutTopology.java#L117-L117
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L331-L331
> {code}
> if ( !conf.containsKey(Configs.SOURCE_DIR) ) {
>   LOG.error(Configs.SOURCE_DIR + " setting is required");
>   throw new RuntimeException(Configs.SOURCE_DIR + " setting is required");
> }
> this.sourceDirPath = new Path( conf.get(Configs.SOURCE_DIR).toString() );
> {code}
> 2. It does not fail fast i.e. at the time of topology submissing. We can fail 
> fast if the hdfs path is invalid or credentials/permissions are not ok. Such 
> errors at this time can only be detected at runtime by looking at the worker 
> logs.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L297-L297



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1910) One topology can't use hdfs spout to read from two locations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378283#comment-15378283
 ] 

ASF GitHub Bot commented on STORM-1910:
---

Github user ptgoetz commented on the issue:

https://github.com/apache/storm/pull/1558
  
+1


> One topology can't use hdfs spout to read from two locations
> 
>
> Key: STORM-1910
> URL: https://issues.apache.org/jira/browse/STORM-1910
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hdfs
>Affects Versions: 1.0.1
>Reporter: Raghav Kumar Gautam
>Assignee: Roshan Naik
> Fix For: 1.1.0
>
>
> The hdfs uri is passed using config:
> {code}
> conf.put(Configs.HDFS_URI, hdfsUri);
> {code}
> I see two problems with this approach:
> 1. If someone wants to used two hdfsUri in same or different spouts - then 
> that does not seem feasible.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/examples/storm-starter/src/jvm/storm/starter/HdfsSpoutTopology.java#L117-L117
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L331-L331
> {code}
> if ( !conf.containsKey(Configs.SOURCE_DIR) ) {
>   LOG.error(Configs.SOURCE_DIR + " setting is required");
>   throw new RuntimeException(Configs.SOURCE_DIR + " setting is required");
> }
> this.sourceDirPath = new Path( conf.get(Configs.SOURCE_DIR).toString() );
> {code}
> 2. It does not fail fast i.e. at the time of topology submissing. We can fail 
> fast if the hdfs path is invalid or credentials/permissions are not ok. Such 
> errors at this time can only be detected at runtime by looking at the worker 
> logs.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L297-L297



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1558: STORM-1910 One topology cannot use hdfs spout to read fro...

[ptgoetz]

Github user ptgoetz commented on the issue:

https://github.com/apache/storm/pull/1558
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Request for reviewing

[Roshan Naik]

STORM-1910 
-roshan


On 7/14/16, 8:59 AM, "Jungtaek Lim"  wrote:

>We seems close to make all issues for 1.0.2 epic resolved soon.
>
>Epic link: https://issues.apache.org/jira/browse/STORM-1855
>All of open issues are in progress of reviewing. Most of them got +1 and
>is
>expected to be merged in soon.
>
>Is anyone having other critical or blocker issues which need to be
>included
>to 1.0.2?
>
>2016년 7월 9일 (토) 오전 12:34, P. Taylor Goetz 님이 작성:
>
>> Thanks Juntaek, I finally got some time to review it.
>>
>> Regarding the 1.0.2 release, there are a few open/in-progress issues
>> remaining, but I think we should be in a position to cut a new RC
>>sometime
>> next week.
>>
>> -Taylor
>>
>>
>> > On Jul 8, 2016, at 7:43 AM, Jungtaek Lim  wrote:
>> >
>> > Hi devs,
>> >
>> > I've a pull request for a critical issue, and the patch modifies the
>> heart
>> > of supervisor code, so I'd like to make my patch get reviewed by more
>> > people in community.
>> >
>> > STORM-1934: Race condition between sync-supervisor and sync-processes
>> > raises several strange issues
>> > 
>> > Discussion is on this PR: https://github.com/apache/storm/pull/1528
>> >
>> > Btw, since Storm 1.0.2 has lots of bugfix marked as 'Critical' and
>> 'Major'
>> > which are reported by users several times, I'd like to see 1.0.2
>> > official release
>> > sooner.
>> >
>> >
>> 
>>https://issues.apache.org/jira/issues/?jql=project%20%3D%20Storm%20AND%20
>>%22epic%20link%22%20%3D%20STORM-1855%20order%20by%20priority
>> >
>> > There're 4 unresolved issues but pull requests are available for
>> 'Critical'
>> > issues so once we resolve them I guess we could get other issues out
>>of
>> > 1.0.2 and start working for releasing.
>> >
>> > Given that STORM-1934 has two binding +1 votes, I'd like to wait for
>> > several days, and merge if no further discussion is in progress.
>> >
>> > Thanks in advance for taking your time.
>> >
>> > Best regards,
>> > Jungtaek Lim (HeartSaVioR)
>>
>>

[jira] [Commented] (STORM-1737) storm-kafka-client has compilation errors with Apache Kafka 0.10

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378057#comment-15378057
 ] 

ASF GitHub Bot commented on STORM-1737:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1556
  
+1.


> storm-kafka-client has compilation errors with Apache Kafka 0.10
> 
>
> Key: STORM-1737
> URL: https://issues.apache.org/jira/browse/STORM-1737
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Hugo Louro
>Priority: Blocker
>
> when compiled with Apache Kafka 0.10 branch getting following errors
> {code}
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[163,51]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[166,45]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[175,51]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[177,45]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[252,41]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1556: STORM-1737: storm-kafka-client has compilation errors wit...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1556
  
+1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378056#comment-15378056
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1560
  
@HeartSaVioR looks good to me. Just a minor nitpick +1 after that.


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1560: STORM-1966 Expand metric having Map type as value into mu...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1560
  
@HeartSaVioR looks good to me. Just a minor nitpick +1 after that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378040#comment-15378040
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user harshach commented on a diff in the pull request:

https://github.com/apache/storm/pull/1560#discussion_r70856636
  
--- Diff: storm-core/src/jvm/org/apache/storm/daemon/StormCommon.java ---
@@ -391,9 +388,13 @@ public static void addEventLogger(Map conf, 
StormTopology topology) {
 List whitelist = (List) 
info.get("whitelist");
 List blacklist = (List) 
info.get("blacklist");
 FilterByMetricName filterPredicate = new 
FilterByMetricName(whitelist, blacklist);
+Boolean expandMapType = 
Utils.getBoolean(info.get("expandMapType"), false);
--- End diff --

can we make these strings into some final String constants


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1560: STORM-1966 Expand metric having Map type as value ...

[harshach]

Github user harshach commented on a diff in the pull request:

https://github.com/apache/storm/pull/1560#discussion_r70856636
  
--- Diff: storm-core/src/jvm/org/apache/storm/daemon/StormCommon.java ---
@@ -391,9 +388,13 @@ public static void addEventLogger(Map conf, 
StormTopology topology) {
 List whitelist = (List) 
info.get("whitelist");
 List blacklist = (List) 
info.get("blacklist");
 FilterByMetricName filterPredicate = new 
FilterByMetricName(whitelist, blacklist);
+Boolean expandMapType = 
Utils.getBoolean(info.get("expandMapType"), false);
--- End diff --

can we make these strings into some final String constants


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1969) Modify HiveTopology to show usage of non-partition table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377900#comment-15377900
 ] 

ASF GitHub Bot commented on STORM-1969:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1564
  
+1


> Modify HiveTopology to show usage of non-partition table
> 
>
> Key: STORM-1969
> URL: https://issues.apache.org/jira/browse/STORM-1969
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hive
>Reporter: Raghav Kumar Gautam
>Assignee: Jungtaek Lim
>Priority: Minor
>
> There're some kinds of topology in storm-hive, but all of them uses partition 
> so we don't have example for accessing non-partition table. It would be 
> better to have one for automated tests.
> Instead of creating a new topology, we can modify HiveTopology to not use 
> partition at all. BucketTestHiveTopology and TridentHiveTopology also use 
> DelimitedRecordHiveMapper.withTimeAsPartitionField() so I think usage of 
> withTimeAsPartitionField() is covered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1564: STORM-1969 Modify HiveTopology to show usage of non-parti...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1564
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1930) Kafka New Client API - Support for Topic Wildcards

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377887#comment-15377887
 ] 

ASF GitHub Bot commented on STORM-1930:
---

Github user nabhanelrahman commented on the issue:

https://github.com/apache/storm/pull/1518
  
@hmcl, just following up; it this PR ready to merge?



> Kafka New Client API - Support for Topic Wildcards
> --
>
> Key: STORM-1930
> URL: https://issues.apache.org/jira/browse/STORM-1930
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-kafka
>Affects Versions: 1.0.2
>Reporter: Hugo Louro
>Assignee: Hugo Louro
>Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1518: re-submitting STORM-1930: Kafka New Client API - Support ...

[nabhanelrahman]

Github user nabhanelrahman commented on the issue:

https://github.com/apache/storm/pull/1518
  
@hmcl, just following up; it this PR ready to merge?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (STORM-1968) Storm logviewer does not work for nimbus.log in secure cluster

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz reassigned STORM-1968:
--

Assignee: Arun Mahadevan  (was: P. Taylor Goetz)

> Storm logviewer does not work for nimbus.log in secure cluster
> --
>
> Key: STORM-1968
> URL: https://issues.apache.org/jira/browse/STORM-1968
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> logviewer invokes "get-log-user-group-whitelist" which tries to get the 
> worker metadata file by invoking "get-log-metadata-file". In the case of 
> nimbus.log clojure-from-yaml-file  returns nil and the authorization fails.
> Modify clojure-from-yaml-file to return an empty map in case of failures so 
> that the authorization can continue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1968) Storm logviewer does not work for nimbus.log in secure cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377248#comment-15377248
 ] 

ASF GitHub Bot commented on STORM-1968:
---

Github user ptgoetz commented on the issue:

https://github.com/apache/storm/pull/1563
  
+1


> Storm logviewer does not work for nimbus.log in secure cluster
> --
>
> Key: STORM-1968
> URL: https://issues.apache.org/jira/browse/STORM-1968
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> logviewer invokes "get-log-user-group-whitelist" which tries to get the 
> worker metadata file by invoking "get-log-metadata-file". In the case of 
> nimbus.log clojure-from-yaml-file  returns nil and the authorization fails.
> Modify clojure-from-yaml-file to return an empty map in case of failures so 
> that the authorization can continue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1563: [STORM-1968] Storm logviewer does not work for nimbus.log...

[ptgoetz]

Github user ptgoetz commented on the issue:

https://github.com/apache/storm/pull/1563
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377244#comment-15377244
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user ptgoetz commented on the issue:

https://github.com/apache/storm/pull/1560
  
+1


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1560: STORM-1966 Expand metric having Map type as value into mu...

[ptgoetz]

Github user ptgoetz commented on the issue:

https://github.com/apache/storm/pull/1560
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1932) Change Duration's value data type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377216#comment-15377216
 ] 

ASF GitHub Bot commented on STORM-1932:
---

Github user darionyaphet commented on a diff in the pull request:

https://github.com/apache/storm/pull/1527#discussion_r70835839
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/trident/windowing/config/WindowConfig.java 
---
@@ -31,13 +31,13 @@
  * Returns the length of the window.
  * @return
  */
-public int getWindowLength();
+public long getWindowLength();
--- End diff --

using generics to decide return value's date type is a good idea ?


> Change Duration's value data type 
> --
>
> Key: STORM-1932
> URL: https://issues.apache.org/jira/browse/STORM-1932
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> BaseWindowedBolt Duration value is integer now . Use long to express time 
> interval maybe better



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1527: [STORM-1932] Change Duration's value data type

[darionyaphet]

Github user darionyaphet commented on a diff in the pull request:

https://github.com/apache/storm/pull/1527#discussion_r70835839
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/trident/windowing/config/WindowConfig.java 
---
@@ -31,13 +31,13 @@
  * Returns the length of the window.
  * @return
  */
-public int getWindowLength();
+public long getWindowLength();
--- End diff --

using generics to decide return value's date type is a good idea ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1932) Change Duration's value data type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377193#comment-15377193
 ] 

ASF GitHub Bot commented on STORM-1932:
---

Github user darionyaphet commented on a diff in the pull request:

https://github.com/apache/storm/pull/1527#discussion_r70834283
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/windowing/CountEvictionPolicy.java ---
@@ -26,10 +26,10 @@
  * @param  the type of event tracked by this policy.
  */
 public class CountEvictionPolicy implements EvictionPolicy {
-private final int threshold;
+private final long threshold;
--- End diff --

the `CountWindow` and `DurationWindow` both extend from WindowConfig and 
`windowLength` and `slideLength` are Long type.


> Change Duration's value data type 
> --
>
> Key: STORM-1932
> URL: https://issues.apache.org/jira/browse/STORM-1932
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> BaseWindowedBolt Duration value is integer now . Use long to express time 
> interval maybe better



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1527: [STORM-1932] Change Duration's value data type

[darionyaphet]

Github user darionyaphet commented on a diff in the pull request:

https://github.com/apache/storm/pull/1527#discussion_r70834283
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/windowing/CountEvictionPolicy.java ---
@@ -26,10 +26,10 @@
  * @param  the type of event tracked by this policy.
  */
 public class CountEvictionPolicy implements EvictionPolicy {
-private final int threshold;
+private final long threshold;
--- End diff --

the `CountWindow` and `DurationWindow` both extend from WindowConfig and 
`windowLength` and `slideLength` are Long type.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Request for reviewing

[Jungtaek Lim]

We seems close to make all issues for 1.0.2 epic resolved soon.

Epic link: https://issues.apache.org/jira/browse/STORM-1855
All of open issues are in progress of reviewing. Most of them got +1 and is
expected to be merged in soon.

Is anyone having other critical or blocker issues which need to be included
to 1.0.2?

2016년 7월 9일 (토) 오전 12:34, P. Taylor Goetz 님이 작성:

> Thanks Juntaek, I finally got some time to review it.
>
> Regarding the 1.0.2 release, there are a few open/in-progress issues
> remaining, but I think we should be in a position to cut a new RC sometime
> next week.
>
> -Taylor
>
>
> > On Jul 8, 2016, at 7:43 AM, Jungtaek Lim  wrote:
> >
> > Hi devs,
> >
> > I've a pull request for a critical issue, and the patch modifies the
> heart
> > of supervisor code, so I'd like to make my patch get reviewed by more
> > people in community.
> >
> > STORM-1934: Race condition between sync-supervisor and sync-processes
> > raises several strange issues
> > 
> > Discussion is on this PR: https://github.com/apache/storm/pull/1528
> >
> > Btw, since Storm 1.0.2 has lots of bugfix marked as 'Critical' and
> 'Major'
> > which are reported by users several times, I'd like to see 1.0.2
> > official release
> > sooner.
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20Storm%20AND%20%22epic%20link%22%20%3D%20STORM-1855%20order%20by%20priority
> >
> > There're 4 unresolved issues but pull requests are available for
> 'Critical'
> > issues so once we resolve them I guess we could get other issues out of
> > 1.0.2 and start working for releasing.
> >
> > Given that STORM-1934 has two binding +1 votes, I'd like to wait for
> > several days, and merge if no further discussion is in progress.
> >
> > Thanks in advance for taking your time.
> >
> > Best regards,
> > Jungtaek Lim (HeartSaVioR)
>
>

[GitHub] storm pull request #1527: [STORM-1932] Change Duration's value data type

[darionyaphet]

Github user darionyaphet commented on a diff in the pull request:

https://github.com/apache/storm/pull/1527#discussion_r70832297
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java ---
@@ -66,10 +66,14 @@ public String toString() {
  * Holds a Time duration for time based windows and sliding intervals.
  */
 public static class Duration {
-public final int value;
+public final long value;
 
 public Duration(int value, TimeUnit timeUnit) {
-this.value = (int) timeUnit.toMillis(value);
+this.value = timeUnit.toMillis(value);
+}
+
+public Duration(long value) {
--- End diff --

actually `Duration(long value)` is represent the time interval in 
milliseconds, so it doesn't time unit to covert . 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1932) Change Duration's value data type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377166#comment-15377166
 ] 

ASF GitHub Bot commented on STORM-1932:
---

Github user darionyaphet commented on a diff in the pull request:

https://github.com/apache/storm/pull/1527#discussion_r70832297
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java ---
@@ -66,10 +66,14 @@ public String toString() {
  * Holds a Time duration for time based windows and sliding intervals.
  */
 public static class Duration {
-public final int value;
+public final long value;
 
 public Duration(int value, TimeUnit timeUnit) {
-this.value = (int) timeUnit.toMillis(value);
+this.value = timeUnit.toMillis(value);
+}
+
+public Duration(long value) {
--- End diff --

actually `Duration(long value)` is represent the time interval in 
milliseconds, so it doesn't time unit to covert . 


> Change Duration's value data type 
> --
>
> Key: STORM-1932
> URL: https://issues.apache.org/jira/browse/STORM-1932
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> BaseWindowedBolt Duration value is integer now . Use long to express time 
> interval maybe better



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1527: [STORM-1932] Change Duration's value data type

[darionyaphet]

Github user darionyaphet commented on a diff in the pull request:

https://github.com/apache/storm/pull/1527#discussion_r70831175
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java ---
@@ -76,7 +76,7 @@ public WindowedBoltExecutor(IWindowedBolt bolt) {
 this.bolt = bolt;
 }
 
-private int getTopologyTimeoutMillis(Map stormConf) {
+private long getTopologyTimeoutMillis(Map stormConf) {
--- End diff --

finally , the return value is `timeout * 1000` , it have convert to 
millisecond . Long is a better choose . 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1932) Change Duration's value data type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377152#comment-15377152
 ] 

ASF GitHub Bot commented on STORM-1932:
---

Github user darionyaphet commented on a diff in the pull request:

https://github.com/apache/storm/pull/1527#discussion_r70831175
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java ---
@@ -76,7 +76,7 @@ public WindowedBoltExecutor(IWindowedBolt bolt) {
 this.bolt = bolt;
 }
 
-private int getTopologyTimeoutMillis(Map stormConf) {
+private long getTopologyTimeoutMillis(Map stormConf) {
--- End diff --

finally , the return value is `timeout * 1000` , it have convert to 
millisecond . Long is a better choose . 


> Change Duration's value data type 
> --
>
> Key: STORM-1932
> URL: https://issues.apache.org/jira/browse/STORM-1932
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> BaseWindowedBolt Duration value is integer now . Use long to express time 
> interval maybe better



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Purpose/intent of multilang-javascript module?

[Marc Zbyszynski]

Hi There!

I'm wondering what the intent of the multilang-javascript module and the
storm.js file is (
https://github.com/apache/storm/tree/master/storm-multilang/javascrip). Is
this intended as a reference or example implementation of a node.js/storm
integration? Or are you intending storm.js to be the canonical node.js
multilang wrapper that everyone should use for real/production storm
projects that use the multilang protocol to communicate with node.js bolts
and spouts?

The reason I ask is that my team is starting to look at adding storm
topologies to a large node.js project, and we are trying to decide whether
to use the storm.js file included in the storm distribution as our starting
point, or use a 3rd party library or roll our own.

I haven't found any 3rd party libraries that seem actively maintained. The
storm.js file looks good, but seems to only expose a limited number of
features (no support for bolts that do not ack automatically, no hooks for
tick tuples, etc.). We'd be happy to submit some PRs to add this
functionality, and add the metadata files to make it a real node.js package.

Is that something you would want to add to storm-multilang? Or is that
beyond the scope of what you intended the multilang-javascript module to be?

Let me know what you think!

Thanks,

Marc

[jira] [Commented] (STORM-1932) Change Duration's value data type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377145#comment-15377145
 ] 

ASF GitHub Bot commented on STORM-1932:
---

Github user darionyaphet commented on the issue:

https://github.com/apache/storm/pull/1527
  
@arunmahadevan why don't have `@isLong` ? millisecond using it to decorate 
is better 


> Change Duration's value data type 
> --
>
> Key: STORM-1932
> URL: https://issues.apache.org/jira/browse/STORM-1932
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> BaseWindowedBolt Duration value is integer now . Use long to express time 
> interval maybe better



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1527: [STORM-1932] Change Duration's value data type

[darionyaphet]

Github user darionyaphet commented on the issue:

https://github.com/apache/storm/pull/1527
  
@arunmahadevan why don't have `@isLong` ? millisecond using it to decorate 
is better 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1971) HDFS Timed Synchronous Policy

[darion yaphet (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377109#comment-15377109
 ] 

darion yaphet commented on STORM-1971:
--

Hi , I checked default.yaml and found topology.tick.tuple.freq.secs is null . 
So if I don't specify this item in config the tick tuple will not send to bolts 
.

> HDFS Timed Synchronous Policy
> -
>
> Key: STORM-1971
> URL: https://issues.apache.org/jira/browse/STORM-1971
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hdfs
>Affects Versions: 0.10.0, 1.0.0
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> When the data need to be wrote to HDFS is not very large in quantity . We 
> need a timed synchronous policy to flush cached date into HDFS periodically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1737) storm-kafka-client has compilation errors with Apache Kafka 0.10

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377113#comment-15377113
 ] 

ASF GitHub Bot commented on STORM-1737:
---

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/1556
  
@harshach @HeartSaVioR @abhishekagarwal87, thank you for your feedback. The 
updated ReadME, including the added Wildcard Support will be out in a few mins.


> storm-kafka-client has compilation errors with Apache Kafka 0.10
> 
>
> Key: STORM-1737
> URL: https://issues.apache.org/jira/browse/STORM-1737
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Hugo Louro
>Priority: Blocker
>
> when compiled with Apache Kafka 0.10 branch getting following errors
> {code}
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[163,51]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[166,45]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[175,51]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[177,45]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[252,41]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1556: STORM-1737: storm-kafka-client has compilation errors wit...

[hmcl]

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/1556
  
@harshach @HeartSaVioR @abhishekagarwal87, thank you for your feedback. The 
updated ReadME, including the added Wildcard Support will be out in a few mins.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1968) Storm logviewer does not work for nimbus.log in secure cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377106#comment-15377106
 ] 

ASF GitHub Bot commented on STORM-1968:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1563
  
+1


> Storm logviewer does not work for nimbus.log in secure cluster
> --
>
> Key: STORM-1968
> URL: https://issues.apache.org/jira/browse/STORM-1968
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> logviewer invokes "get-log-user-group-whitelist" which tries to get the 
> worker metadata file by invoking "get-log-metadata-file". In the case of 
> nimbus.log clojure-from-yaml-file  returns nil and the authorization fails.
> Modify clojure-from-yaml-file to return an empty map in case of failures so 
> that the authorization can continue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1563: [STORM-1968] Storm logviewer does not work for nimbus.log...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1563
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1193) supervisor-test error creating symlinks on windows

[Paul Milliken (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377063#comment-15377063
 ] 

Paul Milliken commented on STORM-1193:
--

This is occurring in our development environment. We have many developers each 
starting and stopping their own isolated storm clusters.

Unfortunately, the nature of the work being done means that they can't really 
share instances. It's also very unlikely (I'd go as far as to say impossible) 
that we'll be able to get the relevant security policies set up on all the 
environments that we would need to.

We were hoping to upgrade to 1.0.1 in order to take advantage of the 
performance improvements that have been put in. Unfortunately, this is likely 
to prevent us from doing that.

> supervisor-test error creating symlinks on windows
> --
>
> Key: STORM-1193
> URL: https://issues.apache.org/jira/browse/STORM-1193
> Project: Apache Storm
>  Issue Type: Bug
> Environment: Windows7
> Java7
>Reporter: Chuck Burgess
>Priority: Minor
>
> The test-worker-launch-command-run-as-user testcase errors rather than fails, 
> due to what seems to be a privilege issue with creating a symlink.
> {quote}
> 5701 \[main] INFO  b.s.d.supervisor - Creating symlinks for worker-id: 
> fake-worker-id storm-id: fake-storm-id to its port artifacts directory
> Uncaught exception, not in assertion.
> expected: nil
> actual: java.nio.file.FileSystemException: 
> C:\path\to\Temp\storm-local8914443830909286033\workers\fake-worker-id\artifacts:
>  A required privilege is not held by the client.
>  at sun.nio.fs.WindowsException.translateToIOException 
> (WindowsException.java:86)
> sun.nio.fs.WindowsException.rethrowAsIOException 
> (WindowsException.java:97)
> sun.nio.fs.WindowsException.rethrowAsIOException 
> (WindowsException.java:102)
> sun.nio.fs.WindowsFileSystemProvider.createSymbolicLink 
> (WindowsFileSystemProvider.java:577)
> java.nio.file.Files.createSymbolicLink (Files.java:994)
> backtype.storm.util$create_symlink_BANG_.invoke (util.clj:604)
> backtype.storm.daemon.supervisor$create_artifacts_link.invoke 
> (supervisor.clj:811)
> backtype.storm.daemon.supervisor/fn (supervisor.clj:898)
> clojure.lang.MultiFn.invoke (MultiFn.java:251)
> backtype.storm.supervisor_test$fn__498$fn__501$fn__502$fn__503.invoke 
> (supervisor_test.clj:472)
> clojure.core$with_redefs_fn.invoke (core.clj:7209)
> backtype.storm.supervisor_test$fn__498$fn__501$fn__502.invoke 
> (supervisor_test.clj:462)
> backtype.storm.supervisor_test$fn__498$fn__501.invoke 
> (supervisor_test.clj:462)
> backtype.storm.supervisor_test/fn (supervisor_test.clj:451)
> {quote}
> I don't know if the testcase itself should be catching the error and 
> presenting it as a failed test, or if the original code should be catching it 
> and reacting.
> What's odd here is that if this is using my own user account and privileges, 
> then it should work, because my account _does_ have permission to create 
> symlinks.
> Tested against master branch (c12e28c829)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1563: [STORM-1968] Storm logviewer does not work for nimbus.log...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1563
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1968) Storm logviewer does not work for nimbus.log in secure cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377070#comment-15377070
 ] 

ASF GitHub Bot commented on STORM-1968:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1563
  
+1


> Storm logviewer does not work for nimbus.log in secure cluster
> --
>
> Key: STORM-1968
> URL: https://issues.apache.org/jira/browse/STORM-1968
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> logviewer invokes "get-log-user-group-whitelist" which tries to get the 
> worker metadata file by invoking "get-log-metadata-file". In the case of 
> nimbus.log clojure-from-yaml-file  returns nil and the authorization fails.
> Modify clojure-from-yaml-file to return an empty map in case of failures so 
> that the authorization can continue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1560: STORM-1966 Expand metric having Map type as value into mu...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1560
  
@harshach Could you take a look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377062#comment-15377062
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1560
  
@harshach Could you take a look?


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1482: STORM-1876: Option to build storm-kafka and storm-...

[hmcl]

Github user hmcl commented on a diff in the pull request:

https://github.com/apache/storm/pull/1482#discussion_r70818175
  
--- Diff: pom.xml ---
@@ -258,6 +258,12 @@
 1.4.0-incubating
 2.6.3
 2.18.1
+
+
+0.8.2.1
+kafka_2.10
+
+
--- End diff --

@abhishekagarwal87 can the property names be a bit more consistent?

A suggestion would be to prefix the properties used by `storm.kafka.client`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1876) Provide option to build storm-kafka against different kafka clients

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377053#comment-15377053
 ] 

ASF GitHub Bot commented on STORM-1876:
---

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/1482
  
+1 besides the neat pick which would be to have the new spout property 
names prefixed by `storm.kafka.client` property names for consistency.


> Provide option to build storm-kafka against different kafka clients
> ---
>
> Key: STORM-1876
> URL: https://issues.apache.org/jira/browse/STORM-1876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: documentation, storm-kafka
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1737) storm-kafka-client has compilation errors with Apache Kafka 0.10

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-1737:

Fix Version/s: (was: 1.0.2)

Removing fix version and adding to 1.0.2 epic.

> storm-kafka-client has compilation errors with Apache Kafka 0.10
> 
>
> Key: STORM-1737
> URL: https://issues.apache.org/jira/browse/STORM-1737
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Hugo Louro
>Priority: Blocker
>
> when compiled with Apache Kafka 0.10 branch getting following errors
> {code}
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[163,51]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[166,45]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[175,51]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[177,45]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> [ERROR] 
> /Users/harsha/code/hwx/storm/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java:[252,41]
>  incompatible types: org.apache.kafka.common.TopicPartition cannot be 
> converted to java.util.Collection
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...

[hmcl]

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/1482
  
+1 besides the neat pick which would be to have the new spout property 
names prefixed by `storm.kafka.client` property names for consistency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1876) Provide option to build storm-kafka against different kafka clients

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377044#comment-15377044
 ] 

ASF GitHub Bot commented on STORM-1876:
---

Github user hmcl commented on a diff in the pull request:

https://github.com/apache/storm/pull/1482#discussion_r70818175
  
--- Diff: pom.xml ---
@@ -258,6 +258,12 @@
 1.4.0-incubating
 2.6.3
 2.18.1
+
+
+0.8.2.1
+kafka_2.10
+
+
--- End diff --

@abhishekagarwal87 can the property names be a bit more consistent?

A suggestion would be to prefix the properties used by `storm.kafka.client`.


> Provide option to build storm-kafka against different kafka clients
> ---
>
> Key: STORM-1876
> URL: https://issues.apache.org/jira/browse/STORM-1876
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: documentation, storm-kafka
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1959) KafkaPartitionOffsetLag.java does not have license

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1959.
-
Resolution: Fixed

> KafkaPartitionOffsetLag.java does not have license
> --
>
> Key: STORM-1959
> URL: https://issues.apache.org/jira/browse/STORM-1959
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka-monitor
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Robert Joseph Evans
>Assignee: Jungtaek Lim
>Priority: Blocker
> Fix For: 2.0.0, 1.1.0
>
>
> RAT is failing  
> external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaPartitionOffsetLag.java
> Looks like this was introduced as a part of STORM-1950



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1973) Using DefaultTopicSelector to replace StaticTopicSelector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377017#comment-15377017
 ] 

ASF GitHub Bot commented on STORM-1973:
---

GitHub user darionyaphet opened a pull request:

https://github.com/apache/storm/pull/1567

[STORM-1973] Using DefaultTopicSelector to replace StaticTopicSelector

[Using DefaultTopicSelector to replace 
StaticTopicSelector](https://issues.apache.org/jira/browse/STORM-1973)

Replace StaticTopicSelector using DefaultTopicSelector , they are the same .

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/darionyaphet/storm STORM-1973

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1567.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1567


commit cc8776d0e2365703a6ef3a4281708ca1822314a6
Author: darionyaphet 
Date:   2016-07-14T14:30:31Z

STORM-1973 : Using DefaultTopicSelector to replace StaticTopicSelector




> Using DefaultTopicSelector to replace StaticTopicSelector
> -
>
> Key: STORM-1973
> URL: https://issues.apache.org/jira/browse/STORM-1973
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-sql
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> Replace StaticTopicSelector using DefaultTopicSelector , they are the same .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1960) Add CORS support to STORM UI Rest api

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1960.
-
   Resolution: Fixed
Fix Version/s: (was: 2.0.0)

> Add CORS support to STORM UI Rest api
> -
>
> Key: STORM-1960
> URL: https://issues.apache.org/jira/browse/STORM-1960
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 1.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1973) Using DefaultTopicSelector to replace StaticTopicSelector

[darion yaphet (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

darion yaphet updated STORM-1973:
-
Summary: Using DefaultTopicSelector to replace StaticTopicSelector  (was: 
Replace StaticTopicSelector using DefaultTopicSelector)

> Using DefaultTopicSelector to replace StaticTopicSelector
> -
>
> Key: STORM-1973
> URL: https://issues.apache.org/jira/browse/STORM-1973
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-sql
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> Replace StaticTopicSelector using DefaultTopicSelector , they are the same .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (STORM-1959) KafkaPartitionOffsetLag.java does not have license

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim reopened STORM-1959:
-

Would like to set epic on this. Reopening. Will resolve issue soon.

> KafkaPartitionOffsetLag.java does not have license
> --
>
> Key: STORM-1959
> URL: https://issues.apache.org/jira/browse/STORM-1959
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka-monitor
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Robert Joseph Evans
>Assignee: Jungtaek Lim
>Priority: Blocker
> Fix For: 2.0.0, 1.1.0
>
>
> RAT is failing  
> external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaPartitionOffsetLag.java
> Looks like this was introduced as a part of STORM-1950



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1567: [STORM-1973] Using DefaultTopicSelector to replace...

[darionyaphet]

GitHub user darionyaphet opened a pull request:

https://github.com/apache/storm/pull/1567

[STORM-1973] Using DefaultTopicSelector to replace StaticTopicSelector

[Using DefaultTopicSelector to replace 
StaticTopicSelector](https://issues.apache.org/jira/browse/STORM-1973)

Replace StaticTopicSelector using DefaultTopicSelector , they are the same .

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/darionyaphet/storm STORM-1973

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1567.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1567


commit cc8776d0e2365703a6ef3a4281708ca1822314a6
Author: darionyaphet 
Date:   2016-07-14T14:30:31Z

STORM-1973 : Using DefaultTopicSelector to replace StaticTopicSelector




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (STORM-1973) Replace StaticTopicSelector using DefaultTopicSelector

[darion yaphet (JIRA)]

darion yaphet created STORM-1973:


 Summary: Replace StaticTopicSelector using DefaultTopicSelector
 Key: STORM-1973
 URL: https://issues.apache.org/jira/browse/STORM-1973
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-sql
Reporter: darion yaphet
Assignee: darion yaphet


Replace StaticTopicSelector using DefaultTopicSelector , they are the same .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376996#comment-15376996
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user abhishekagarwal87 commented on the issue:

https://github.com/apache/storm/pull/1560
  
Thanks @HeartSaVioR . Looks good to me


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1560: STORM-1966 Expand metric having Map type as value into mu...

[abhishekagarwal87]

Github user abhishekagarwal87 commented on the issue:

https://github.com/apache/storm/pull/1560
  
Thanks @HeartSaVioR . Looks good to me


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376989#comment-15376989
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1560
  
Addressed and squashed.


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376953#comment-15376953
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user abhishekagarwal87 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1560#discussion_r70808499
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/MetricsConsumerBolt.java ---
@@ -43,19 +44,21 @@
 OutputCollector _collector;
 Object _registrationArgument;
 private final int _maxRetainMetricTuples;
-private Predicate _filterPredicate;
+private final Predicate _filterPredicate;
+private final DataPointExpander _populator;
 
 private final BlockingQueue _taskQueue;
 private Thread _taskExecuteThread;
 private volatile boolean _running = true;
 
 public MetricsConsumerBolt(String consumerClassName, Object 
registrationArgument, int maxRetainMetricTuples,
-   Predicate 
filterPredicate) {
+   Predicate 
filterPredicate, DataPointExpander populator) {
--- End diff --

Minor nit. populator variable name can be changed as well. 


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376960#comment-15376960
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user abhishekagarwal87 commented on the issue:

https://github.com/apache/storm/pull/1560
  
Minor nit regarding variable name. Other than that +1. 
Please make sure to squash commits before you merge :)


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1560: STORM-1966 Expand metric having Map type as value into mu...

[abhishekagarwal87]

Github user abhishekagarwal87 commented on the issue:

https://github.com/apache/storm/pull/1560
  
Minor nit regarding variable name. Other than that +1. 
Please make sure to squash commits before you merge :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1193) supervisor-test error creating symlinks on windows

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376963#comment-15376963
 ] 

Jungtaek Lim commented on STORM-1193:
-

Blobstore feature which is introduced at 1.0.0 heavily uses symbolic link, 
which works well with well-known Linux distro and OSX, but bothers Windows 
users.

We prepared windows user guide because of that (there's deadlink on website 
which should be fixed soon), so you can refer 
https://github.com/apache/storm/blob/master/docs/windows-users-guide.md on 
this. I guess you're already aware of.
I believe that the account which runs supervisor doesn't need to be 
administrator. If not it's bad of MS (sorry!)

Personally I don't think we have to get rid of usage of symbolic link. It 
should be not easy and need lots of effort.

Btw, supervisor is a daemon which should be launched only one for each node. Do 
multiple users launch supervisors?

> supervisor-test error creating symlinks on windows
> --
>
> Key: STORM-1193
> URL: https://issues.apache.org/jira/browse/STORM-1193
> Project: Apache Storm
>  Issue Type: Bug
> Environment: Windows7
> Java7
>Reporter: Chuck Burgess
>Priority: Minor
>
> The test-worker-launch-command-run-as-user testcase errors rather than fails, 
> due to what seems to be a privilege issue with creating a symlink.
> {quote}
> 5701 \[main] INFO  b.s.d.supervisor - Creating symlinks for worker-id: 
> fake-worker-id storm-id: fake-storm-id to its port artifacts directory
> Uncaught exception, not in assertion.
> expected: nil
> actual: java.nio.file.FileSystemException: 
> C:\path\to\Temp\storm-local8914443830909286033\workers\fake-worker-id\artifacts:
>  A required privilege is not held by the client.
>  at sun.nio.fs.WindowsException.translateToIOException 
> (WindowsException.java:86)
> sun.nio.fs.WindowsException.rethrowAsIOException 
> (WindowsException.java:97)
> sun.nio.fs.WindowsException.rethrowAsIOException 
> (WindowsException.java:102)
> sun.nio.fs.WindowsFileSystemProvider.createSymbolicLink 
> (WindowsFileSystemProvider.java:577)
> java.nio.file.Files.createSymbolicLink (Files.java:994)
> backtype.storm.util$create_symlink_BANG_.invoke (util.clj:604)
> backtype.storm.daemon.supervisor$create_artifacts_link.invoke 
> (supervisor.clj:811)
> backtype.storm.daemon.supervisor/fn (supervisor.clj:898)
> clojure.lang.MultiFn.invoke (MultiFn.java:251)
> backtype.storm.supervisor_test$fn__498$fn__501$fn__502$fn__503.invoke 
> (supervisor_test.clj:472)
> clojure.core$with_redefs_fn.invoke (core.clj:7209)
> backtype.storm.supervisor_test$fn__498$fn__501$fn__502.invoke 
> (supervisor_test.clj:462)
> backtype.storm.supervisor_test$fn__498$fn__501.invoke 
> (supervisor_test.clj:462)
> backtype.storm.supervisor_test/fn (supervisor_test.clj:451)
> {quote}
> I don't know if the testcase itself should be catching the error and 
> presenting it as a failed test, or if the original code should be catching it 
> and reacting.
> What's odd here is that if this is using my own user account and privileges, 
> then it should work, because my account _does_ have permission to create 
> symlinks.
> Tested against master branch (c12e28c829)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376961#comment-15376961
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1560#discussion_r70809388
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/MetricsConsumerBolt.java ---
@@ -43,19 +44,21 @@
 OutputCollector _collector;
 Object _registrationArgument;
 private final int _maxRetainMetricTuples;
-private Predicate _filterPredicate;
+private final Predicate _filterPredicate;
+private final DataPointExpander _populator;
 
 private final BlockingQueue _taskQueue;
 private Thread _taskExecuteThread;
 private volatile boolean _running = true;
 
 public MetricsConsumerBolt(String consumerClassName, Object 
registrationArgument, int maxRetainMetricTuples,
-   Predicate 
filterPredicate) {
+   Predicate 
filterPredicate, DataPointExpander populator) {
--- End diff --

Oh I missed that. Thanks!


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1560: STORM-1966 Expand metric having Map type as value ...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1560#discussion_r70809388
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/MetricsConsumerBolt.java ---
@@ -43,19 +44,21 @@
 OutputCollector _collector;
 Object _registrationArgument;
 private final int _maxRetainMetricTuples;
-private Predicate _filterPredicate;
+private final Predicate _filterPredicate;
+private final DataPointExpander _populator;
 
 private final BlockingQueue _taskQueue;
 private Thread _taskExecuteThread;
 private volatile boolean _running = true;
 
 public MetricsConsumerBolt(String consumerClassName, Object 
registrationArgument, int maxRetainMetricTuples,
-   Predicate 
filterPredicate) {
+   Predicate 
filterPredicate, DataPointExpander populator) {
--- End diff --

Oh I missed that. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1560: STORM-1966 Expand metric having Map type as value ...

[abhishekagarwal87]

Github user abhishekagarwal87 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1560#discussion_r70808499
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/MetricsConsumerBolt.java ---
@@ -43,19 +44,21 @@
 OutputCollector _collector;
 Object _registrationArgument;
 private final int _maxRetainMetricTuples;
-private Predicate _filterPredicate;
+private final Predicate _filterPredicate;
+private final DataPointExpander _populator;
 
 private final BlockingQueue _taskQueue;
 private Thread _taskExecuteThread;
 private volatile boolean _running = true;
 
 public MetricsConsumerBolt(String consumerClassName, Object 
registrationArgument, int maxRetainMetricTuples,
-   Predicate 
filterPredicate) {
+   Predicate 
filterPredicate, DataPointExpander populator) {
--- End diff --

Minor nit. populator variable name can be changed as well. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1560: STORM-1966 Expand metric having Map type as value into mu...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1560
  
Thanks for thoughtful review, @abhishekagarwal87 ! Addressed your review 
comments. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376932#comment-15376932
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1560
  
Thanks for thoughtful review, @abhishekagarwal87 ! Addressed your review 
comments. 


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1193) supervisor-test error creating symlinks on windows

[Paul Milliken (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376930#comment-15376930
 ] 

Paul Milliken commented on STORM-1193:
--

I've just encountered an error which looks very much like this in the 
supervisor itself when trying to deploy when upgrading from Storm 0.10.0 to 
1.0.1.

In our environment, it is not possible for all people who need it to be granted 
the relevant permissions (due to corporate security policies etc.).

I've verified that running Storm as an administrator made it work, but that 
shouldn't be necessary.

> supervisor-test error creating symlinks on windows
> --
>
> Key: STORM-1193
> URL: https://issues.apache.org/jira/browse/STORM-1193
> Project: Apache Storm
>  Issue Type: Bug
> Environment: Windows7
> Java7
>Reporter: Chuck Burgess
>Priority: Minor
>
> The test-worker-launch-command-run-as-user testcase errors rather than fails, 
> due to what seems to be a privilege issue with creating a symlink.
> {quote}
> 5701 \[main] INFO  b.s.d.supervisor - Creating symlinks for worker-id: 
> fake-worker-id storm-id: fake-storm-id to its port artifacts directory
> Uncaught exception, not in assertion.
> expected: nil
> actual: java.nio.file.FileSystemException: 
> C:\path\to\Temp\storm-local8914443830909286033\workers\fake-worker-id\artifacts:
>  A required privilege is not held by the client.
>  at sun.nio.fs.WindowsException.translateToIOException 
> (WindowsException.java:86)
> sun.nio.fs.WindowsException.rethrowAsIOException 
> (WindowsException.java:97)
> sun.nio.fs.WindowsException.rethrowAsIOException 
> (WindowsException.java:102)
> sun.nio.fs.WindowsFileSystemProvider.createSymbolicLink 
> (WindowsFileSystemProvider.java:577)
> java.nio.file.Files.createSymbolicLink (Files.java:994)
> backtype.storm.util$create_symlink_BANG_.invoke (util.clj:604)
> backtype.storm.daemon.supervisor$create_artifacts_link.invoke 
> (supervisor.clj:811)
> backtype.storm.daemon.supervisor/fn (supervisor.clj:898)
> clojure.lang.MultiFn.invoke (MultiFn.java:251)
> backtype.storm.supervisor_test$fn__498$fn__501$fn__502$fn__503.invoke 
> (supervisor_test.clj:472)
> clojure.core$with_redefs_fn.invoke (core.clj:7209)
> backtype.storm.supervisor_test$fn__498$fn__501$fn__502.invoke 
> (supervisor_test.clj:462)
> backtype.storm.supervisor_test$fn__498$fn__501.invoke 
> (supervisor_test.clj:462)
> backtype.storm.supervisor_test/fn (supervisor_test.clj:451)
> {quote}
> I don't know if the testcase itself should be catching the error and 
> presenting it as a failed test, or if the original code should be catching it 
> and reacting.
> What's odd here is that if this is using my own user account and privileges, 
> then it should work, because my account _does_ have permission to create 
> symlinks.
> Tested against master branch (c12e28c829)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1565: STORM-1970: external project examples refator

[darionyaphet]

Github user darionyaphet commented on the issue:

https://github.com/apache/storm/pull/1565
  
What is it doing ? Do you add more desc ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1970) external project examples refator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376924#comment-15376924
 ] 

ASF GitHub Bot commented on STORM-1970:
---

Github user darionyaphet commented on the issue:

https://github.com/apache/storm/pull/1565
  
What is it doing ? Do you add more desc ?


> external project examples refator
> -
>
> Key: STORM-1970
> URL: https://issues.apache.org/jira/browse/STORM-1970
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> refactor example projects:
> *storm-elasticsearch
> *storm-hbase
> *storm-hdfs   
> *storm-hive   
> *storm-jdbc
> *storm-kafka
> *storm-mongodb
> *storm-mqtt
> *storm-opentsdb
> *storm-redis
> *storm-solr



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1971) HDFS Timed Synchronous Policy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376922#comment-15376922
 ] 

ASF GitHub Bot commented on STORM-1971:
---

GitHub user darionyaphet opened a pull request:

https://github.com/apache/storm/pull/1566

[STORM-1971] HDFS Timed Synchronous Policy

[STORM-1971 HDFS Timed Synchronous 
Policy](https://issues.apache.org/jira/browse/STORM-1971)

When the data need to be wrote to HDFS is not very large in quantity . 

We need a timed synchronous policy to flush cached date into HDFS 
periodically.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/darionyaphet/storm STORM-1971

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1566.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1566


commit 86e036a8544040557f6b514e0d27f119c6473727
Author: darionyaphet 
Date:   2016-07-14T13:34:45Z

STORM-1971 : HDFS Timed Synchronous Policy




> HDFS Timed Synchronous Policy
> -
>
> Key: STORM-1971
> URL: https://issues.apache.org/jira/browse/STORM-1971
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hdfs
>Affects Versions: 0.10.0, 1.0.0
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> When the data need to be wrote to HDFS is not very large in quantity . We 
> need a timed synchronous policy to flush cached date into HDFS periodically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1566: [STORM-1971] HDFS Timed Synchronous Policy

[darionyaphet]

GitHub user darionyaphet opened a pull request:

https://github.com/apache/storm/pull/1566

[STORM-1971] HDFS Timed Synchronous Policy

[STORM-1971 HDFS Timed Synchronous 
Policy](https://issues.apache.org/jira/browse/STORM-1971)

When the data need to be wrote to HDFS is not very large in quantity . 

We need a timed synchronous policy to flush cached date into HDFS 
periodically.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/darionyaphet/storm STORM-1971

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1566.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1566


commit 86e036a8544040557f6b514e0d27f119c6473727
Author: darionyaphet 
Date:   2016-07-14T13:34:45Z

STORM-1971 : HDFS Timed Synchronous Policy




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (STORM-1972) Storm throws java.lang.ClassNotFoundException on Bolt class

[Ilya Kholinow (JIRA)]

Ilya Kholinow created STORM-1972:


 Summary: Storm throws java.lang.ClassNotFoundException on Bolt 
class
 Key: STORM-1972
 URL: https://issues.apache.org/jira/browse/STORM-1972
 Project: Apache Storm
  Issue Type: Bug
Affects Versions: 1.0.1
Reporter: Ilya Kholinow


I'm trying to debug very simple topology (1 spout 1 bolt)

public class JoinerTopologyTest {

public static void main(String[] args) throws IOException {
Config conf = new Config();
conf.setNumWorkers(5);
conf.setDebug(true);

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("SPOUT-1",new MySpout(),1);
builder.setBolt("BOLT-1",new Bolt1(), 3)
.shuffleGrouping("SPOUT-1");
//builder.setBolt("JOINER", new JoinerBolt(),1)
//.shuffleGrouping("BOLT-1")
//.shuffleGrouping("SPOUT-1","str1");

final LocalCluster cluster = new LocalCluster();
cluster.submitTopology("TOPO1",conf,builder.createTopology());


System.in.read();

cluster.shutdown();
}
}

But when i run it from InteliJ IDEA i get:

java.lang.RuntimeException: java.lang.ClassNotFoundException: 
com.pixonic.zephyr.compaction.tests.Bolt1 at 
org.apache.storm.utils.Utils.javaDeserialize(Utils.java:181) 
~[storm-core-1.0.1.jar:1.0.1] at 
org.apache.storm.utils.Utils.getSetComponentObject(Utils.java:430) 
~[storm-core-1.0.1.jar:1.0.1]
and

[Thread-15] ERROR o.a.s.d.worker - Error on initialization of server mk-worker 
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.apache.storm.daemon.acker at 
org.apache.storm.utils.Utils.javaDeserialize(Utils.java:181) 
~[storm-core-1.0.1.jar:1.0.1] at 
org.apache.storm.utils.Utils.getSetComponentObject(Utils.java:430) 
~[storm-core-1.0.1.jar:1.0.1]
but same topology runs well in Cluster mode. PS in my pom.xml in debug mode i 
have:


org.apache.storm
storm-core
1.0.1
compile


project to reproduce bug: 
https://github.com/holinov/storm-101-localcluster/tree/master




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1972) Storm throws java.lang.ClassNotFoundException on Bolt class

[Ilya Kholinow (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ilya Kholinow updated STORM-1972:
-
Description: 
I'm trying to debug very simple topology (1 spout 2 bolts)

public class JoinerTopologyTest {

public static void main(String[] args) throws IOException {
Config conf = new Config();
conf.setNumWorkers(5);
conf.setDebug(true);

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("SPOUT-1",new MySpout(),1);
builder.setBolt("BOLT-1",new Bolt1(), 3)
.shuffleGrouping("SPOUT-1");
builder.setBolt("JOINER", new JoinerBolt(),1)
.shuffleGrouping("BOLT-1")
.shuffleGrouping("SPOUT-1","str1");

final LocalCluster cluster = new LocalCluster();
cluster.submitTopology("TOPO1",conf,builder.createTopology());


System.in.read();

cluster.shutdown();
}
}

But when i run it from InteliJ IDEA i get:

java.lang.RuntimeException: java.lang.ClassNotFoundException: 
com.pixonic.zephyr.compaction.tests.Bolt1 at 
org.apache.storm.utils.Utils.javaDeserialize(Utils.java:181) 
~[storm-core-1.0.1.jar:1.0.1] at 
org.apache.storm.utils.Utils.getSetComponentObject(Utils.java:430) 
~[storm-core-1.0.1.jar:1.0.1]
and

[Thread-15] ERROR o.a.s.d.worker - Error on initialization of server mk-worker 
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.apache.storm.daemon.acker at 
org.apache.storm.utils.Utils.javaDeserialize(Utils.java:181) 
~[storm-core-1.0.1.jar:1.0.1] at 
org.apache.storm.utils.Utils.getSetComponentObject(Utils.java:430) 
~[storm-core-1.0.1.jar:1.0.1]
but same topology runs well in Cluster mode. PS in my pom.xml in debug mode i 
have:


org.apache.storm
storm-core
1.0.1


project to reproduce bug: 
https://github.com/holinov/storm-101-localcluster/tree/master


  was:
I'm trying to debug very simple topology (1 spout 1 bolt)

public class JoinerTopologyTest {

public static void main(String[] args) throws IOException {
Config conf = new Config();
conf.setNumWorkers(5);
conf.setDebug(true);

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("SPOUT-1",new MySpout(),1);
builder.setBolt("BOLT-1",new Bolt1(), 3)
.shuffleGrouping("SPOUT-1");
//builder.setBolt("JOINER", new JoinerBolt(),1)
//.shuffleGrouping("BOLT-1")
//.shuffleGrouping("SPOUT-1","str1");

final LocalCluster cluster = new LocalCluster();
cluster.submitTopology("TOPO1",conf,builder.createTopology());


System.in.read();

cluster.shutdown();
}
}

But when i run it from InteliJ IDEA i get:

java.lang.RuntimeException: java.lang.ClassNotFoundException: 
com.pixonic.zephyr.compaction.tests.Bolt1 at 
org.apache.storm.utils.Utils.javaDeserialize(Utils.java:181) 
~[storm-core-1.0.1.jar:1.0.1] at 
org.apache.storm.utils.Utils.getSetComponentObject(Utils.java:430) 
~[storm-core-1.0.1.jar:1.0.1]
and

[Thread-15] ERROR o.a.s.d.worker - Error on initialization of server mk-worker 
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.apache.storm.daemon.acker at 
org.apache.storm.utils.Utils.javaDeserialize(Utils.java:181) 
~[storm-core-1.0.1.jar:1.0.1] at 
org.apache.storm.utils.Utils.getSetComponentObject(Utils.java:430) 
~[storm-core-1.0.1.jar:1.0.1]
but same topology runs well in Cluster mode. PS in my pom.xml in debug mode i 
have:


org.apache.storm
storm-core
1.0.1
compile


project to reproduce bug: 
https://github.com/holinov/storm-101-localcluster/tree/master



> Storm throws java.lang.ClassNotFoundException on Bolt class
> ---
>
> Key: STORM-1972
> URL: https://issues.apache.org/jira/browse/STORM-1972
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.1
>Reporter: Ilya Kholinow
>
> I'm trying to debug very simple topology (1 spout 2 bolts)
> public class JoinerTopologyTest {
> public static void main(String[] args) throws IOException {
> Config conf = new Config();
> conf.setNumWorkers(5);
> conf.setDebug(true);
> TopologyBuilder builder = new TopologyBuilder();
> builder.setSpout("SPOUT-1",new MySpout(),1);
> builder.setBolt("BOLT-1",new Bolt1(), 3)
> .shuffleGrouping("SPOUT-1");
> builder.setBolt("JOINER", new JoinerBolt(),1)
> .shuffleGrouping("BOLT-1")
> .shuffleGrouping("SPOUT-1","str1");
> final LocalCluster cluster = new LocalCluster();
> cluster.submitTopology("TOPO1",conf,builder.createTopology());
> System.in.read();
> cluster.shutdown();
> }
> }
> But when i run it from InteliJ IDEA i get:
> java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> com.pixonic.zephyr.compaction.tests.Bolt1 at 
> 

[GitHub] storm pull request #1560: STORM-1966 Expand metric having Map type as value ...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1560#discussion_r70804467
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java ---
@@ -54,6 +54,25 @@ public String toString() {
 }
 public String name; 
 public Object value;
+
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (!(o instanceof DataPoint)) return false;
+
+DataPoint dataPoint = (DataPoint) o;
+
+if (name != null ? !name.equals(dataPoint.name) : 
dataPoint.name != null) return false;
--- End diff --

Since we're on JDK7, I can just use Objects.equals and Objects.deepEquals.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1971) HDFS Timed Synchronous Policy

[Aaron Dossett (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376905#comment-15376905
 ] 

Aaron Dossett commented on STORM-1971:
--

HDFS bolt does support tick tuples to periodically flush data at a configurable 
interval.  Does that address this concern?

> HDFS Timed Synchronous Policy
> -
>
> Key: STORM-1971
> URL: https://issues.apache.org/jira/browse/STORM-1971
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hdfs
>Affects Versions: 0.10.0, 1.0.0
>Reporter: darion yaphet
>Assignee: darion yaphet
>
> When the data need to be wrote to HDFS is not very large in quantity . We 
> need a timed synchronous policy to flush cached date into HDFS periodically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1971) HDFS Timed Synchronous Policy

[darion yaphet (JIRA)]

darion yaphet created STORM-1971:


 Summary: HDFS Timed Synchronous Policy
 Key: STORM-1971
 URL: https://issues.apache.org/jira/browse/STORM-1971
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-hdfs
Affects Versions: 0.10.0, 1.0.0
Reporter: darion yaphet
Assignee: darion yaphet


When the data need to be wrote to HDFS is not very large in quantity . We need 
a timed synchronous policy to flush cached date into HDFS periodically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1968) Storm logviewer does not work for nimbus.log in secure cluster

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-1968:


I think this can be included to 1.0.2 as well, since it's already got +1.

> Storm logviewer does not work for nimbus.log in secure cluster
> --
>
> Key: STORM-1968
> URL: https://issues.apache.org/jira/browse/STORM-1968
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> logviewer invokes "get-log-user-group-whitelist" which tries to get the 
> worker metadata file by invoking "get-log-metadata-file". In the case of 
> nimbus.log clojure-from-yaml-file  returns nil and the authorization fails.
> Modify clojure-from-yaml-file to return an empty map in case of failures so 
> that the authorization can continue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1939) Frequent InterruptedException raised by ShellBoltMessageQueue.poll

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1939.
-
   Resolution: Fixed
Fix Version/s: 1.1.0
   1.0.2
   2.0.0

Merged into master, 1.x, 1.0.x branches.

> Frequent InterruptedException raised by ShellBoltMessageQueue.poll
> --
>
> Key: STORM-1939
> URL: https://issues.apache.org/jira/browse/STORM-1939
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.1
>Reporter: Dan Blanchard
>Assignee: Jungtaek Lim
> Fix For: 2.0.0, 1.0.2, 1.1.0
>
>
> We've recently started testing out Storm 1.0.1 on a beta cluster we have 
> setup, and we've noticed that one of our topologies frequently crashes with 
> the following stack trace:
> {code:java}
> java.lang.InterruptedException 
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
>  
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
>  
> at 
> org.apache.storm.utils.ShellBoltMessageQueue.poll(ShellBoltMessageQueue.java:104)
>  
> at 
> org.apache.storm.task.ShellBolt$BoltWriterRunnable.run(ShellBolt.java:383) 
> at java.lang.Thread.run(Thread.java:745)
> {code}
> We're using a lot of Python components with streamparse 3.0.0.dev3 and are 
> using the [MessagePackSerializer that was originally from 
> pyleus|https://github.com/YelpArchive/pyleus/blob/develop/topology_builder/src/main/java/com/yelp/pyleus/serializer/MessagePackSerializer.java]
>  with all the instances of "backtype" replaced with "org.apache".
> Aside from the frequent bolt deaths from these exceptions, things seem to 
> work, so I'm not sure what's going on here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Maintaining example topologies for external components

[Xin Wang]

Thanks, here is the PR https://github.com/apache/storm/pull/1565.

2016-07-14 12:44 GMT+08:00 Abhishek Agarwal :

> Not at all Xin. Please go ahead.
> On Jul 14, 2016 7:46 AM, "Xin Wang"  wrote:
>
> > hi Abhishek,
> > I have done a lot for this. Do you mind I file a Jira and make a pull
> > request for this?
> >
> >
> > 2016-07-07 17:47 GMT+08:00 Abhishek Agarwal :
> >
> > > Alright. The majority seems to be in agreement over submodules inside
> > > examples. I will do the clean up of existing modules. Post that, we can
> > > follow the same rule for new modules.
> > >
> > > On Thu, Jun 30, 2016 at 8:32 PM, Cody Innowhere 
> > > wrote:
> > >
> > > > We can have examples in perspective modules and gather the example
> > > > binaries(jars) together into one directory (each in a sub directory)
> by
> > > > adding a maven task.
> > > >
> > > > On Thu, Jun 30, 2016 at 10:12 PM, Jungtaek Lim 
> > > wrote:
> > > >
> > > > > Thanks Abhishek for bring good topic to discuss.
> > > > > Personally I'm either fine. The thing I'd just like to see is
> having
> > a
> > > > > general rule across all external modules.
> > > > >
> > > > > Thanks,
> > > > > Jungtaek Lim (HeartSaVioR)
> > > > >
> > > > >
> > > > > 2016년 6월 30일 (목) 오후 11:09, Satish Duggana <
> sdugg...@hortonworks.com
> > >님이
> > > > 작성:
> > > > >
> > > > > > Hi Xin,
> > > > > > I am fine with having multiple submodules in samples/examples
> > module.
> > > > > >
> > > > > > Thanks,
> > > > > > Satish.
> > > > > >
> > > > > > On 6/30/16, 6:51 PM, "Xin Wang"  wrote:
> > > > > >
> > > > > > hi Satish,
> > > > > > In one `examples` directory not mean they must be in the same
> java
> > > > maven
> > > > > > project. we can create several projects. e.g.
> > > examples/storm-opentsdb,
> > > > > > examples/storm-redis.
> > > > > >
> > > > > > 2016-06-30 21:04 GMT+08:00 Xin Wang :
> > > > > >
> > > > > > > +1 for the former.
> > > > > > >
> > > > > > > * examples have the same entry will be very convenient for
> users
> > to
> > > > > > > looking for.
> > > > > > > * refer to other projects like spark, it has the `examples`
> > module
> > > > > > > including `streaming`, `sql`, `ml`.
> > > > > > >
> > > > > > > Also, I'm happy to take part in.
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Xin
> > > > > > >
> > > > > > > 2016-06-30 16:39 GMT+08:00 Abhishek Agarwal <
> > abhishc...@gmail.com
> > > >:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >> Right now the example topologies/classes for some external
> > modules
> > > > are
> > > > > > >> being put up in the test folder itself. The problem I see is
> > that,
> > > > > > >> -> the example code isn't really test code so test folder
> isn't
> > > the
> > > > > > right
> > > > > > >> fit.
> > > > > > >> -> people, who are looking for example code, may not find the
> > > > example
> > > > > > >> code.
> > > > > > >>
> > > > > > >> I can see two solutions -
> > > > > > >>  -> Have module level example directories e.g.
> > > > > examples/storm-opentsdb,
> > > > > > >> examples/storm-redis
> > > > > > >> -> Have an examples folder within module itself and example
> > > > topologies
> > > > > > are
> > > > > > >> put there.
> > > > > > >>
> > > > > > >> In any case, it is better to document github links of example
> > code
> > > > in
> > > > > > the
> > > > > > >> documentation of any external module.
> > > > > > >>
> > > > > > >> Let me know what you guys think.
> > > > > > >>
> > > > > > >> --
> > > > > > >> Regards,
> > > > > > >> Abhishek Agarwal
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Abhishek Agarwal
> > >
> >
>

[GitHub] storm pull request #1543: STORM-1939 Ignore InterruptedException on ShellBol...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1543


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1939) Frequent InterruptedException raised by ShellBoltMessageQueue.poll

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376882#comment-15376882
 ] 

ASF GitHub Bot commented on STORM-1939:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1543


> Frequent InterruptedException raised by ShellBoltMessageQueue.poll
> --
>
> Key: STORM-1939
> URL: https://issues.apache.org/jira/browse/STORM-1939
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.1
>Reporter: Dan Blanchard
>Assignee: Jungtaek Lim
>
> We've recently started testing out Storm 1.0.1 on a beta cluster we have 
> setup, and we've noticed that one of our topologies frequently crashes with 
> the following stack trace:
> {code:java}
> java.lang.InterruptedException 
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
>  
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
>  
> at 
> org.apache.storm.utils.ShellBoltMessageQueue.poll(ShellBoltMessageQueue.java:104)
>  
> at 
> org.apache.storm.task.ShellBolt$BoltWriterRunnable.run(ShellBolt.java:383) 
> at java.lang.Thread.run(Thread.java:745)
> {code}
> We're using a lot of Python components with streamparse 3.0.0.dev3 and are 
> using the [MessagePackSerializer that was originally from 
> pyleus|https://github.com/YelpArchive/pyleus/blob/develop/topology_builder/src/main/java/com/yelp/pyleus/serializer/MessagePackSerializer.java]
>  with all the instances of "backtype" replaced with "org.apache".
> Aside from the frequent bolt deaths from these exceptions, things seem to 
> work, so I'm not sure what's going on here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376874#comment-15376874
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1560#discussion_r70801243
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java ---
@@ -54,6 +54,25 @@ public String toString() {
 }
 public String name; 
 public Object value;
+
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (!(o instanceof DataPoint)) return false;
+
+DataPoint dataPoint = (DataPoint) o;
+
+if (name != null ? !name.equals(dataPoint.name) : 
dataPoint.name != null) return false;
--- End diff --

This is IntelliJ generated code so might not beauty and doesn't rely on 
other library. StringUtils.equals() would be better. I'll address it.


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1560: STORM-1966 Expand metric having Map type as value ...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1560#discussion_r70801243
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java ---
@@ -54,6 +54,25 @@ public String toString() {
 }
 public String name; 
 public Object value;
+
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (!(o instanceof DataPoint)) return false;
+
+DataPoint dataPoint = (DataPoint) o;
+
+if (name != null ? !name.equals(dataPoint.name) : 
dataPoint.name != null) return false;
--- End diff --

This is IntelliJ generated code so might not beauty and doesn't rely on 
other library. StringUtils.equals() would be better. I'll address it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1966) Expand metric having Map type as value into multiple metrics based on entries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376872#comment-15376872
 ] 

ASF GitHub Bot commented on STORM-1966:
---

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1560#discussion_r70800983
  
--- Diff: conf/storm.yaml.example ---
@@ -45,6 +45,10 @@
 ## - when none of configuration for metric filter are specified, it'll be 
treated as 'pass all'.
 ## - you need to specify either whitelist or blacklist, or none of them. 
You can't specify both of them.
 ## - you can specify multiple whitelist / blacklist with regular expression
+## populateMapType: expand metric with map type as value to multiple 
metrics
+## - set to true when you would like to apply filter to populated metrics
--- End diff --

Yes I was thinking about that but missed it. I'll address it.


> Expand metric having Map type as value into multiple metrics based on entries
> -
>
> Key: STORM-1966
> URL: https://issues.apache.org/jira/browse/STORM-1966
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We're introducing "metrics filter" (STORM-1700) into Storm 1.1.0, which can 
> give a control of volume and kinds of metrics to users.
> After playing with metrics, I found that most of built-in metrics in Storm 
> (core and storm-kafka) are having Map as value which have been expected to be 
> populated from Metrics Consumer. Since filter resides on metrics consumer 
> bolt (not injected to metrics consumer) filter cannot know how metrics are 
> populated, thus can't filter out some of populated metrics.
> For example, let's say we have metric which name is 'A' and value is \{"B": 
> 1, "C": 2\}. For now we can't filter out 'A.C' and keep only 'A.B' since 
> filter even doesn't know 'A' will be changed to 'A.B' and 'A.C'.
> Since well-known metrics consumer (like storm-graphite) already supports 
> populating metrics from one level map of value, I'd like to support this from 
> Storm side and apply filter to populated metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)