[jira] [Created] (STORM-2731) Simple checks in Storm Windowing
Boyang Jerry Peng created STORM-2731: Summary: Simple checks in Storm Windowing Key: STORM-2731 URL: https://issues.apache.org/jira/browse/STORM-2731 Project: Apache Storm Issue Type: Improvement Reporter: Boyang Jerry Peng Assignee: Boyang Jerry Peng Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2445) Topology log search refers supervisor ID as host of worker which contains UUID
[ https://issues.apache.org/jira/browse/STORM-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2445. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~kabhwan], I merged this into master. > Topology log search refers supervisor ID as host of worker which contains UUID > -- > > Key: STORM-2445 > URL: https://issues.apache.org/jira/browse/STORM-2445 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > Fix For: 2.0.0 > > Attachments: Screen Shot 2017-03-31 at 5.38.07 PM.png > > Time Spent: 10m > Remaining Estimate: 0h > > It seems to take supervisor ID as worker's host in topology log search, so > API request to logviewer fails. > Please check attachment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2730) Add in config options for acker cpu and memory
Ethan Li created STORM-2730: --- Summary: Add in config options for acker cpu and memory Key: STORM-2730 URL: https://issues.apache.org/jira/browse/STORM-2730 Project: Apache Storm Issue Type: Improvement Reporter: Ethan Li Assignee: Ethan Li Priority: Minor We want to add in configurations for acker cpu and memory requirements instead of just using topology.component.resources.onheap.memory.mb etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2670) move storm-client-misc to external/http-forwarding-metrics-consumer
[ https://issues.apache.org/jira/browse/STORM-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2670. Resolution: Duplicate This was fixed in a better way by STORM-2702 > move storm-client-misc to external/http-forwarding-metrics-consumer > --- > > Key: STORM-2670 > URL: https://issues.apache.org/jira/browse/STORM-2670 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > storm-client-misc is not named very well, we should fix it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2702) Better load generation testing tools
[ https://issues.apache.org/jira/browse/STORM-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2702. Resolution: Fixed Fix Version/s: 2.0.0 > Better load generation testing tools > > > Key: STORM-2702 > URL: https://issues.apache.org/jira/browse/STORM-2702 > Project: Apache Storm > Issue Type: Improvement > Components: storm-loadgen >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 2.0.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > We have some tools that test load generation, but it would be nice to > standardize it a bit and clean things up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2635) Deep log search doesn’t work when there’s no topology in topology history
[ https://issues.apache.org/jira/browse/STORM-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2635. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~kabhwan], I merged this into master. > Deep log search doesn’t work when there’s no topology in topology history > - > > Key: STORM-2635 > URL: https://issues.apache.org/jira/browse/STORM-2635 > Project: Apache Storm > Issue Type: Bug > Components: storm-server >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When doing deep search with cluster which doesn't ever killed topology (hence > no history on topology histories), Nimbus throws error on getting topology > histories. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2678) Improve performance of LoadAwareShuffleGrouping
[ https://issues.apache.org/jira/browse/STORM-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2678. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~kabhwan], I merged this into master. > Improve performance of LoadAwareShuffleGrouping > > > Key: STORM-2678 > URL: https://issues.apache.org/jira/browse/STORM-2678 > Project: Apache Storm > Issue Type: Bug > Components: storm-client >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > Fix For: 2.0.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > While reviewing STORM-2306, I found a room to improve > LoadAwareShuffleGrouping, basically taking similar approach with > ShuffleGrouping. > The idea is constructing virtual ring (array) with proper length which tasks > are randomly distributed to the ring respecting their loads, and just do > sequential access like we do in ShuffleGrouping. Maybe slower to reflect load > information due to construct virtual ring but no need to call randInt and > additional loop for each tuple emission. > We may also want to reduce call of System.currentTimeMillis(), and if we get > rid of that, it should be better. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2724) ExecutorService in WaterMarkEventGenerator never shutdown
[ https://issues.apache.org/jira/browse/STORM-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2724: -- Affects Version/s: 2.0.0 > ExecutorService in WaterMarkEventGenerator never shutdown > - > > Key: STORM-2724 > URL: https://issues.apache.org/jira/browse/STORM-2724 > Project: Apache Storm > Issue Type: Bug > Components: storm-client, storm-core >Affects Versions: 2.0.0, 1.0.4, 1.1.1 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > Time Spent: 40m > Remaining Estimate: 0h > > I have seen a topology with event time windowing never terminated on local > mode. While looking into detail on thread dump I found only one non-daemon > thread prevents process to be not finished: executorService in > WaterMarkEventGenerator. > Btw, I dumped thread via jstack but impossible to find from jstack result > because it doesn't have thread factory hence thread name is pool-*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2729) Fix NPE in WorkerState runWorkerStartHooks and runWorkerShutdownHooks methods
Ethan Li created STORM-2729:
---
Summary: Fix NPE in WorkerState runWorkerStartHooks and
runWorkerShutdownHooks methods
Key: STORM-2729
URL: https://issues.apache.org/jira/browse/STORM-2729
Project: Apache Storm
Issue Type: Bug
Components: storm-client
Reporter: Ethan Li
Assignee: Ethan Li
Priority: Critical
This happens when I tried to run 0.10 topology on 2.x cluster. It's because in
0.10,
{code:java}
struct StormTopology {
//ids must be unique across maps
// #workers to use is in conf
1: required map spouts;
2: required map bolts;
3: required map state_spouts;
#reserved 4: optional list worker_hooks;
#reserved 5: optional list dependency_jars;
#reserved 6: optional list dependency_artifacts;
7: optional string storm_version;
8: optional string jdk_version;
}
{code}
The worker_hooks field is commented out. When [WorkerState. runWorkerStartHooks
()|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java#L586]
calls get_worker_hooks(), it returns null.
worker_hooks is an optional field in StormTopology struct
([storm.thrift|https://github.com/apache/storm/blob/master/storm-client/src/storm.thrift#L127]).
So when we call get_worker_hooks(), we need to check with
is_set_worker_hooks().
BTW, this can be a common issue for all thrift structs. We should always check
is_set_xxx before we call get_xxx if xxx is an optional field.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (STORM-2728) Generic Resource Scheduling to avoid starvation and handle affinity
Govind Menon created STORM-2728: --- Summary: Generic Resource Scheduling to avoid starvation and handle affinity Key: STORM-2728 URL: https://issues.apache.org/jira/browse/STORM-2728 Project: Apache Storm Issue Type: Sub-task Components: storm-core Reporter: Govind Menon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2726) Allow Topologies and Supervisors to specify a generic resource set
Govind Menon created STORM-2726: --- Summary: Allow Topologies and Supervisors to specify a generic resource set Key: STORM-2726 URL: https://issues.apache.org/jira/browse/STORM-2726 Project: Apache Storm Issue Type: Sub-task Components: storm-core Reporter: Govind Menon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2727) Initialization of generic resource configurations
Govind Menon created STORM-2727: --- Summary: Initialization of generic resource configurations Key: STORM-2727 URL: https://issues.apache.org/jira/browse/STORM-2727 Project: Apache Storm Issue Type: Sub-task Components: storm-core Reporter: Govind Menon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2725) Support GPUs and other generic resource types in scheduling of topologes
[ https://issues.apache.org/jira/browse/STORM-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Govind Menon updated STORM-2725: Summary: Support GPUs and other generic resource types in scheduling of topologes (was: Support GPUs and other generic resource types in schedi) > Support GPUs and other generic resource types in scheduling of topologes > > > Key: STORM-2725 > URL: https://issues.apache.org/jira/browse/STORM-2725 > Project: Apache Storm > Issue Type: Epic > Components: storm-core >Reporter: Govind Menon >Assignee: Govind Menon > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2725) Support GPUs and other generic resource types in schedi
Govind Menon created STORM-2725: --- Summary: Support GPUs and other generic resource types in schedi Key: STORM-2725 URL: https://issues.apache.org/jira/browse/STORM-2725 Project: Apache Storm Issue Type: Epic Components: storm-core Reporter: Govind Menon Assignee: Govind Menon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2724) ExecutorService in WaterMarkEventGenerator never shutdown
[ https://issues.apache.org/jira/browse/STORM-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157015#comment-16157015 ] Jungtaek Lim commented on STORM-2724: - This issue affects all 1.x version lines. > ExecutorService in WaterMarkEventGenerator never shutdown > - > > Key: STORM-2724 > URL: https://issues.apache.org/jira/browse/STORM-2724 > Project: Apache Storm > Issue Type: Bug > Components: storm-client, storm-core >Affects Versions: 1.0.4, 1.1.1 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > I have seen a topology with event time windowing never terminated on local > mode. While looking into detail on thread dump I found only one non-daemon > thread prevents process to be not finished: executorService in > WaterMarkEventGenerator. > Btw, I dumped thread via jstack but impossible to find from jstack result > because it doesn't have thread factory hence thread name is pool-*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2724) ExecutorService in WaterMarkEventGenerator never shutdown
Jungtaek Lim created STORM-2724: --- Summary: ExecutorService in WaterMarkEventGenerator never shutdown Key: STORM-2724 URL: https://issues.apache.org/jira/browse/STORM-2724 Project: Apache Storm Issue Type: Bug Components: storm-client, storm-core Affects Versions: 1.0.4, 1.1.1 Reporter: Jungtaek Lim Assignee: Jungtaek Lim I have seen a topology with event time windowing never terminated on local mode. While looking into detail on thread dump I found only one non-daemon thread prevents process to be not finished: executorService in WaterMarkEventGenerator. Btw, I dumped thread via jstack but impossible to find from jstack result because it doesn't have thread factory hence thread name is pool-*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
