[jira] [Commented] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
[ https://issues.apache.org/jira/browse/STORM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098242#comment-15098242 ] Garrick Dasbach commented on STORM-1471: Not sure how far out the JStorm port is, but in the interim, maybe it would be sufficient to make the Kafka fetch asynchronous. That way we can allow long-polling without blocking the spout. > Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig > - > > Key: STORM-1471 > URL: https://issues.apache.org/jira/browse/STORM-1471 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Reporter: Garrick Dasbach > > We currently have an issue in our storm cluster where our Kafka brokers are > under heavy load due to too many fetch requests from storm. We've narrowed > the problem to the way Fetch Requests are build in KafkaUtils. When using > the FetchRequestBuilder, storm provides overrides for all the properties > except minBytes. The default for that field is 0 (even though the Kafka > default for the high-level consumer is 1). When paired with a maxWait > 0, > this creates a situation where the broker can immediately return a response > without waiting (due to minBytes 0). This puts a heavy load on the brokers > and defeats the purpose of any long polling. > By making this a SpoutConfig option, it will allow the user to set that as > appropriate for their situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: JStorm CGroup
I would love to see true support for mesos, YARN, openstack, etc. added, but I also see stand alone mode offering a lot more flexibility, especially in the area of scheduling, than a two level scheduler can currently offer. It is on my roadmap to look into after the JStorm migration (just started), Resource Aware Scheduling (almost done needs testing and better isolation), and adding in automatic elasticity around topology specified SLAs (working with a few researchers around some prototypes in this area). To be able to support running on other cluster technologies in a proper way we need to provide plugability in a few different places. First we need a way for a scheduler/cluster to request topology specific dedicated resources, and for nimbus to provision, manage, monitor, and ideally resize (for elasticity) those resources. With security and resource aware scheduling, we need these external requests to be on a per topology bases, not bolted on like they are now. This would also necessitate the schedulers being updated so that they could take advantage of these new APIs requesting external resources either when a topology explicitly asks to be on a given external resource, or optionally when dedicated resources are no longer available and the topology has specified the proper configurations/credentials to allow it to run using those external resources. That handles scheduling, but there are some additional features that storm offers which other systems don't yet offer, and many never will. For example the storm blob store API is similar to the dist cache in YARN, but it we can do in place replacement without relaunching. We also favor fast fail and I don't think all of these types of clusters will nor should offer the process monitoring and re-spawning needed for it. As such we would need some sort of a supervisor that would also run under YARN/mesos, etc to provide this extra functionality. I have not totally thought about all of what it would need from a plugability standpoint to make that work. There is also the logviewer which does more then just logs, so we would need some pluggable way to be able to point people to where their logs/artifacts are, and to monitor the resource usage of the logs (perhaps that part should move off to the supervisor). All of that seems like a lot more work compared to providing a pluggable interface in the supervisor that would allow for it to provision, manage, monitor, and again possibly resize, local workers. In fact I see a lot of potential overlap between the two of them and the pluggability that would be needed in the supervisor for running on mesos, YARN, etc. - Bobby On Thursday, January 14, 2016 12:39 AM, Erik Weatherswrote: Perhaps rather than just bolting on "cgroup support", we could instead open a dialogue about having Mesos support be a core feature of Storm. The current integration is a bit unwieldy & hackish at the moment, arising from the conflicting natures of Mesos and Storm w.r.t. scheduling of resources. i.e., Storm assumes you have existing "slots" for running workers on, whereas Mesos is more dynamic, requiring frameworks that run on top of it to tell Mesos just how many resources (CPUs, Memory, etc.) are needed by the framework's tasks. One example of an issue with Storm-on-Mesos: the Storm logviewer is completely busted when you are using Mesos, I filed a ticket with a description of the issue and proposed modifications to allow it to function: - https://issues.apache.org/jira/browse/STORM-1342 Furthermore, there are fundamental behaviors in Storm that don't mesh well with Mesos: - the interfaces of INimbus (allSlotsAvailableForScheduling(), assignSlots(), getForcedScheduler(), etc.) make it difficult to create an ideal Mesos integration framework, since they don't allow the Mesos integration code to *really* know what's going on from the Nimbus's perspective. e.g., - knowing which topologies & how many workers need to be scheduled at any given moment. - since the integration code cannot know what is actually needed to be run when it receives offers from Mesos, it just hoards those offers, leading to resource starvation in the Mesos cluster. - the "fallback" behavior of allowing the topology to settle for having less worker processes than requested should be disable-able. For carefully tuned topologies it is quite bad to run on less than the expected number of worker processes. - also, this behavior endangers the idea of having the Mesos integration code *only* hoard Mesos offers after a successful round-trip through the allSlotsAvailableForScheduling() polling calls (i.e., only hoard when we know there are pending topologies). It's dangerous because while we wait for another call to allSlotsAvailableForScheduling(), the Nimbus may have decided that it's okie dokie
[jira] [Created] (STORM-1473) Enable search on the daemon log
Zhuo Liu created STORM-1473: --- Summary: Enable search on the daemon log Key: STORM-1473 URL: https://issues.apache.org/jira/browse/STORM-1473 Project: Apache Storm Issue Type: Improvement Components: storm-core Affects Versions: 1.0.0 Reporter: Zhuo Liu Priority: Minor Fix For: 1.0.0 Search works fine for worker log. It would be nice that we can also enable this good feature for the daemon logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1473] enable log search for daemon logs
GitHub user zhuoliu opened a pull request: https://github.com/apache/storm/pull/1015 [STORM-1473] enable log search for daemon logs You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhuoliu/storm 1473 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1015.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1015 commit c9cdcbc76a7b65b03dce09541f2df40a97a6e1fd Author: zhuolDate: 2016-01-14T17:46:49Z [STORM-1473] enable log search for daemon logs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1473) Enable search on the daemon log
[ https://issues.apache.org/jira/browse/STORM-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098496#comment-15098496 ] ASF GitHub Bot commented on STORM-1473: --- GitHub user zhuoliu opened a pull request: https://github.com/apache/storm/pull/1015 [STORM-1473] enable log search for daemon logs You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhuoliu/storm 1473 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1015.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1015 commit c9cdcbc76a7b65b03dce09541f2df40a97a6e1fd Author: zhuolDate: 2016-01-14T17:46:49Z [STORM-1473] enable log search for daemon logs > Enable search on the daemon log > --- > > Key: STORM-1473 > URL: https://issues.apache.org/jira/browse/STORM-1473 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Zhuo Liu >Priority: Minor > Fix For: 1.0.0 > > > Search works fine for worker log. > It would be nice that we can also enable this good feature for the daemon > logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1449) Fix Kafka spout to maintain backward compatibility to pass byte[] instead of ByteBuffer
[ https://issues.apache.org/jira/browse/STORM-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097851#comment-15097851 ] ASF GitHub Bot commented on STORM-1449: --- Github user priyank5485 commented on the pull request: https://github.com/apache/storm/pull/994#issuecomment-171576791 Just ran in to an issue where this fix does not work. Correct me if i am wrong. But a fat storm jar that is compiled with older version of storm-kafka will still use byte[] and the newer version of storm now uses ByteBuffer. Hence it throws an exception at runtime if storm is upgraded. I think upgrade of storm should still work with older topology jars. > Fix Kafka spout to maintain backward compatibility to pass byte[] instead of > ByteBuffer > --- > > Key: STORM-1449 > URL: https://issues.apache.org/jira/browse/STORM-1449 > Project: Apache Storm > Issue Type: Bug >Reporter: Arun Mahadevan >Assignee: Arun Mahadevan > Fix For: 1.0.0 > > > STORM-1220 introduced some changes where in one place it passes ByteBuffer as > is instead of byte[]. Existing bolts that expects a byte[] fails due to this > with "java.lang.RuntimeException: java.lang.ClassCastException: > java.nio.HeapByteBuffer cannot be cast to [B " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1449] Fix Kafka spout to maintain backw...
Github user priyank5485 commented on the pull request: https://github.com/apache/storm/pull/994#issuecomment-171576791 Just ran in to an issue where this fix does not work. Correct me if i am wrong. But a fat storm jar that is compiled with older version of storm-kafka will still use byte[] and the newer version of storm now uses ByteBuffer. Hence it throws an exception at runtime if storm is upgraded. I think upgrade of storm should still work with older topology jars. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...
Github user d2r closed the pull request at: https://github.com/apache/storm/pull/1011 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/1011#issuecomment-171802477 See #1010 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default
[ https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099024#comment-15099024 ] ASF GitHub Bot commented on STORM-1452: --- Github user d2r closed the pull request at: https://github.com/apache/storm/pull/1013 > Worker "profiler" actions broken by default > --- > > Key: STORM-1452 > URL: https://issues.apache.org/jira/browse/STORM-1452 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Blocker > > * The profiler script flight.bash is not packaged by default. > * The default options enable the Oracle specific flight-recorder that > requires a support subscription. > The option to enable the profiler should not be enabled by default. Other > actions such as worker restart, debugging, and heap can remain enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default
[ https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099023#comment-15099023 ] ASF GitHub Bot commented on STORM-1452: --- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/1013#issuecomment-171802210 See #1012 > Worker "profiler" actions broken by default > --- > > Key: STORM-1452 > URL: https://issues.apache.org/jira/browse/STORM-1452 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Blocker > > * The profiler script flight.bash is not packaged by default. > * The default options enable the Oracle specific flight-recorder that > requires a support subscription. > The option to enable the profiler should not be enabled by default. Other > actions such as worker restart, debugging, and heap can remain enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1012#issuecomment-171816513 Works for me. Thanks @d2r. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/1012#issuecomment-171802145 Yes, I can add something. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default
[ https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099022#comment-15099022 ] ASF GitHub Bot commented on STORM-1452: --- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/1012#issuecomment-171802145 Yes, I can add something. > Worker "profiler" actions broken by default > --- > > Key: STORM-1452 > URL: https://issues.apache.org/jira/browse/STORM-1452 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Blocker > > * The profiler script flight.bash is not packaged by default. > * The default options enable the Oracle specific flight-recorder that > requires a support subscription. > The option to enable the profiler should not be enabled by default. Other > actions such as worker restart, debugging, and heap can remain enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/1013#issuecomment-171802210 See #1012 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...
Github user d2r closed the pull request at: https://github.com/apache/storm/pull/1013 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/1016#issuecomment-171802669 Mostly minor comments, otherwise looks ok to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099034#comment-15099034 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/1016#issuecomment-171802669 Mostly minor comments, otherwise looks ok to me. > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
[ https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099032#comment-15099032 ] ASF GitHub Bot commented on STORM-1470: --- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/1011#issuecomment-171802477 See #1010 > org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication > -- > > Key: STORM-1470 > URL: https://issues.apache.org/jira/browse/STORM-1470 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Blocker > > {noformat} > 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for > /favicon.ico > java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64 > at > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) > at > org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) > at > org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at > org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) > at > org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) > at > org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) > at > org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > at > org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) > {noformat} > We already shade commons-codec:commons-codec, but we don't apply that shading > to org.apache.hadoop:hadoop-auth. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
[ https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099033#comment-15099033 ] ASF GitHub Bot commented on STORM-1470: --- Github user d2r closed the pull request at: https://github.com/apache/storm/pull/1011 > org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication > -- > > Key: STORM-1470 > URL: https://issues.apache.org/jira/browse/STORM-1470 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Blocker > > {noformat} > 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for > /favicon.ico > java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64 > at > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) > at > org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) > at > org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at > org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) > at > org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) > at > org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) > at > org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > at > org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) > {noformat} > We already shade commons-codec:commons-codec, but we don't apply that shading > to org.apache.hadoop:hadoop-auth. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/1012#issuecomment-171805811 @ptgoetz , added. Do you think it is OK? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default
[ https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099067#comment-15099067 ] ASF GitHub Bot commented on STORM-1452: --- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/1012#issuecomment-171805811 @ptgoetz , added. Do you think it is OK? > Worker "profiler" actions broken by default > --- > > Key: STORM-1452 > URL: https://issues.apache.org/jira/browse/STORM-1452 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Blocker > > * The profiler script flight.bash is not packaged by default. > * The default options enable the Oracle specific flight-recorder that > requires a support subscription. > The option to enable the profiler should not be enabled by default. Other > actions such as worker restart, debugging, and heap can remain enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101195#comment-15101195 ] ASF GitHub Bot commented on STORM-1450: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49821028 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -43,32 +43,84 @@ */ public class RAS_Node { private static final Logger LOG = LoggerFactory.getLogger(RAS_Node.class); -private Map_topIdToUsedSlots = new HashMap (); -private Set _freeSlots = new HashSet(); + +//A map consisting of all workers on the node. +//The key of the map is the worker id and the value is the corresponding workerslot object +Map _slots = new HashMap (); + +// A map describing which topologies are using which slots on this node. The format of the map is the following: +// {TopologyId -> {WorkerId -> {Executors}}} +private Map > _topIdToUsedSlots = new HashMap >(); + private final String _nodeId; private String _hostname; private boolean _isAlive; private SupervisorDetails _sup; -private Double _availMemory; -private Double _availCPU; -private Cluster _cluster; -private Topologies _topologies; +private Double _availMemory = 0.0; +private Double _availCPU = 0.0; +private final Cluster _cluster; +private final Topologies _topologies; -public RAS_Node(String nodeId, Set allPorts, boolean isAlive, -SupervisorDetails sup, Cluster cluster, Topologies topologies) { +public RAS_Node(String nodeId, SupervisorDetails sup, Cluster cluster, Topologies topologies, Map workerIdToWorker, Map > assignmentMap) { +//Node ID and supervisor ID are the same. _nodeId = nodeId; -_isAlive = isAlive; -if (_isAlive && allPorts != null) { -for (int port : allPorts) { -_freeSlots.add(new WorkerSlot(_nodeId, port)); -} -_sup = sup; +if (sup == null) { +_isAlive = false; +} else { +_isAlive = !cluster.isBlackListed(_nodeId); +} + +_cluster = cluster; +_topologies = topologies; + +// initialize slots for this node +if (workerIdToWorker != null) { +_slots = workerIdToWorker; +} + +//initialize assignment map +if (assignmentMap != null) { +_topIdToUsedSlots = assignmentMap; +} + +//check if node is alive +if (_isAlive && sup != null) { _hostname = sup.getHost(); +_sup = sup; _availMemory = getTotalMemoryResources(); _availCPU = getTotalCpuResources(); + +LOG.debug("Found a {} Node {} {}", +_isAlive ? "living" : "dead", _nodeId, sup.getAllPorts()); +LOG.debug("resources_mem: {}, resources_CPU: {}", sup.getTotalMemory(), sup.getTotalCPU()); +//intialize resource usages on node +intializeResources(); +} +} + +/** + * intializes resource usages on node --- End diff -- will fix > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2.
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49821059 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -103,92 +182,37 @@ public boolean isAlive() { } public boolean isTotallyFree() { -return _topIdToUsedSlots.isEmpty(); +return getUsedSlots().isEmpty(); --- End diff -- its used in resource_aware_scheduler_test.clj --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49821028 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -43,32 +43,84 @@ */ public class RAS_Node { private static final Logger LOG = LoggerFactory.getLogger(RAS_Node.class); -private Map_topIdToUsedSlots = new HashMap (); -private Set _freeSlots = new HashSet(); + +//A map consisting of all workers on the node. +//The key of the map is the worker id and the value is the corresponding workerslot object +Map _slots = new HashMap (); + +// A map describing which topologies are using which slots on this node. The format of the map is the following: +// {TopologyId -> {WorkerId -> {Executors}}} +private Map > _topIdToUsedSlots = new HashMap >(); + private final String _nodeId; private String _hostname; private boolean _isAlive; private SupervisorDetails _sup; -private Double _availMemory; -private Double _availCPU; -private Cluster _cluster; -private Topologies _topologies; +private Double _availMemory = 0.0; +private Double _availCPU = 0.0; +private final Cluster _cluster; +private final Topologies _topologies; -public RAS_Node(String nodeId, Set allPorts, boolean isAlive, -SupervisorDetails sup, Cluster cluster, Topologies topologies) { +public RAS_Node(String nodeId, SupervisorDetails sup, Cluster cluster, Topologies topologies, Map workerIdToWorker, Map > assignmentMap) { +//Node ID and supervisor ID are the same. _nodeId = nodeId; -_isAlive = isAlive; -if (_isAlive && allPorts != null) { -for (int port : allPorts) { -_freeSlots.add(new WorkerSlot(_nodeId, port)); -} -_sup = sup; +if (sup == null) { +_isAlive = false; +} else { +_isAlive = !cluster.isBlackListed(_nodeId); +} + +_cluster = cluster; +_topologies = topologies; + +// initialize slots for this node +if (workerIdToWorker != null) { +_slots = workerIdToWorker; +} + +//initialize assignment map +if (assignmentMap != null) { +_topIdToUsedSlots = assignmentMap; +} + +//check if node is alive +if (_isAlive && sup != null) { _hostname = sup.getHost(); +_sup = sup; _availMemory = getTotalMemoryResources(); _availCPU = getTotalCpuResources(); + +LOG.debug("Found a {} Node {} {}", +_isAlive ? "living" : "dead", _nodeId, sup.getAllPorts()); +LOG.debug("resources_mem: {}, resources_CPU: {}", sup.getTotalMemory(), sup.getTotalCPU()); +//intialize resource usages on node +intializeResources(); +} +} + +/** + * intializes resource usages on node --- End diff -- will fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49820988 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/SchedulerAssignment.java --- @@ -55,4 +56,6 @@ public Set getExecutors(); public Set getSlots(); + +public MapgetSlotToExecutors(); --- End diff -- @d2r should I delete all of the public declarations for all the methods in this class then? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101194#comment-15101194 ] ASF GitHub Bot commented on STORM-1450: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49820997 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/WorkerSlot.java --- @@ -40,11 +47,8 @@ public int getPort() { return port; } -public WorkerSlot allocateResource(double memOnHeap, double memOffHeap, double cpu) { -this.memOnHeap += memOnHeap; -this.memOffHeap += memOffHeap; -this.cpu += cpu; -return this; +public String getId() { +return this.getNodeId() + ":" + this.getPort(); --- End diff -- will remove > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101193#comment-15101193 ] ASF GitHub Bot commented on STORM-1450: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49820988 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/SchedulerAssignment.java --- @@ -55,4 +56,6 @@ public Set getExecutors(); public Set getSlots(); + +public MapgetSlotToExecutors(); --- End diff -- @d2r should I delete all of the public declarations for all the methods in this class then? > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49820997 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/WorkerSlot.java --- @@ -40,11 +47,8 @@ public int getPort() { return port; } -public WorkerSlot allocateResource(double memOnHeap, double memOffHeap, double cpu) { -this.memOnHeap += memOnHeap; -this.memOffHeap += memOffHeap; -this.cpu += cpu; -return this; +public String getId() { +return this.getNodeId() + ":" + this.getPort(); --- End diff -- will remove --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49821516 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/DefaultResourceAwareStrategy.java --- @@ -121,7 +116,7 @@ public SchedulingResult schedule(TopologyDetails td) { td.getTaskResourceReqList(exec), entry.getKey() }); WorkerSlot targetSlot = this.findWorkerForExec(exec, td, schedulerAssignmentMap); if (targetSlot != null) { -RAS_Node targetNode = this.idToNode(targetSlot.getNodeId()); +NodeDetails targetNode = this.idToNode(targetSlot.getNodeId()); --- End diff -- will refactor --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-1476) Filter -c options from args and add them as part of storm.options
Satish Duggana created STORM-1476: - Summary: Filter -c options from args and add them as part of storm.options Key: STORM-1476 URL: https://issues.apache.org/jira/browse/STORM-1476 Project: Apache Storm Issue Type: Bug Reporter: Satish Duggana Assignee: Satish Duggana -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101212#comment-15101212 ] ASF GitHub Bot commented on STORM-1450: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49821516 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/DefaultResourceAwareStrategy.java --- @@ -121,7 +116,7 @@ public SchedulingResult schedule(TopologyDetails td) { td.getTaskResourceReqList(exec), entry.getKey() }); WorkerSlot targetSlot = this.findWorkerForExec(exec, td, schedulerAssignmentMap); if (targetSlot != null) { -RAS_Node targetNode = this.idToNode(targetSlot.getNodeId()); +NodeDetails targetNode = this.idToNode(targetSlot.getNodeId()); --- End diff -- will refactor > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101197#comment-15101197 ] ASF GitHub Bot commented on STORM-1450: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49821059 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -103,92 +182,37 @@ public boolean isAlive() { } public boolean isTotallyFree() { -return _topIdToUsedSlots.isEmpty(); +return getUsedSlots().isEmpty(); --- End diff -- its used in resource_aware_scheduler_test.clj > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101198#comment-15101198 ] ASF GitHub Bot commented on STORM-1450: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49821063 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -103,92 +182,37 @@ public boolean isAlive() { } public boolean isTotallyFree() { -return _topIdToUsedSlots.isEmpty(); +return getUsedSlots().isEmpty(); } public int totalSlotsFree() { -return _freeSlots.size(); +return getFreeSlots().size(); } public int totalSlotsUsed() { -int total = 0; -for (Set slots : _topIdToUsedSlots.values()) { -total += slots.size(); -} -return total; +return getUsedSlots().size(); } public int totalSlots() { -return totalSlotsFree() + totalSlotsUsed(); +return _slots.size(); } public int totalSlotsUsed(String topId) { -int total = 0; -Set slots = _topIdToUsedSlots.get(topId); -if (slots != null) { -total = slots.size(); -} -return total; -} - -private void validateSlot(WorkerSlot ws) { -if (!_nodeId.equals(ws.getNodeId())) { -throw new IllegalArgumentException( -"Trying to add a slot to the wrong node " + ws + -" is not a part of " + _nodeId); -} -} - -void addOrphanedSlot(WorkerSlot ws) { -if (_isAlive) { -throw new IllegalArgumentException("Orphaned Slots " + -"only are allowed on dead nodes."); -} -validateSlot(ws); -if (_freeSlots.contains(ws)) { -return; -} -for (Set used : _topIdToUsedSlots.values()) { -if (used.contains(ws)) { -return; -} -} -_freeSlots.add(ws); -} - -boolean assignInternal(WorkerSlot ws, String topId, boolean dontThrow) { -validateSlot(ws); -if (!_freeSlots.remove(ws)) { -if (dontThrow) { -return true; -} -throw new IllegalStateException("Assigning a slot that was not free " + ws); -} -Set usedSlots = _topIdToUsedSlots.get(topId); -if (usedSlots == null) { -usedSlots = new HashSet(); -_topIdToUsedSlots.put(topId, usedSlots); -} -usedSlots.add(ws); -return false; +return getUsedSlots(topId).size(); } /** * Free all slots on this node. This will update the Cluster too. */ -public void freeAllSlots() { + public void freeAllSlots() { --- End diff -- will fix > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49821113 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -247,19 +253,23 @@ private void freeMemory(double amount) { private void freeCPU(double amount) { LOG.debug("freeing {} CPU on node...avail CPU: {}", amount, getHostname(), _availCPU); -if ((_availCPU + amount) > getAvailableCpuResources()) { -LOG.warn("Freeing more CPU than there exists!"); +if ((_availCPU + amount) > getTotalCpuResources()) { +LOG.warn("Freeing more CPU than there exists! CPU trying to free: {} Total CPU on Node: {}", (_availMemory + amount), getTotalCpuResources()); return; } _availCPU += amount; } +/** + * get the amount of memory used by a worker + */ public double getMemoryUsedByWorker(WorkerSlot ws) { TopologyDetails topo = findTopologyUsingWorker(ws); if (topo == null) { return 0.0; } Collection execs = getExecutors(ws, _cluster); +LOG.info("getMemoryUsedByWorker executors: {}", execs); --- End diff -- will get rid of --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49821063 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -103,92 +182,37 @@ public boolean isAlive() { } public boolean isTotallyFree() { -return _topIdToUsedSlots.isEmpty(); +return getUsedSlots().isEmpty(); } public int totalSlotsFree() { -return _freeSlots.size(); +return getFreeSlots().size(); } public int totalSlotsUsed() { -int total = 0; -for (Set slots : _topIdToUsedSlots.values()) { -total += slots.size(); -} -return total; +return getUsedSlots().size(); } public int totalSlots() { -return totalSlotsFree() + totalSlotsUsed(); +return _slots.size(); } public int totalSlotsUsed(String topId) { -int total = 0; -Set slots = _topIdToUsedSlots.get(topId); -if (slots != null) { -total = slots.size(); -} -return total; -} - -private void validateSlot(WorkerSlot ws) { -if (!_nodeId.equals(ws.getNodeId())) { -throw new IllegalArgumentException( -"Trying to add a slot to the wrong node " + ws + -" is not a part of " + _nodeId); -} -} - -void addOrphanedSlot(WorkerSlot ws) { -if (_isAlive) { -throw new IllegalArgumentException("Orphaned Slots " + -"only are allowed on dead nodes."); -} -validateSlot(ws); -if (_freeSlots.contains(ws)) { -return; -} -for (Set used : _topIdToUsedSlots.values()) { -if (used.contains(ws)) { -return; -} -} -_freeSlots.add(ws); -} - -boolean assignInternal(WorkerSlot ws, String topId, boolean dontThrow) { -validateSlot(ws); -if (!_freeSlots.remove(ws)) { -if (dontThrow) { -return true; -} -throw new IllegalStateException("Assigning a slot that was not free " + ws); -} -Set usedSlots = _topIdToUsedSlots.get(topId); -if (usedSlots == null) { -usedSlots = new HashSet(); -_topIdToUsedSlots.put(topId, usedSlots); -} -usedSlots.add(ws); -return false; +return getUsedSlots(topId).size(); } /** * Free all slots on this node. This will update the Cluster too. */ -public void freeAllSlots() { + public void freeAllSlots() { --- End diff -- will fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101200#comment-15101200 ] ASF GitHub Bot commented on STORM-1450: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49821113 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -247,19 +253,23 @@ private void freeMemory(double amount) { private void freeCPU(double amount) { LOG.debug("freeing {} CPU on node...avail CPU: {}", amount, getHostname(), _availCPU); -if ((_availCPU + amount) > getAvailableCpuResources()) { -LOG.warn("Freeing more CPU than there exists!"); +if ((_availCPU + amount) > getTotalCpuResources()) { +LOG.warn("Freeing more CPU than there exists! CPU trying to free: {} Total CPU on Node: {}", (_availMemory + amount), getTotalCpuResources()); return; } _availCPU += amount; } +/** + * get the amount of memory used by a worker + */ public double getMemoryUsedByWorker(WorkerSlot ws) { TopologyDetails topo = findTopologyUsingWorker(ws); if (topo == null) { return 0.0; } Collection execs = getExecutors(ws, _cluster); +LOG.info("getMemoryUsedByWorker executors: {}", execs); --- End diff -- will get rid of > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default
[ https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099123#comment-15099123 ] ASF GitHub Bot commented on STORM-1452: --- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1012#issuecomment-171816513 Works for me. Thanks @d2r. +1 > Worker "profiler" actions broken by default > --- > > Key: STORM-1452 > URL: https://issues.apache.org/jira/browse/STORM-1452 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Blocker > > * The profiler script flight.bash is not packaged by default. > * The default options enable the Oracle specific flight-recorder that > requires a support subscription. > The option to enable the profiler should not be enabled by default. Other > actions such as worker restart, debugging, and heap can remain enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1476) Filter -c options from args and add them as part of storm.options
[ https://issues.apache.org/jira/browse/STORM-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101401#comment-15101401 ] ASF GitHub Bot commented on STORM-1476: --- GitHub user satishd opened a pull request: https://github.com/apache/storm/pull/1017 STORM-1476 Filter -c options from args and add them as part of storm.options …options You can merge this pull request into a Git repository by running: $ git pull https://github.com/satishd/storm STORM-1476 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1017.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1017 commit 10b0d5842a12dbb772fb97fd26271d395396aa31 Author: Satish DugganaDate: 2016-01-15T07:52:04Z STORM-1476 Filter -c options from args and add them as part of storm.options > Filter -c options from args and add them as part of storm.options > - > > Key: STORM-1476 > URL: https://issues.apache.org/jira/browse/STORM-1476 > Project: Apache Storm > Issue Type: Bug >Reporter: Satish Duggana >Assignee: Satish Duggana > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1476 Filter -c options from args and add...
GitHub user satishd opened a pull request: https://github.com/apache/storm/pull/1017 STORM-1476 Filter -c options from args and add them as part of storm.options â¦options You can merge this pull request into a Git repository by running: $ git pull https://github.com/satishd/storm STORM-1476 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1017.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1017 commit 10b0d5842a12dbb772fb97fd26271d395396aa31 Author: Satish DugganaDate: 2016-01-15T07:52:04Z STORM-1476 Filter -c options from args and add them as part of storm.options --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Storm 1450
GitHub user jerrypeng opened a pull request: https://github.com/apache/storm/pull/1016 Storm 1450 Code refactored: 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. Each RAS_Node will now be initialized with a map of all its assignments. Each RAS_Node will also figure out resources used and available. Removed unnecessary functions. 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly modify it 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so that the semantics of what a scheduling strategy should do will be more clear. Each scheduling strategy shouldn't be actually assigning anything. The strategy should only calculate a scheduling. Bug fixes: 1. Minor bug in displaying the assigned resources for a supervisor on the UI. The function updateSupervisorResources was placed in the wrong place 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that was done. Added Additional unit tests to test You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerrypeng/storm STORM-1450 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1016.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1016 commit 829ea117c8b382ef72d3c5df841534b89f06dffa Author: Boyang Jerry PengDate: 2016-01-13T18:28:38Z [STORM-1450] - Fix bugs and refactor code in ResourceAwareScheduler commit 688366902d98a7fdc17d854a33ff975c6a60ccab Author: Boyang Jerry Peng Date: 2016-01-13T20:13:54Z adding an additional test and cleaning up --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (STORM-1474) Address remaining minor review comments for STORM-1199
[ https://issues.apache.org/jira/browse/STORM-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1474: --- Issue Type: Sub-task (was: Bug) Parent: STORM-1199 > Address remaining minor review comments for STORM-1199 > -- > > Key: STORM-1474 > URL: https://issues.apache.org/jira/browse/STORM-1474 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-hdfs >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Minor > > Address the last few pending review comments from > https://github.com/apache/storm/pull/936 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1474) Address remaining minor review comments for STORM-1199
Roshan Naik created STORM-1474: -- Summary: Address remaining minor review comments for STORM-1199 Key: STORM-1474 URL: https://issues.apache.org/jira/browse/STORM-1474 Project: Apache Storm Issue Type: Bug Components: storm-hdfs Reporter: Roshan Naik Assignee: Roshan Naik Priority: Minor Address the last few pending review comments from https://github.com/apache/storm/pull/936 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098862#comment-15098862 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49785320 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/WorkerSlot.java --- @@ -40,11 +47,8 @@ public int getPort() { return port; } -public WorkerSlot allocateResource(double memOnHeap, double memOffHeap, double cpu) { -this.memOnHeap += memOnHeap; -this.memOffHeap += memOffHeap; -this.cpu += cpu; -return this; +public String getId() { +return this.getNodeId() + ":" + this.getPort(); --- End diff -- minor: Do not need `this.` when calling methods. > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user zhuoliu commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49785382 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java --- @@ -662,22 +662,50 @@ public void setStatusMap(MapstatusMap) { this.status.putAll(statusMap); } -public void setResources(String topologyId, Double[] resources) { -this.resources.put(topologyId, resources); +/** + * Set the amount of resources used used by a topology. Used for displaying resource information on the UI + * @param topologyId + * @param resources describes the resources requested and assigned to topology in the following format in an array: + * {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu} + */ +public void setTopologyResources(String topologyId, Double[] resources) { --- End diff -- Yep, I think that would be helpful. Both for topology and supervisor resources. We may file a separate JIRA for that or do it within this pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1010#issuecomment-171780642 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49785320 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/WorkerSlot.java --- @@ -40,11 +47,8 @@ public int getPort() { return port; } -public WorkerSlot allocateResource(double memOnHeap, double memOffHeap, double cpu) { -this.memOnHeap += memOnHeap; -this.memOffHeap += memOffHeap; -this.cpu += cpu; -return this; +public String getId() { +return this.getNodeId() + ":" + this.getPort(); --- End diff -- minor: Do not need `this.` when calling methods. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098863#comment-15098863 ] ASF GitHub Bot commented on STORM-1450: --- Github user zhuoliu commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49785382 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java --- @@ -662,22 +662,50 @@ public void setStatusMap(MapstatusMap) { this.status.putAll(statusMap); } -public void setResources(String topologyId, Double[] resources) { -this.resources.put(topologyId, resources); +/** + * Set the amount of resources used used by a topology. Used for displaying resource information on the UI + * @param topologyId + * @param resources describes the resources requested and assigned to topology in the following format in an array: + * {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu} + */ +public void setTopologyResources(String topologyId, Double[] resources) { --- End diff -- Yep, I think that would be helpful. Both for topology and supervisor resources. We may file a separate JIRA for that or do it within this pull request. > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
[ https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098869#comment-15098869 ] ASF GitHub Bot commented on STORM-1470: --- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1010#issuecomment-171780642 +1 > org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication > -- > > Key: STORM-1470 > URL: https://issues.apache.org/jira/browse/STORM-1470 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Blocker > > {noformat} > 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for > /favicon.ico > java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64 > at > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) > at > org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) > at > org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at > org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) > at > org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) > at > org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) > at > org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > at > org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) > {noformat} > We already shade commons-codec:commons-codec, but we don't apply that shading > to org.apache.hadoop:hadoop-auth. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098877#comment-15098877 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49786540 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -43,32 +43,84 @@ */ public class RAS_Node { private static final Logger LOG = LoggerFactory.getLogger(RAS_Node.class); -private Map_topIdToUsedSlots = new HashMap (); -private Set _freeSlots = new HashSet(); + +//A map consisting of all workers on the node. +//The key of the map is the worker id and the value is the corresponding workerslot object +Map _slots = new HashMap (); + +// A map describing which topologies are using which slots on this node. The format of the map is the following: +// {TopologyId -> {WorkerId -> {Executors}}} +private Map > _topIdToUsedSlots = new HashMap >(); + private final String _nodeId; private String _hostname; private boolean _isAlive; private SupervisorDetails _sup; -private Double _availMemory; -private Double _availCPU; -private Cluster _cluster; -private Topologies _topologies; +private Double _availMemory = 0.0; +private Double _availCPU = 0.0; +private final Cluster _cluster; +private final Topologies _topologies; -public RAS_Node(String nodeId, Set allPorts, boolean isAlive, -SupervisorDetails sup, Cluster cluster, Topologies topologies) { +public RAS_Node(String nodeId, SupervisorDetails sup, Cluster cluster, Topologies topologies, Map workerIdToWorker, Map > assignmentMap) { +//Node ID and supervisor ID are the same. _nodeId = nodeId; -_isAlive = isAlive; -if (_isAlive && allPorts != null) { -for (int port : allPorts) { -_freeSlots.add(new WorkerSlot(_nodeId, port)); -} -_sup = sup; +if (sup == null) { +_isAlive = false; +} else { +_isAlive = !cluster.isBlackListed(_nodeId); +} + +_cluster = cluster; +_topologies = topologies; + +// initialize slots for this node +if (workerIdToWorker != null) { +_slots = workerIdToWorker; +} + +//initialize assignment map +if (assignmentMap != null) { +_topIdToUsedSlots = assignmentMap; +} + +//check if node is alive +if (_isAlive && sup != null) { _hostname = sup.getHost(); +_sup = sup; _availMemory = getTotalMemoryResources(); _availCPU = getTotalCpuResources(); + +LOG.debug("Found a {} Node {} {}", +_isAlive ? "living" : "dead", _nodeId, sup.getAllPorts()); +LOG.debug("resources_mem: {}, resources_CPU: {}", sup.getTotalMemory(), sup.getTotalCPU()); +//intialize resource usages on node +intializeResources(); +} +} + +/** + * intializes resource usages on node --- End diff -- `initializes` > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49786540 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -43,32 +43,84 @@ */ public class RAS_Node { private static final Logger LOG = LoggerFactory.getLogger(RAS_Node.class); -private Map_topIdToUsedSlots = new HashMap (); -private Set _freeSlots = new HashSet(); + +//A map consisting of all workers on the node. +//The key of the map is the worker id and the value is the corresponding workerslot object +Map _slots = new HashMap (); + +// A map describing which topologies are using which slots on this node. The format of the map is the following: +// {TopologyId -> {WorkerId -> {Executors}}} +private Map > _topIdToUsedSlots = new HashMap >(); + private final String _nodeId; private String _hostname; private boolean _isAlive; private SupervisorDetails _sup; -private Double _availMemory; -private Double _availCPU; -private Cluster _cluster; -private Topologies _topologies; +private Double _availMemory = 0.0; +private Double _availCPU = 0.0; +private final Cluster _cluster; +private final Topologies _topologies; -public RAS_Node(String nodeId, Set allPorts, boolean isAlive, -SupervisorDetails sup, Cluster cluster, Topologies topologies) { +public RAS_Node(String nodeId, SupervisorDetails sup, Cluster cluster, Topologies topologies, Map workerIdToWorker, Map > assignmentMap) { +//Node ID and supervisor ID are the same. _nodeId = nodeId; -_isAlive = isAlive; -if (_isAlive && allPorts != null) { -for (int port : allPorts) { -_freeSlots.add(new WorkerSlot(_nodeId, port)); -} -_sup = sup; +if (sup == null) { +_isAlive = false; +} else { +_isAlive = !cluster.isBlackListed(_nodeId); +} + +_cluster = cluster; +_topologies = topologies; + +// initialize slots for this node +if (workerIdToWorker != null) { +_slots = workerIdToWorker; +} + +//initialize assignment map +if (assignmentMap != null) { +_topIdToUsedSlots = assignmentMap; +} + +//check if node is alive +if (_isAlive && sup != null) { _hostname = sup.getHost(); +_sup = sup; _availMemory = getTotalMemoryResources(); _availCPU = getTotalCpuResources(); + +LOG.debug("Found a {} Node {} {}", +_isAlive ? "living" : "dead", _nodeId, sup.getAllPorts()); +LOG.debug("resources_mem: {}, resources_CPU: {}", sup.getTotalMemory(), sup.getTotalCPU()); +//intialize resource usages on node +intializeResources(); +} +} + +/** + * intializes resource usages on node --- End diff -- `initializes` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49786777 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -103,92 +182,37 @@ public boolean isAlive() { } public boolean isTotallyFree() { -return _topIdToUsedSlots.isEmpty(); +return getUsedSlots().isEmpty(); --- End diff -- Is this method used? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098881#comment-15098881 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49786777 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -103,92 +182,37 @@ public boolean isAlive() { } public boolean isTotallyFree() { -return _topIdToUsedSlots.isEmpty(); +return getUsedSlots().isEmpty(); --- End diff -- Is this method used? > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49786868 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -103,92 +182,37 @@ public boolean isAlive() { } public boolean isTotallyFree() { -return _topIdToUsedSlots.isEmpty(); +return getUsedSlots().isEmpty(); } public int totalSlotsFree() { -return _freeSlots.size(); +return getFreeSlots().size(); } public int totalSlotsUsed() { -int total = 0; -for (Set slots : _topIdToUsedSlots.values()) { -total += slots.size(); -} -return total; +return getUsedSlots().size(); } public int totalSlots() { -return totalSlotsFree() + totalSlotsUsed(); +return _slots.size(); } public int totalSlotsUsed(String topId) { -int total = 0; -Set slots = _topIdToUsedSlots.get(topId); -if (slots != null) { -total = slots.size(); -} -return total; -} - -private void validateSlot(WorkerSlot ws) { -if (!_nodeId.equals(ws.getNodeId())) { -throw new IllegalArgumentException( -"Trying to add a slot to the wrong node " + ws + -" is not a part of " + _nodeId); -} -} - -void addOrphanedSlot(WorkerSlot ws) { -if (_isAlive) { -throw new IllegalArgumentException("Orphaned Slots " + -"only are allowed on dead nodes."); -} -validateSlot(ws); -if (_freeSlots.contains(ws)) { -return; -} -for (Set used : _topIdToUsedSlots.values()) { -if (used.contains(ws)) { -return; -} -} -_freeSlots.add(ws); -} - -boolean assignInternal(WorkerSlot ws, String topId, boolean dontThrow) { -validateSlot(ws); -if (!_freeSlots.remove(ws)) { -if (dontThrow) { -return true; -} -throw new IllegalStateException("Assigning a slot that was not free " + ws); -} -Set usedSlots = _topIdToUsedSlots.get(topId); -if (usedSlots == null) { -usedSlots = new HashSet(); -_topIdToUsedSlots.put(topId, usedSlots); -} -usedSlots.add(ws); -return false; +return getUsedSlots(topId).size(); } /** * Free all slots on this node. This will update the Cluster too. */ -public void freeAllSlots() { + public void freeAllSlots() { --- End diff -- revert? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098886#comment-15098886 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49786868 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -103,92 +182,37 @@ public boolean isAlive() { } public boolean isTotallyFree() { -return _topIdToUsedSlots.isEmpty(); +return getUsedSlots().isEmpty(); } public int totalSlotsFree() { -return _freeSlots.size(); +return getFreeSlots().size(); } public int totalSlotsUsed() { -int total = 0; -for (Set slots : _topIdToUsedSlots.values()) { -total += slots.size(); -} -return total; +return getUsedSlots().size(); } public int totalSlots() { -return totalSlotsFree() + totalSlotsUsed(); +return _slots.size(); } public int totalSlotsUsed(String topId) { -int total = 0; -Set slots = _topIdToUsedSlots.get(topId); -if (slots != null) { -total = slots.size(); -} -return total; -} - -private void validateSlot(WorkerSlot ws) { -if (!_nodeId.equals(ws.getNodeId())) { -throw new IllegalArgumentException( -"Trying to add a slot to the wrong node " + ws + -" is not a part of " + _nodeId); -} -} - -void addOrphanedSlot(WorkerSlot ws) { -if (_isAlive) { -throw new IllegalArgumentException("Orphaned Slots " + -"only are allowed on dead nodes."); -} -validateSlot(ws); -if (_freeSlots.contains(ws)) { -return; -} -for (Set used : _topIdToUsedSlots.values()) { -if (used.contains(ws)) { -return; -} -} -_freeSlots.add(ws); -} - -boolean assignInternal(WorkerSlot ws, String topId, boolean dontThrow) { -validateSlot(ws); -if (!_freeSlots.remove(ws)) { -if (dontThrow) { -return true; -} -throw new IllegalStateException("Assigning a slot that was not free " + ws); -} -Set usedSlots = _topIdToUsedSlots.get(topId); -if (usedSlots == null) { -usedSlots = new HashSet(); -_topIdToUsedSlots.put(topId, usedSlots); -} -usedSlots.add(ws); -return false; +return getUsedSlots(topId).size(); } /** * Free all slots on this node. This will update the Cluster too. */ -public void freeAllSlots() { + public void freeAllSlots() { --- End diff -- revert? > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49787051 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -247,19 +253,23 @@ private void freeMemory(double amount) { private void freeCPU(double amount) { LOG.debug("freeing {} CPU on node...avail CPU: {}", amount, getHostname(), _availCPU); -if ((_availCPU + amount) > getAvailableCpuResources()) { -LOG.warn("Freeing more CPU than there exists!"); +if ((_availCPU + amount) > getTotalCpuResources()) { +LOG.warn("Freeing more CPU than there exists! CPU trying to free: {} Total CPU on Node: {}", (_availMemory + amount), getTotalCpuResources()); return; } _availCPU += amount; } +/** + * get the amount of memory used by a worker + */ public double getMemoryUsedByWorker(WorkerSlot ws) { TopologyDetails topo = findTopologyUsingWorker(ws); if (topo == null) { return 0.0; } Collection execs = getExecutors(ws, _cluster); +LOG.info("getMemoryUsedByWorker executors: {}", execs); --- End diff -- Is it useful for this to be INFO level? This could be a lot of text to log. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098890#comment-15098890 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49787051 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java --- @@ -247,19 +253,23 @@ private void freeMemory(double amount) { private void freeCPU(double amount) { LOG.debug("freeing {} CPU on node...avail CPU: {}", amount, getHostname(), _availCPU); -if ((_availCPU + amount) > getAvailableCpuResources()) { -LOG.warn("Freeing more CPU than there exists!"); +if ((_availCPU + amount) > getTotalCpuResources()) { +LOG.warn("Freeing more CPU than there exists! CPU trying to free: {} Total CPU on Node: {}", (_availMemory + amount), getTotalCpuResources()); return; } _availCPU += amount; } +/** + * get the amount of memory used by a worker + */ public double getMemoryUsedByWorker(WorkerSlot ws) { TopologyDetails topo = findTopologyUsingWorker(ws); if (topo == null) { return 0.0; } Collection execs = getExecutors(ws, _cluster); +LOG.info("getMemoryUsedByWorker executors: {}", execs); --- End diff -- Is it useful for this to be INFO level? This could be a lot of text to log. > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098902#comment-15098902 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49787766 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/DefaultResourceAwareStrategy.java --- @@ -121,7 +116,7 @@ public SchedulingResult schedule(TopologyDetails td) { td.getTaskResourceReqList(exec), entry.getKey() }); WorkerSlot targetSlot = this.findWorkerForExec(exec, td, schedulerAssignmentMap); if (targetSlot != null) { -RAS_Node targetNode = this.idToNode(targetSlot.getNodeId()); +NodeDetails targetNode = this.idToNode(targetSlot.getNodeId()); --- End diff -- It seems there is a chunk of duplicate code here and in the for loop below. We are adding one line to both, but maybe it would be better to pull them out into a helper method to avoid accidental mistakes in the future? > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49788091 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/IStrategy.java --- @@ -36,7 +31,7 @@ /** * initialize prior to scheduling */ -public void prepare(Topologies topologies, Cluster cluster, MapuserMap, RAS_Nodes nodes); +public void prepare(ClusterStateData clusterStateData); --- End diff -- minor: public not needed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49787766 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/DefaultResourceAwareStrategy.java --- @@ -121,7 +116,7 @@ public SchedulingResult schedule(TopologyDetails td) { td.getTaskResourceReqList(exec), entry.getKey() }); WorkerSlot targetSlot = this.findWorkerForExec(exec, td, schedulerAssignmentMap); if (targetSlot != null) { -RAS_Node targetNode = this.idToNode(targetSlot.getNodeId()); +NodeDetails targetNode = this.idToNode(targetSlot.getNodeId()); --- End diff -- It seems there is a chunk of duplicate code here and in the for loop below. We are adding one line to both, but maybe it would be better to pull them out into a helper method to avoid accidental mistakes in the future? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098904#comment-15098904 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49788091 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/IStrategy.java --- @@ -36,7 +31,7 @@ /** * initialize prior to scheduling */ -public void prepare(Topologies topologies, Cluster cluster, MapuserMap, RAS_Nodes nodes); +public void prepare(ClusterStateData clusterStateData); --- End diff -- minor: public not needed > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49783662 --- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj --- @@ -701,11 +701,7 @@ ;; making a map from node+port to WorkerSlot with allocated resources node+port->slot (into {} (for [[[node port] [mem-on-heap mem-off-heap cpu]] worker->resources] {[node port] - (doto (WorkerSlot. node port) - (.allocateResource - mem-on-heap - mem-off-heap - cpu))})) + (doto (WorkerSlot. node port mem-on-heap mem-off-heap cpu))})) --- End diff -- Can we remove the `doto` here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....
Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/936#issuecomment-171777228 @roshannaik Thanks for the patch. I have merged this request to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098839#comment-15098839 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49783662 --- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj --- @@ -701,11 +701,7 @@ ;; making a map from node+port to WorkerSlot with allocated resources node+port->slot (into {} (for [[[node port] [mem-on-heap mem-off-heap cpu]] worker->resources] {[node port] - (doto (WorkerSlot. node port) - (.allocateResource - mem-on-heap - mem-off-heap - cpu))})) + (doto (WorkerSlot. node port mem-on-heap mem-off-heap cpu))})) --- End diff -- Can we remove the `doto` here? > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49783732 --- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj --- @@ -701,11 +701,7 @@ ;; making a map from node+port to WorkerSlot with allocated resources node+port->slot (into {} (for [[[node port] [mem-on-heap mem-off-heap cpu]] worker->resources] {[node port] - (doto (WorkerSlot. node port) - (.allocateResource - mem-on-heap - mem-off-heap - cpu))})) + (doto (WorkerSlot. node port mem-on-heap mem-off-heap cpu))})) --- End diff -- will remove --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1199) Create HDFS Spout
[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098840#comment-15098840 ] ASF GitHub Bot commented on STORM-1199: --- Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/936#issuecomment-171777228 @roshannaik Thanks for the patch. I have merged this request to master. > Create HDFS Spout > - > > Key: STORM-1199 > URL: https://issues.apache.org/jira/browse/STORM-1199 > Project: Apache Storm > Issue Type: New Feature >Reporter: Roshan Naik >Assignee: Roshan Naik > Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, > hdfs-spout.1.patch > > > Create an HDFS spout so that Storm can suck in data from files in a HDFS > directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1199) Create HDFS Spout
[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098838#comment-15098838 ] ASF GitHub Bot commented on STORM-1199: --- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/936 > Create HDFS Spout > - > > Key: STORM-1199 > URL: https://issues.apache.org/jira/browse/STORM-1199 > Project: Apache Storm > Issue Type: New Feature >Reporter: Roshan Naik >Assignee: Roshan Naik > Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, > hdfs-spout.1.patch > > > Create an HDFS spout so that Storm can suck in data from files in a HDFS > directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1199) Create HDFS Spout
[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt resolved STORM-1199. - Resolution: Fixed Fix Version/s: 1.0.0 > Create HDFS Spout > - > > Key: STORM-1199 > URL: https://issues.apache.org/jira/browse/STORM-1199 > Project: Apache Storm > Issue Type: New Feature >Reporter: Roshan Naik >Assignee: Roshan Naik > Fix For: 1.0.0 > > Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, > hdfs-spout.1.patch > > > Create an HDFS spout so that Storm can suck in data from files in a HDFS > directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098841#comment-15098841 ] ASF GitHub Bot commented on STORM-1450: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49783732 --- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj --- @@ -701,11 +701,7 @@ ;; making a map from node+port to WorkerSlot with allocated resources node+port->slot (into {} (for [[[node port] [mem-on-heap mem-off-heap cpu]] worker->resources] {[node port] - (doto (WorkerSlot. node port) - (.allocateResource - mem-on-heap - mem-off-heap - cpu))})) + (doto (WorkerSlot. node port mem-on-heap mem-off-heap cpu))})) --- End diff -- will remove > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49783928 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java --- @@ -662,22 +662,50 @@ public void setStatusMap(MapstatusMap) { this.status.putAll(statusMap); } -public void setResources(String topologyId, Double[] resources) { -this.resources.put(topologyId, resources); +/** + * Set the amount of resources used used by a topology. Used for displaying resource information on the UI + * @param topologyId + * @param resources describes the resources requested and assigned to topology in the following format in an array: + * {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu} + */ +public void setTopologyResources(String topologyId, Double[] resources) { --- End diff -- Can we replace `Double[]` with a class that is appropriate to the data? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098846#comment-15098846 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49783928 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java --- @@ -662,22 +662,50 @@ public void setStatusMap(MapstatusMap) { this.status.putAll(statusMap); } -public void setResources(String topologyId, Double[] resources) { -this.resources.put(topologyId, resources); +/** + * Set the amount of resources used used by a topology. Used for displaying resource information on the UI + * @param topologyId + * @param resources describes the resources requested and assigned to topology in the following format in an array: + * {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu} + */ +public void setTopologyResources(String topologyId, Double[] resources) { --- End diff -- Can we replace `Double[]` with a class that is appropriate to the data? > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098855#comment-15098855 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49784882 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/SchedulerAssignment.java --- @@ -55,4 +56,6 @@ public Set getExecutors(); public Set getSlots(); + +public MapgetSlotToExecutors(); --- End diff -- very minor: The interface is public, so it's redundant to mark each method declaration as public. > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49784882 --- Diff: storm-core/src/jvm/org/apache/storm/scheduler/SchedulerAssignment.java --- @@ -55,4 +56,6 @@ public Set getExecutors(); public Set getSlots(); + +public MapgetSlotToExecutors(); --- End diff -- very minor: The interface is public, so it's redundant to mark each method declaration as public. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1450: - Description: Code refactored: 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. Each RAS_Node will now be initialized with a map of all its assignments. Each RAS_Node will also figure out resources used and available. Removed unnecessary functions. 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly modify it 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so that the semantics of what a scheduling strategy should do will be more clear. Each scheduling strategy shouldn't be actually assigning anything. The strategy should only calculate a scheduling. Bug fixes: 1. Minor bug in displaying the assigned resources for a supervisor on the UI. The function updateSupervisorResources was placed in the wrong place 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that was done. was: Code refactored: 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. Each RAS_Node will now be initialized with a map of all its assignments. Each RAS_Node will also figure out resources used and available. 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly modify it 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so that the semantics of what a scheduling strategy should do will be more clear. Each scheduling strategy shouldn't be actually assigning anything. The strategy should only calculate a scheduling. Bug fixes: 1. Minor bug in displaying the assigned resources for a supervisor on the UI. The function updateSupervisorResources was placed in the wrong place 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that was done. > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1199) Create HDFS Spout
[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098799#comment-15098799 ] ASF GitHub Bot commented on STORM-1199: --- Github user roshannaik commented on the pull request: https://github.com/apache/storm/pull/936#issuecomment-171771850 As per request from @harshach, I have opened up STORM-1474 for addressing the pending minor review comments. Looks like @harshach is away, @Parth-Brahmbhatt could you take a look into committing it ? > Create HDFS Spout > - > > Key: STORM-1199 > URL: https://issues.apache.org/jira/browse/STORM-1199 > Project: Apache Storm > Issue Type: New Feature >Reporter: Roshan Naik >Assignee: Roshan Naik > Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, > hdfs-spout.1.patch > > > Create an HDFS spout so that Storm can suck in data from files in a HDFS > directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....
Github user roshannaik commented on the pull request: https://github.com/apache/storm/pull/936#issuecomment-171771850 As per request from @harshach, I have opened up STORM-1474 for addressing the pending minor review comments. Looks like @harshach is away, @Parth-Brahmbhatt could you take a look into committing it ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49781622 --- Diff: storm-core/test/jvm/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java --- @@ -939,6 +945,76 @@ public void TestEvictTopologyFromItself() { * If users are above his or her guarantee, check if topology eviction works correct */ @Test +public void Test() { --- End diff -- Rename? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098807#comment-15098807 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49781622 --- Diff: storm-core/test/jvm/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java --- @@ -939,6 +945,76 @@ public void TestEvictTopologyFromItself() { * If users are above his or her guarantee, check if topology eviction works correct */ @Test +public void Test() { --- End diff -- Rename? > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49781708 --- Diff: storm-core/test/jvm/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java --- @@ -1083,6 +1159,15 @@ public void TestOverGuaranteeEviction() { Assert.assertEquals("# of pending topologies", 0, rs.getUser("bobby").getTopologiesPending().size()); Assert.assertEquals("# of attempted topologies", 0, rs.getUser("bobby").getTopologiesAttempted().size()); Assert.assertEquals("# of invalid topologies", 0, rs.getUser("bobby").getTopologiesInvalid().size()); + +LOG.info("Assignments: {}", cluster.getAssignments()); +for (Map.Entryentry : cluster.getAssignments().entrySet()) { +LOG.info("Topology id: {}", entry.getKey()); +for(WorkerSlot target: entry.getValue().getSlots()) { +LOG.info("target resources onheap: {} offheap: {} cpu: {}", target.getAllocatedMemOnHeap(), target.getAllocatedMemOffHeap(), target.getAllocatedCpu()); +} + +} --- End diff -- Revert? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098808#comment-15098808 ] ASF GitHub Bot commented on STORM-1450: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1016#discussion_r49781708 --- Diff: storm-core/test/jvm/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java --- @@ -1083,6 +1159,15 @@ public void TestOverGuaranteeEviction() { Assert.assertEquals("# of pending topologies", 0, rs.getUser("bobby").getTopologiesPending().size()); Assert.assertEquals("# of attempted topologies", 0, rs.getUser("bobby").getTopologiesAttempted().size()); Assert.assertEquals("# of invalid topologies", 0, rs.getUser("bobby").getTopologiesInvalid().size()); + +LOG.info("Assignments: {}", cluster.getAssignments()); +for (Map.Entryentry : cluster.getAssignments().entrySet()) { +LOG.info("Topology id: {}", entry.getKey()); +for(WorkerSlot target: entry.getValue().getSlots()) { +LOG.info("target resources onheap: {} offheap: {} cpu: {}", target.getAllocatedMemOnHeap(), target.getAllocatedMemOffHeap(), target.getAllocatedCpu()); +} + +} --- End diff -- Revert? > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1199) Create HDFS Spout
[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098948#comment-15098948 ] Roshan Naik commented on STORM-1199: Thanks all for your input/feedback/reviews .. they were very useful. > Create HDFS Spout > - > > Key: STORM-1199 > URL: https://issues.apache.org/jira/browse/STORM-1199 > Project: Apache Storm > Issue Type: New Feature >Reporter: Roshan Naik >Assignee: Roshan Naik > Fix For: 1.0.0 > > Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, > hdfs-spout.1.patch > > > Create an HDFS spout so that Storm can suck in data from files in a HDFS > directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default
[ https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098958#comment-15098958 ] ASF GitHub Bot commented on STORM-1452: --- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1012#issuecomment-171793561 Not really specific to this pull request, but I missed it before when the profiling functionality was originally added. If profiling is enabled, we turn on the "Commercial Features" of the JVM: `worker.profiler.childopts: "-XX:+UnlockCommercialFeatures -XX:+FlightRecorder` The current Oracle Java license [1] states: >Some of the packages described in the Installation of Oracle Java SE Product Editions section above install Commercial Features that are restricted to Oracle Java SE Advanced, Oracle Java SE Advanced Desktop and/or Oracle Java SE Suite. **This means that even if you download an Oracle Java SE package for free under the Java BCLA, you must separately license from Oracle (and pay the appropriate license fee) the right to use any Commercial Features, described in Table 1-1 below, included in these packages.** If my interpretation of the license is correct, that means users must have purchased a license from Orace for each node that has profiling enabled. (Read the license [1] for more information and examples.) I think that's probably okay, but we may want to document that somewhere (`defaults.yaml` maybe?), so users don't accidentally run afoul of Oracle's licensing terms. [1] http://www.oracle.com/technetwork/java/javase/terms/products/index.html > Worker "profiler" actions broken by default > --- > > Key: STORM-1452 > URL: https://issues.apache.org/jira/browse/STORM-1452 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Blocker > > * The profiler script flight.bash is not packaged by default. > * The default options enable the Oracle specific flight-recorder that > requires a support subscription. > The option to enable the profiler should not be enabled by default. Other > actions such as worker restart, debugging, and heap can remain enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1012#issuecomment-171793561 Not really specific to this pull request, but I missed it before when the profiling functionality was originally added. If profiling is enabled, we turn on the "Commercial Features" of the JVM: `worker.profiler.childopts: "-XX:+UnlockCommercialFeatures -XX:+FlightRecorder` The current Oracle Java license [1] states: >Some of the packages described in the Installation of Oracle Java SE Product Editions section above install Commercial Features that are restricted to Oracle Java SE Advanced, Oracle Java SE Advanced Desktop and/or Oracle Java SE Suite. **This means that even if you download an Oracle Java SE package for free under the Java BCLA, you must separately license from Oracle (and pay the appropriate license fee) the right to use any Commercial Features, described in Table 1-1 below, included in these packages.** If my interpretation of the license is correct, that means users must have purchased a license from Orace for each node that has profiling enabled. (Read the license [1] for more information and examples.) I think that's probably okay, but we may want to document that somewhere (`defaults.yaml` maybe?), so users don't accidentally run afoul of Oracle's licensing terms. [1] http://www.oracle.com/technetwork/java/javase/terms/products/index.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1466: Move the org.apache.thrift7 namesp...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1007#discussion_r49791863 --- Diff: storm-core/pom.xml --- @@ -618,7 +618,7 @@ org.apache.thrift - org.apache.thrift7 + org.apache.storm.thrift --- End diff -- We should remove the old comment about backward compatibility. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1466) Move the org.apache.thrift7 namespace to something correct/sensible
[ https://issues.apache.org/jira/browse/STORM-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098970#comment-15098970 ] ASF GitHub Bot commented on STORM-1466: --- Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/1007#discussion_r49791863 --- Diff: storm-core/pom.xml --- @@ -618,7 +618,7 @@ org.apache.thrift - org.apache.thrift7 + org.apache.storm.thrift --- End diff -- We should remove the old comment about backward compatibility. > Move the org.apache.thrift7 namespace to something correct/sensible > --- > > Key: STORM-1466 > URL: https://issues.apache.org/jira/browse/STORM-1466 > Project: Apache Storm > Issue Type: Bug >Reporter: Kyle Nusbaum >Assignee: Kyle Nusbaum > > Thrift is still shaded to org.apache.thrift7 even though we are not using > Thrift 7. > We should also add something for temporary backwards compatibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1475) storm-elasticsearch should support ES 2.X
Aaron Dossett created STORM-1475: Summary: storm-elasticsearch should support ES 2.X Key: STORM-1475 URL: https://issues.apache.org/jira/browse/STORM-1475 Project: Apache Storm Issue Type: Improvement Components: storm-elasticsearch Affects Versions: 1.0.0 Reporter: Aaron Dossett -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1475) storm-elasticsearch should support ES 2.X
[ https://issues.apache.org/jira/browse/STORM-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Dossett reassigned STORM-1475: Assignee: Aaron Dossett > storm-elasticsearch should support ES 2.X > - > > Key: STORM-1475 > URL: https://issues.apache.org/jira/browse/STORM-1475 > Project: Apache Storm > Issue Type: Improvement > Components: storm-elasticsearch >Affects Versions: 1.0.0 >Reporter: Aaron Dossett >Assignee: Aaron Dossett > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1011#issuecomment-171796975 Personally, I don't think we need separate pull requests for each branch yet (master and 1.x have not diverged enough to cause conflicts), as it kind of confuses things in terms of +1s and comments. It might be easier to add a note requesting that it also be applied to another branch, e.g. "I would also like to see this applied to the X branch." If for some reason the patch couldn't be applied to the other branch(es), the committer doing the merge could either resolve the conflicts or request an additional pull request against the conflicted branch. Thoughts? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
[ https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098979#comment-15098979 ] ASF GitHub Bot commented on STORM-1470: --- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1011#issuecomment-171796975 Personally, I don't think we need separate pull requests for each branch yet (master and 1.x have not diverged enough to cause conflicts), as it kind of confuses things in terms of +1s and comments. It might be easier to add a note requesting that it also be applied to another branch, e.g. "I would also like to see this applied to the X branch." If for some reason the patch couldn't be applied to the other branch(es), the committer doing the merge could either resolve the conflicts or request an additional pull request against the conflicted branch. Thoughts? > org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication > -- > > Key: STORM-1470 > URL: https://issues.apache.org/jira/browse/STORM-1470 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Derek Dagit >Assignee: Derek Dagit >Priority: Blocker > > {noformat} > 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for > /favicon.ico > java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64 > at > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) > at > org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) > at > org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) > at > org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at > org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) > at > org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) > at > org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) > at > org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) > at > org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > at > org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) > {noformat} > We already shade commons-codec:commons-codec, but we don't apply that shading > to org.apache.hadoop:hadoop-auth. -- This message was sent by Atlassian JIRA (v6.3.4#6332)