[jira] [Commented] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig

2016-01-14 Thread Garrick Dasbach (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098242#comment-15098242
 ] 

Garrick Dasbach commented on STORM-1471:


Not sure how far out the JStorm port is, but in the interim, maybe it would be 
sufficient to make the Kafka fetch asynchronous.  That way we can allow 
long-polling without blocking the spout.  

> Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
> -
>
> Key: STORM-1471
> URL: https://issues.apache.org/jira/browse/STORM-1471
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Garrick Dasbach
>
> We currently have an issue in our storm cluster where our Kafka brokers are 
> under heavy load due to too many fetch requests from storm.  We've narrowed 
> the problem to the way Fetch Requests are build in KafkaUtils.  When using 
> the FetchRequestBuilder, storm provides overrides for all the properties 
> except minBytes.  The default for that field is 0 (even though the Kafka 
> default for the high-level consumer is 1).  When paired with a maxWait > 0, 
> this creates a situation where the broker can immediately return a response 
> without waiting (due to minBytes 0).  This puts a heavy load on the brokers 
> and defeats the purpose of any long polling.
> By making this a SpoutConfig option, it will allow the user to set that as 
> appropriate for their situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: JStorm CGroup

2016-01-14 Thread Bobby Evans
I would love to see true support for mesos, YARN, openstack, etc. added, but I 
also see stand alone mode offering a lot more flexibility, especially in the 
area of scheduling, than a two level scheduler can currently offer.  It is on 
my roadmap to look into after the JStorm migration (just started), Resource 
Aware Scheduling (almost done needs testing and better isolation), and adding 
in automatic elasticity around topology specified SLAs (working with a few 
researchers around some prototypes in this area).

To be able to support running on other cluster technologies in a proper way we 
need to provide plugability in a few different places.
First we need a way for a scheduler/cluster to request topology specific 
dedicated resources, and for nimbus to provision, manage, monitor, and ideally 
resize (for elasticity) those resources.  With security and resource aware 
scheduling, we need these external requests to be on a per topology bases, not 
bolted on like they are now.  This would also necessitate the schedulers being 
updated so that they could take advantage of these new APIs requesting external 
resources either when a topology explicitly asks to be on a given external 
resource, or optionally when dedicated resources are no longer available and 
the topology has specified the proper configurations/credentials to allow it to 
run using those external resources.

That handles scheduling, but there are some additional features that storm 
offers which other systems don't yet offer, and many never will.  For example 
the storm blob store API is similar to the dist cache in YARN, but it we can do 
in place replacement without relaunching.  We also favor fast fail and I don't 
think all of these types of clusters will nor should offer the process 
monitoring and re-spawning needed for it.  As such we would need some sort of a 
supervisor that would also run under YARN/mesos, etc to provide this extra 
functionality.  I have not totally thought about all of what it would need from 
a plugability standpoint to make that work.  There is also the logviewer which 
does more then just logs, so we would need some pluggable way to be able to 
point people to where their logs/artifacts are, and to monitor the resource 
usage of the logs (perhaps that part should move off to the supervisor). All of 
that seems like a lot more work compared to providing a pluggable interface in 
the supervisor that would allow for it to provision, manage, monitor, and again 
possibly resize, local workers.  In fact I see a lot of potential overlap 
between the two of them and the pluggability that would be needed in the 
supervisor for running on mesos, YARN, etc.

- Bobby 

On Thursday, January 14, 2016 12:39 AM, Erik Weathers 
 wrote:
 

 Perhaps rather than just bolting on "cgroup support", we could instead open
a dialogue about having Mesos support be a core feature of Storm.

The current integration is a bit unwieldy & hackish at the moment, arising
from the conflicting natures of Mesos and Storm w.r.t. scheduling of
resources.  i.e., Storm assumes you have existing "slots" for running
workers on, whereas Mesos is more dynamic, requiring frameworks that run on
top of it to tell Mesos just how many resources (CPUs, Memory, etc.) are
needed by the framework's tasks.

One example of an issue with Storm-on-Mesos:  the Storm logviewer is
completely busted when you are using Mesos, I filed a ticket with a
description of the issue and proposed modifications to allow it to function:

  - https://issues.apache.org/jira/browse/STORM-1342

Furthermore, there are fundamental behaviors in Storm that don't mesh well
with Mesos:

  - the interfaces of INimbus (allSlotsAvailableForScheduling(),
  assignSlots(), getForcedScheduler(), etc.) make it difficult to create an
  ideal Mesos integration framework, since they don't allow the Mesos
  integration code to *really* know what's going on from the Nimbus's
  perspective. e.g.,
      - knowing which topologies & how many workers need to be scheduled at
      any given moment.
      - since the integration code cannot know what is actually needed to
      be run when it receives offers from Mesos, it just hoards those offers,
      leading to resource starvation in the Mesos cluster.
  - the "fallback" behavior of allowing the topology to settle for having
  less worker processes than requested should be disable-able.  For carefully
  tuned topologies it is quite bad to run on less than the expected number of
  worker processes.
      - also, this behavior endangers the idea of having the Mesos
      integration code *only* hoard Mesos offers after a successful round-trip
      through the allSlotsAvailableForScheduling() polling calls (i.e., only
      hoard when we know there are pending topologies).  It's dangerous because
      while we wait for another call to allSlotsAvailableForScheduling(), the
      Nimbus may have decided that it's okie dokie 

[jira] [Created] (STORM-1473) Enable search on the daemon log

2016-01-14 Thread Zhuo Liu (JIRA)
Zhuo Liu created STORM-1473:
---

 Summary: Enable search on the daemon log
 Key: STORM-1473
 URL: https://issues.apache.org/jira/browse/STORM-1473
 Project: Apache Storm
  Issue Type: Improvement
  Components: storm-core
Affects Versions: 1.0.0
Reporter: Zhuo Liu
Priority: Minor
 Fix For: 1.0.0


Search works fine for worker log.
It would be nice that we can also enable this good feature for the daemon logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1473] enable log search for daemon logs

2016-01-14 Thread zhuoliu
GitHub user zhuoliu opened a pull request:

https://github.com/apache/storm/pull/1015

[STORM-1473] enable log search for daemon logs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhuoliu/storm 1473

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1015.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1015


commit c9cdcbc76a7b65b03dce09541f2df40a97a6e1fd
Author: zhuol 
Date:   2016-01-14T17:46:49Z

[STORM-1473] enable log search for daemon logs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1473) Enable search on the daemon log

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098496#comment-15098496
 ] 

ASF GitHub Bot commented on STORM-1473:
---

GitHub user zhuoliu opened a pull request:

https://github.com/apache/storm/pull/1015

[STORM-1473] enable log search for daemon logs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhuoliu/storm 1473

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1015.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1015


commit c9cdcbc76a7b65b03dce09541f2df40a97a6e1fd
Author: zhuol 
Date:   2016-01-14T17:46:49Z

[STORM-1473] enable log search for daemon logs




> Enable search on the daemon log
> ---
>
> Key: STORM-1473
> URL: https://issues.apache.org/jira/browse/STORM-1473
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Zhuo Liu
>Priority: Minor
> Fix For: 1.0.0
>
>
> Search works fine for worker log.
> It would be nice that we can also enable this good feature for the daemon 
> logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1449) Fix Kafka spout to maintain backward compatibility to pass byte[] instead of ByteBuffer

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097851#comment-15097851
 ] 

ASF GitHub Bot commented on STORM-1449:
---

Github user priyank5485 commented on the pull request:

https://github.com/apache/storm/pull/994#issuecomment-171576791
  
Just ran in to an issue where this fix does not work. Correct me if i am 
wrong. But a fat storm jar that is compiled with older version of storm-kafka 
will still use byte[] and the newer version of storm now uses ByteBuffer. Hence 
it throws an exception at runtime if storm is upgraded. I think upgrade of 
storm should still work with older topology jars. 


> Fix Kafka spout to maintain backward compatibility to pass byte[] instead of 
> ByteBuffer
> ---
>
> Key: STORM-1449
> URL: https://issues.apache.org/jira/browse/STORM-1449
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
> Fix For: 1.0.0
>
>
> STORM-1220 introduced some changes where in one place it passes ByteBuffer as 
> is instead of byte[]. Existing bolts that expects a byte[] fails due to this 
> with "java.lang.RuntimeException: java.lang.ClassCastException: 
> java.nio.HeapByteBuffer cannot be cast to [B "



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1449] Fix Kafka spout to maintain backw...

2016-01-14 Thread priyank5485
Github user priyank5485 commented on the pull request:

https://github.com/apache/storm/pull/994#issuecomment-171576791
  
Just ran in to an issue where this fix does not work. Correct me if i am 
wrong. But a fat storm jar that is compiled with older version of storm-kafka 
will still use byte[] and the newer version of storm now uses ByteBuffer. Hence 
it throws an exception at runtime if storm is upgraded. I think upgrade of 
storm should still work with older topology jars. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...

2016-01-14 Thread d2r
Github user d2r closed the pull request at:

https://github.com/apache/storm/pull/1011


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...

2016-01-14 Thread d2r
Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/1011#issuecomment-171802477
  
See #1010


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099024#comment-15099024
 ] 

ASF GitHub Bot commented on STORM-1452:
---

Github user d2r closed the pull request at:

https://github.com/apache/storm/pull/1013


> Worker "profiler" actions broken by default
> ---
>
> Key: STORM-1452
> URL: https://issues.apache.org/jira/browse/STORM-1452
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> * The profiler script flight.bash is not packaged by default.
> * The default options enable the Oracle specific flight-recorder that 
> requires a support subscription.
> The option to enable the profiler should not be enabled by default.  Other 
> actions such as worker restart, debugging, and heap can remain enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099023#comment-15099023
 ] 

ASF GitHub Bot commented on STORM-1452:
---

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/1013#issuecomment-171802210
  
See #1012 


> Worker "profiler" actions broken by default
> ---
>
> Key: STORM-1452
> URL: https://issues.apache.org/jira/browse/STORM-1452
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> * The profiler script flight.bash is not packaged by default.
> * The default options enable the Oracle specific flight-recorder that 
> requires a support subscription.
> The option to enable the profiler should not be enabled by default.  Other 
> actions such as worker restart, debugging, and heap can remain enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...

2016-01-14 Thread ptgoetz
Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1012#issuecomment-171816513
  
Works for me. Thanks @d2r.

+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...

2016-01-14 Thread d2r
Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/1012#issuecomment-171802145
  
Yes, I can add something.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099022#comment-15099022
 ] 

ASF GitHub Bot commented on STORM-1452:
---

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/1012#issuecomment-171802145
  
Yes, I can add something.


> Worker "profiler" actions broken by default
> ---
>
> Key: STORM-1452
> URL: https://issues.apache.org/jira/browse/STORM-1452
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> * The profiler script flight.bash is not packaged by default.
> * The default options enable the Oracle specific flight-recorder that 
> requires a support subscription.
> The option to enable the profiler should not be enabled by default.  Other 
> actions such as worker restart, debugging, and heap can remain enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...

2016-01-14 Thread d2r
Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/1013#issuecomment-171802210
  
See #1012 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...

2016-01-14 Thread d2r
Github user d2r closed the pull request at:

https://github.com/apache/storm/pull/1013


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/1016#issuecomment-171802669
  
Mostly minor comments, otherwise looks ok to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099034#comment-15099034
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/1016#issuecomment-171802669
  
Mostly minor comments, otherwise looks ok to me.


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099032#comment-15099032
 ] 

ASF GitHub Bot commented on STORM-1470:
---

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/1011#issuecomment-171802477
  
See #1010


> org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
> --
>
> Key: STORM-1470
> URL: https://issues.apache.org/jira/browse/STORM-1470
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> {noformat}
> 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for 
> /favicon.ico
> java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64
> at 
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We already shade commons-codec:commons-codec, but we don't apply that shading 
> to org.apache.hadoop:hadoop-auth.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099033#comment-15099033
 ] 

ASF GitHub Bot commented on STORM-1470:
---

Github user d2r closed the pull request at:

https://github.com/apache/storm/pull/1011


> org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
> --
>
> Key: STORM-1470
> URL: https://issues.apache.org/jira/browse/STORM-1470
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> {noformat}
> 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for 
> /favicon.ico
> java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64
> at 
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We already shade commons-codec:commons-codec, but we don't apply that shading 
> to org.apache.hadoop:hadoop-auth.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...

2016-01-14 Thread d2r
Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/1012#issuecomment-171805811
  
@ptgoetz , added.  Do you think it is OK?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099067#comment-15099067
 ] 

ASF GitHub Bot commented on STORM-1452:
---

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/1012#issuecomment-171805811
  
@ptgoetz , added.  Do you think it is OK?


> Worker "profiler" actions broken by default
> ---
>
> Key: STORM-1452
> URL: https://issues.apache.org/jira/browse/STORM-1452
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> * The profiler script flight.bash is not packaged by default.
> * The default options enable the Oracle specific flight-recorder that 
> requires a support subscription.
> The option to enable the profiler should not be enabled by default.  Other 
> actions such as worker restart, debugging, and heap can remain enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101195#comment-15101195
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49821028
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -43,32 +43,84 @@
  */
 public class RAS_Node {
 private static final Logger LOG = 
LoggerFactory.getLogger(RAS_Node.class);
-private Map _topIdToUsedSlots = new 
HashMap();
-private Set _freeSlots = new HashSet();
+
+//A map consisting of all workers on the node.
+//The key of the map is the worker id and the value is the 
corresponding workerslot object
+Map _slots = new HashMap ();
+
+// A map describing which topologies are using which slots on this 
node.  The format of the map is the following:
+// {TopologyId -> {WorkerId -> {Executors}}}
+private Map> 
_topIdToUsedSlots = new HashMap>();
+
 private final String _nodeId;
 private String _hostname;
 private boolean _isAlive;
 private SupervisorDetails _sup;
-private Double _availMemory;
-private Double _availCPU;
-private Cluster _cluster;
-private Topologies _topologies;
+private Double _availMemory = 0.0;
+private Double _availCPU = 0.0;
+private final Cluster _cluster;
+private final Topologies _topologies;
 
-public RAS_Node(String nodeId, Set allPorts, boolean isAlive,
-SupervisorDetails sup, Cluster cluster, Topologies 
topologies) {
+public RAS_Node(String nodeId, SupervisorDetails sup, Cluster cluster, 
Topologies topologies, Map workerIdToWorker, Map> assignmentMap) {
+//Node ID and supervisor ID are the same.
 _nodeId = nodeId;
-_isAlive = isAlive;
-if (_isAlive && allPorts != null) {
-for (int port : allPorts) {
-_freeSlots.add(new WorkerSlot(_nodeId, port));
-}
-_sup = sup;
+if (sup == null) {
+_isAlive = false;
+} else {
+_isAlive = !cluster.isBlackListed(_nodeId);
+}
+
+_cluster = cluster;
+_topologies = topologies;
+
+// initialize slots for this node
+if (workerIdToWorker != null) {
+_slots = workerIdToWorker;
+}
+
+//initialize assignment map
+if (assignmentMap != null) {
+_topIdToUsedSlots = assignmentMap;
+}
+
+//check if node is alive
+if (_isAlive && sup != null) {
 _hostname = sup.getHost();
+_sup = sup;
 _availMemory = getTotalMemoryResources();
 _availCPU = getTotalCpuResources();
+
+LOG.debug("Found a {} Node {} {}",
+_isAlive ? "living" : "dead", _nodeId, 
sup.getAllPorts());
+LOG.debug("resources_mem: {}, resources_CPU: {}", 
sup.getTotalMemory(), sup.getTotalCPU());
+//intialize resource usages on node
+intializeResources();
+}
+}
+
+/**
+ * intializes resource usages on node
--- End diff --

will fix


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. 

[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread jerrypeng
Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49821059
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -103,92 +182,37 @@ public boolean isAlive() {
 }
 
 public boolean isTotallyFree() {
-return _topIdToUsedSlots.isEmpty();
+return getUsedSlots().isEmpty();
--- End diff --

its used in resource_aware_scheduler_test.clj


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread jerrypeng
Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49821028
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -43,32 +43,84 @@
  */
 public class RAS_Node {
 private static final Logger LOG = 
LoggerFactory.getLogger(RAS_Node.class);
-private Map _topIdToUsedSlots = new 
HashMap();
-private Set _freeSlots = new HashSet();
+
+//A map consisting of all workers on the node.
+//The key of the map is the worker id and the value is the 
corresponding workerslot object
+Map _slots = new HashMap ();
+
+// A map describing which topologies are using which slots on this 
node.  The format of the map is the following:
+// {TopologyId -> {WorkerId -> {Executors}}}
+private Map> 
_topIdToUsedSlots = new HashMap>();
+
 private final String _nodeId;
 private String _hostname;
 private boolean _isAlive;
 private SupervisorDetails _sup;
-private Double _availMemory;
-private Double _availCPU;
-private Cluster _cluster;
-private Topologies _topologies;
+private Double _availMemory = 0.0;
+private Double _availCPU = 0.0;
+private final Cluster _cluster;
+private final Topologies _topologies;
 
-public RAS_Node(String nodeId, Set allPorts, boolean isAlive,
-SupervisorDetails sup, Cluster cluster, Topologies 
topologies) {
+public RAS_Node(String nodeId, SupervisorDetails sup, Cluster cluster, 
Topologies topologies, Map workerIdToWorker, Map> assignmentMap) {
+//Node ID and supervisor ID are the same.
 _nodeId = nodeId;
-_isAlive = isAlive;
-if (_isAlive && allPorts != null) {
-for (int port : allPorts) {
-_freeSlots.add(new WorkerSlot(_nodeId, port));
-}
-_sup = sup;
+if (sup == null) {
+_isAlive = false;
+} else {
+_isAlive = !cluster.isBlackListed(_nodeId);
+}
+
+_cluster = cluster;
+_topologies = topologies;
+
+// initialize slots for this node
+if (workerIdToWorker != null) {
+_slots = workerIdToWorker;
+}
+
+//initialize assignment map
+if (assignmentMap != null) {
+_topIdToUsedSlots = assignmentMap;
+}
+
+//check if node is alive
+if (_isAlive && sup != null) {
 _hostname = sup.getHost();
+_sup = sup;
 _availMemory = getTotalMemoryResources();
 _availCPU = getTotalCpuResources();
+
+LOG.debug("Found a {} Node {} {}",
+_isAlive ? "living" : "dead", _nodeId, 
sup.getAllPorts());
+LOG.debug("resources_mem: {}, resources_CPU: {}", 
sup.getTotalMemory(), sup.getTotalCPU());
+//intialize resource usages on node
+intializeResources();
+}
+}
+
+/**
+ * intializes resource usages on node
--- End diff --

will fix


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread jerrypeng
Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49820988
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/SchedulerAssignment.java ---
@@ -55,4 +56,6 @@
 public Set getExecutors();
 
 public Set getSlots();
+
+public Map 
getSlotToExecutors();
--- End diff --

@d2r should I delete all of the public declarations for all the methods in 
this class then?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101194#comment-15101194
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49820997
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/WorkerSlot.java ---
@@ -40,11 +47,8 @@ public int getPort() {
 return port;
 }
 
-public WorkerSlot allocateResource(double memOnHeap, double 
memOffHeap, double cpu) {
-this.memOnHeap += memOnHeap;
-this.memOffHeap += memOffHeap;
-this.cpu += cpu;
-return this;
+public String getId() {
+return this.getNodeId() + ":" + this.getPort();
--- End diff --

will remove


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101193#comment-15101193
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49820988
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/SchedulerAssignment.java ---
@@ -55,4 +56,6 @@
 public Set getExecutors();
 
 public Set getSlots();
+
+public Map 
getSlotToExecutors();
--- End diff --

@d2r should I delete all of the public declarations for all the methods in 
this class then?


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread jerrypeng
Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49820997
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/WorkerSlot.java ---
@@ -40,11 +47,8 @@ public int getPort() {
 return port;
 }
 
-public WorkerSlot allocateResource(double memOnHeap, double 
memOffHeap, double cpu) {
-this.memOnHeap += memOnHeap;
-this.memOffHeap += memOffHeap;
-this.cpu += cpu;
-return this;
+public String getId() {
+return this.getNodeId() + ":" + this.getPort();
--- End diff --

will remove


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread jerrypeng
Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49821516
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/DefaultResourceAwareStrategy.java
 ---
@@ -121,7 +116,7 @@ public SchedulingResult schedule(TopologyDetails td) {
 td.getTaskResourceReqList(exec), entry.getKey() });
 WorkerSlot targetSlot = this.findWorkerForExec(exec, 
td, schedulerAssignmentMap);
 if (targetSlot != null) {
-RAS_Node targetNode = 
this.idToNode(targetSlot.getNodeId());
+NodeDetails targetNode = 
this.idToNode(targetSlot.getNodeId());
--- End diff --

will refactor


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Created] (STORM-1476) Filter -c options from args and add them as part of storm.options

2016-01-14 Thread Satish Duggana (JIRA)
Satish Duggana created STORM-1476:
-

 Summary: Filter -c options from args and add them as part of 
storm.options
 Key: STORM-1476
 URL: https://issues.apache.org/jira/browse/STORM-1476
 Project: Apache Storm
  Issue Type: Bug
Reporter: Satish Duggana
Assignee: Satish Duggana






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101212#comment-15101212
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49821516
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/DefaultResourceAwareStrategy.java
 ---
@@ -121,7 +116,7 @@ public SchedulingResult schedule(TopologyDetails td) {
 td.getTaskResourceReqList(exec), entry.getKey() });
 WorkerSlot targetSlot = this.findWorkerForExec(exec, 
td, schedulerAssignmentMap);
 if (targetSlot != null) {
-RAS_Node targetNode = 
this.idToNode(targetSlot.getNodeId());
+NodeDetails targetNode = 
this.idToNode(targetSlot.getNodeId());
--- End diff --

will refactor


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101197#comment-15101197
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49821059
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -103,92 +182,37 @@ public boolean isAlive() {
 }
 
 public boolean isTotallyFree() {
-return _topIdToUsedSlots.isEmpty();
+return getUsedSlots().isEmpty();
--- End diff --

its used in resource_aware_scheduler_test.clj


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101198#comment-15101198
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49821063
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -103,92 +182,37 @@ public boolean isAlive() {
 }
 
 public boolean isTotallyFree() {
-return _topIdToUsedSlots.isEmpty();
+return getUsedSlots().isEmpty();
 }
 
 public int totalSlotsFree() {
-return _freeSlots.size();
+return getFreeSlots().size();
 }
 
 public int totalSlotsUsed() {
-int total = 0;
-for (Set slots : _topIdToUsedSlots.values()) {
-total += slots.size();
-}
-return total;
+return getUsedSlots().size();
 }
 
 public int totalSlots() {
-return totalSlotsFree() + totalSlotsUsed();
+return _slots.size();
 }
 
 public int totalSlotsUsed(String topId) {
-int total = 0;
-Set slots = _topIdToUsedSlots.get(topId);
-if (slots != null) {
-total = slots.size();
-}
-return total;
-}
-
-private void validateSlot(WorkerSlot ws) {
-if (!_nodeId.equals(ws.getNodeId())) {
-throw new IllegalArgumentException(
-"Trying to add a slot to the wrong node " + ws +
-" is not a part of " + _nodeId);
-}
-}
-
-void addOrphanedSlot(WorkerSlot ws) {
-if (_isAlive) {
-throw new IllegalArgumentException("Orphaned Slots " +
-"only are allowed on dead nodes.");
-}
-validateSlot(ws);
-if (_freeSlots.contains(ws)) {
-return;
-}
-for (Set used : _topIdToUsedSlots.values()) {
-if (used.contains(ws)) {
-return;
-}
-}
-_freeSlots.add(ws);
-}
-
-boolean assignInternal(WorkerSlot ws, String topId, boolean dontThrow) 
{
-validateSlot(ws);
-if (!_freeSlots.remove(ws)) {
-if (dontThrow) {
-return true;
-}
-throw new IllegalStateException("Assigning a slot that was not 
free " + ws);
-}
-Set usedSlots = _topIdToUsedSlots.get(topId);
-if (usedSlots == null) {
-usedSlots = new HashSet();
-_topIdToUsedSlots.put(topId, usedSlots);
-}
-usedSlots.add(ws);
-return false;
+return getUsedSlots(topId).size();
 }
 
 /**
  * Free all slots on this node.  This will update the Cluster too.
  */
-public void freeAllSlots() {
+ public void freeAllSlots() {
--- End diff --

will fix


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread jerrypeng
Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49821113
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -247,19 +253,23 @@ private void freeMemory(double amount) {
 
 private void freeCPU(double amount) {
 LOG.debug("freeing {} CPU on node...avail CPU: {}", amount, 
getHostname(), _availCPU);
-if ((_availCPU + amount) > getAvailableCpuResources()) {
-LOG.warn("Freeing more CPU than there exists!");
+if ((_availCPU + amount) > getTotalCpuResources()) {
+LOG.warn("Freeing more CPU than there exists! CPU trying to 
free: {} Total CPU on Node: {}", (_availMemory + amount), 
getTotalCpuResources());
 return;
 }
 _availCPU += amount;
 }
 
+/**
+ * get the amount of memory used by a worker
+ */
 public double getMemoryUsedByWorker(WorkerSlot ws) {
 TopologyDetails topo = findTopologyUsingWorker(ws);
 if (topo == null) {
 return 0.0;
 }
 Collection execs = getExecutors(ws, _cluster);
+LOG.info("getMemoryUsedByWorker executors: {}", execs);
--- End diff --

will get rid of


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread jerrypeng
Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49821063
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -103,92 +182,37 @@ public boolean isAlive() {
 }
 
 public boolean isTotallyFree() {
-return _topIdToUsedSlots.isEmpty();
+return getUsedSlots().isEmpty();
 }
 
 public int totalSlotsFree() {
-return _freeSlots.size();
+return getFreeSlots().size();
 }
 
 public int totalSlotsUsed() {
-int total = 0;
-for (Set slots : _topIdToUsedSlots.values()) {
-total += slots.size();
-}
-return total;
+return getUsedSlots().size();
 }
 
 public int totalSlots() {
-return totalSlotsFree() + totalSlotsUsed();
+return _slots.size();
 }
 
 public int totalSlotsUsed(String topId) {
-int total = 0;
-Set slots = _topIdToUsedSlots.get(topId);
-if (slots != null) {
-total = slots.size();
-}
-return total;
-}
-
-private void validateSlot(WorkerSlot ws) {
-if (!_nodeId.equals(ws.getNodeId())) {
-throw new IllegalArgumentException(
-"Trying to add a slot to the wrong node " + ws +
-" is not a part of " + _nodeId);
-}
-}
-
-void addOrphanedSlot(WorkerSlot ws) {
-if (_isAlive) {
-throw new IllegalArgumentException("Orphaned Slots " +
-"only are allowed on dead nodes.");
-}
-validateSlot(ws);
-if (_freeSlots.contains(ws)) {
-return;
-}
-for (Set used : _topIdToUsedSlots.values()) {
-if (used.contains(ws)) {
-return;
-}
-}
-_freeSlots.add(ws);
-}
-
-boolean assignInternal(WorkerSlot ws, String topId, boolean dontThrow) 
{
-validateSlot(ws);
-if (!_freeSlots.remove(ws)) {
-if (dontThrow) {
-return true;
-}
-throw new IllegalStateException("Assigning a slot that was not 
free " + ws);
-}
-Set usedSlots = _topIdToUsedSlots.get(topId);
-if (usedSlots == null) {
-usedSlots = new HashSet();
-_topIdToUsedSlots.put(topId, usedSlots);
-}
-usedSlots.add(ws);
-return false;
+return getUsedSlots(topId).size();
 }
 
 /**
  * Free all slots on this node.  This will update the Cluster too.
  */
-public void freeAllSlots() {
+ public void freeAllSlots() {
--- End diff --

will fix


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101200#comment-15101200
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49821113
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -247,19 +253,23 @@ private void freeMemory(double amount) {
 
 private void freeCPU(double amount) {
 LOG.debug("freeing {} CPU on node...avail CPU: {}", amount, 
getHostname(), _availCPU);
-if ((_availCPU + amount) > getAvailableCpuResources()) {
-LOG.warn("Freeing more CPU than there exists!");
+if ((_availCPU + amount) > getTotalCpuResources()) {
+LOG.warn("Freeing more CPU than there exists! CPU trying to 
free: {} Total CPU on Node: {}", (_availMemory + amount), 
getTotalCpuResources());
 return;
 }
 _availCPU += amount;
 }
 
+/**
+ * get the amount of memory used by a worker
+ */
 public double getMemoryUsedByWorker(WorkerSlot ws) {
 TopologyDetails topo = findTopologyUsingWorker(ws);
 if (topo == null) {
 return 0.0;
 }
 Collection execs = getExecutors(ws, _cluster);
+LOG.info("getMemoryUsedByWorker executors: {}", execs);
--- End diff --

will get rid of


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15099123#comment-15099123
 ] 

ASF GitHub Bot commented on STORM-1452:
---

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1012#issuecomment-171816513
  
Works for me. Thanks @d2r.

+1


> Worker "profiler" actions broken by default
> ---
>
> Key: STORM-1452
> URL: https://issues.apache.org/jira/browse/STORM-1452
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> * The profiler script flight.bash is not packaged by default.
> * The default options enable the Oracle specific flight-recorder that 
> requires a support subscription.
> The option to enable the profiler should not be enabled by default.  Other 
> actions such as worker restart, debugging, and heap can remain enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1476) Filter -c options from args and add them as part of storm.options

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101401#comment-15101401
 ] 

ASF GitHub Bot commented on STORM-1476:
---

GitHub user satishd opened a pull request:

https://github.com/apache/storm/pull/1017

STORM-1476 Filter -c options from args and add them as part of storm.options

…options

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/satishd/storm STORM-1476

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1017.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1017


commit 10b0d5842a12dbb772fb97fd26271d395396aa31
Author: Satish Duggana 
Date:   2016-01-15T07:52:04Z

STORM-1476 Filter -c options from args and add them as part of storm.options




> Filter -c options from args and add them as part of storm.options
> -
>
> Key: STORM-1476
> URL: https://issues.apache.org/jira/browse/STORM-1476
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Satish Duggana
>Assignee: Satish Duggana
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: STORM-1476 Filter -c options from args and add...

2016-01-14 Thread satishd
GitHub user satishd opened a pull request:

https://github.com/apache/storm/pull/1017

STORM-1476 Filter -c options from args and add them as part of storm.options

…options

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/satishd/storm STORM-1476

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1017.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1017


commit 10b0d5842a12dbb772fb97fd26271d395396aa31
Author: Satish Duggana 
Date:   2016-01-15T07:52:04Z

STORM-1476 Filter -c options from args and add them as part of storm.options




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: Storm 1450

2016-01-14 Thread jerrypeng
GitHub user jerrypeng opened a pull request:

https://github.com/apache/storm/pull/1016

Storm 1450

Code refactored:
1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. 
Each RAS_Node will now be initialized with a map of all its assignments. Each 
RAS_Node will also figure out resources used and available. Removed unnecessary 
functions.
2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
modify it
3. Added a wrapping layer for RAS_Node to feed into scheduling strategies 
so that the semantics of what a scheduling strategy should do will be more 
clear. Each scheduling strategy shouldn't be actually assigning anything. The 
strategy should only calculate a scheduling.

Bug fixes:
1. Minor bug in displaying the assigned resources for a supervisor on the 
UI. The function updateSupervisorResources was placed in the wrong place
2. Minor bug fix in freeing memory in RAS_Node there was some wrong math 
that was done.

Added Additional unit tests to test

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerrypeng/storm STORM-1450

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1016.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1016


commit 829ea117c8b382ef72d3c5df841534b89f06dffa
Author: Boyang Jerry Peng 
Date:   2016-01-13T18:28:38Z

[STORM-1450] - Fix bugs and refactor code in ResourceAwareScheduler

commit 688366902d98a7fdc17d854a33ff975c6a60ccab
Author: Boyang Jerry Peng 
Date:   2016-01-13T20:13:54Z

adding an additional test and cleaning up




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Updated] (STORM-1474) Address remaining minor review comments for STORM-1199

2016-01-14 Thread Roshan Naik (JIRA)

 [ 
https://issues.apache.org/jira/browse/STORM-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roshan Naik updated STORM-1474:
---
Issue Type: Sub-task  (was: Bug)
Parent: STORM-1199

> Address remaining minor review comments for STORM-1199
> --
>
> Key: STORM-1474
> URL: https://issues.apache.org/jira/browse/STORM-1474
> Project: Apache Storm
>  Issue Type: Sub-task
>  Components: storm-hdfs
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>Priority: Minor
>
> Address the last few pending review comments  from 
> https://github.com/apache/storm/pull/936



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (STORM-1474) Address remaining minor review comments for STORM-1199

2016-01-14 Thread Roshan Naik (JIRA)
Roshan Naik created STORM-1474:
--

 Summary: Address remaining minor review comments for STORM-1199
 Key: STORM-1474
 URL: https://issues.apache.org/jira/browse/STORM-1474
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-hdfs
Reporter: Roshan Naik
Assignee: Roshan Naik
Priority: Minor


Address the last few pending review comments  from 
https://github.com/apache/storm/pull/936



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098862#comment-15098862
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49785320
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/WorkerSlot.java ---
@@ -40,11 +47,8 @@ public int getPort() {
 return port;
 }
 
-public WorkerSlot allocateResource(double memOnHeap, double 
memOffHeap, double cpu) {
-this.memOnHeap += memOnHeap;
-this.memOffHeap += memOffHeap;
-this.cpu += cpu;
-return this;
+public String getId() {
+return this.getNodeId() + ":" + this.getPort();
--- End diff --

minor: Do not need `this.` when calling methods.


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread zhuoliu
Github user zhuoliu commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49785382
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java ---
@@ -662,22 +662,50 @@ public void setStatusMap(Map 
statusMap) {
 this.status.putAll(statusMap);
 }
 
-public void setResources(String topologyId, Double[] resources) {
-this.resources.put(topologyId, resources);
+/**
+ * Set the amount of resources used used by a topology. Used for 
displaying resource information on the UI
+ * @param topologyId
+ * @param resources describes the resources requested and assigned to 
topology in the following format in an array:
+ *  {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, 
assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
+ */
+public void setTopologyResources(String topologyId, Double[] 
resources) {
--- End diff --

Yep, I think that would be helpful. Both for topology and supervisor 
resources. We may file a separate JIRA for that or do it within this pull 
request.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...

2016-01-14 Thread ptgoetz
Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1010#issuecomment-171780642
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49785320
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/WorkerSlot.java ---
@@ -40,11 +47,8 @@ public int getPort() {
 return port;
 }
 
-public WorkerSlot allocateResource(double memOnHeap, double 
memOffHeap, double cpu) {
-this.memOnHeap += memOnHeap;
-this.memOffHeap += memOffHeap;
-this.cpu += cpu;
-return this;
+public String getId() {
+return this.getNodeId() + ":" + this.getPort();
--- End diff --

minor: Do not need `this.` when calling methods.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098863#comment-15098863
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user zhuoliu commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49785382
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java ---
@@ -662,22 +662,50 @@ public void setStatusMap(Map 
statusMap) {
 this.status.putAll(statusMap);
 }
 
-public void setResources(String topologyId, Double[] resources) {
-this.resources.put(topologyId, resources);
+/**
+ * Set the amount of resources used used by a topology. Used for 
displaying resource information on the UI
+ * @param topologyId
+ * @param resources describes the resources requested and assigned to 
topology in the following format in an array:
+ *  {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, 
assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
+ */
+public void setTopologyResources(String topologyId, Double[] 
resources) {
--- End diff --

Yep, I think that would be helpful. Both for topology and supervisor 
resources. We may file a separate JIRA for that or do it within this pull 
request.


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098869#comment-15098869
 ] 

ASF GitHub Bot commented on STORM-1470:
---

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1010#issuecomment-171780642
  
+1


> org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
> --
>
> Key: STORM-1470
> URL: https://issues.apache.org/jira/browse/STORM-1470
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> {noformat}
> 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for 
> /favicon.ico
> java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64
> at 
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We already shade commons-codec:commons-codec, but we don't apply that shading 
> to org.apache.hadoop:hadoop-auth.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098877#comment-15098877
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49786540
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -43,32 +43,84 @@
  */
 public class RAS_Node {
 private static final Logger LOG = 
LoggerFactory.getLogger(RAS_Node.class);
-private Map _topIdToUsedSlots = new 
HashMap();
-private Set _freeSlots = new HashSet();
+
+//A map consisting of all workers on the node.
+//The key of the map is the worker id and the value is the 
corresponding workerslot object
+Map _slots = new HashMap ();
+
+// A map describing which topologies are using which slots on this 
node.  The format of the map is the following:
+// {TopologyId -> {WorkerId -> {Executors}}}
+private Map> 
_topIdToUsedSlots = new HashMap>();
+
 private final String _nodeId;
 private String _hostname;
 private boolean _isAlive;
 private SupervisorDetails _sup;
-private Double _availMemory;
-private Double _availCPU;
-private Cluster _cluster;
-private Topologies _topologies;
+private Double _availMemory = 0.0;
+private Double _availCPU = 0.0;
+private final Cluster _cluster;
+private final Topologies _topologies;
 
-public RAS_Node(String nodeId, Set allPorts, boolean isAlive,
-SupervisorDetails sup, Cluster cluster, Topologies 
topologies) {
+public RAS_Node(String nodeId, SupervisorDetails sup, Cluster cluster, 
Topologies topologies, Map workerIdToWorker, Map> assignmentMap) {
+//Node ID and supervisor ID are the same.
 _nodeId = nodeId;
-_isAlive = isAlive;
-if (_isAlive && allPorts != null) {
-for (int port : allPorts) {
-_freeSlots.add(new WorkerSlot(_nodeId, port));
-}
-_sup = sup;
+if (sup == null) {
+_isAlive = false;
+} else {
+_isAlive = !cluster.isBlackListed(_nodeId);
+}
+
+_cluster = cluster;
+_topologies = topologies;
+
+// initialize slots for this node
+if (workerIdToWorker != null) {
+_slots = workerIdToWorker;
+}
+
+//initialize assignment map
+if (assignmentMap != null) {
+_topIdToUsedSlots = assignmentMap;
+}
+
+//check if node is alive
+if (_isAlive && sup != null) {
 _hostname = sup.getHost();
+_sup = sup;
 _availMemory = getTotalMemoryResources();
 _availCPU = getTotalCpuResources();
+
+LOG.debug("Found a {} Node {} {}",
+_isAlive ? "living" : "dead", _nodeId, 
sup.getAllPorts());
+LOG.debug("resources_mem: {}, resources_CPU: {}", 
sup.getTotalMemory(), sup.getTotalCPU());
+//intialize resource usages on node
+intializeResources();
+}
+}
+
+/**
+ * intializes resource usages on node
--- End diff --

`initializes`


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor 

[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49786540
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -43,32 +43,84 @@
  */
 public class RAS_Node {
 private static final Logger LOG = 
LoggerFactory.getLogger(RAS_Node.class);
-private Map _topIdToUsedSlots = new 
HashMap();
-private Set _freeSlots = new HashSet();
+
+//A map consisting of all workers on the node.
+//The key of the map is the worker id and the value is the 
corresponding workerslot object
+Map _slots = new HashMap ();
+
+// A map describing which topologies are using which slots on this 
node.  The format of the map is the following:
+// {TopologyId -> {WorkerId -> {Executors}}}
+private Map> 
_topIdToUsedSlots = new HashMap>();
+
 private final String _nodeId;
 private String _hostname;
 private boolean _isAlive;
 private SupervisorDetails _sup;
-private Double _availMemory;
-private Double _availCPU;
-private Cluster _cluster;
-private Topologies _topologies;
+private Double _availMemory = 0.0;
+private Double _availCPU = 0.0;
+private final Cluster _cluster;
+private final Topologies _topologies;
 
-public RAS_Node(String nodeId, Set allPorts, boolean isAlive,
-SupervisorDetails sup, Cluster cluster, Topologies 
topologies) {
+public RAS_Node(String nodeId, SupervisorDetails sup, Cluster cluster, 
Topologies topologies, Map workerIdToWorker, Map> assignmentMap) {
+//Node ID and supervisor ID are the same.
 _nodeId = nodeId;
-_isAlive = isAlive;
-if (_isAlive && allPorts != null) {
-for (int port : allPorts) {
-_freeSlots.add(new WorkerSlot(_nodeId, port));
-}
-_sup = sup;
+if (sup == null) {
+_isAlive = false;
+} else {
+_isAlive = !cluster.isBlackListed(_nodeId);
+}
+
+_cluster = cluster;
+_topologies = topologies;
+
+// initialize slots for this node
+if (workerIdToWorker != null) {
+_slots = workerIdToWorker;
+}
+
+//initialize assignment map
+if (assignmentMap != null) {
+_topIdToUsedSlots = assignmentMap;
+}
+
+//check if node is alive
+if (_isAlive && sup != null) {
 _hostname = sup.getHost();
+_sup = sup;
 _availMemory = getTotalMemoryResources();
 _availCPU = getTotalCpuResources();
+
+LOG.debug("Found a {} Node {} {}",
+_isAlive ? "living" : "dead", _nodeId, 
sup.getAllPorts());
+LOG.debug("resources_mem: {}, resources_CPU: {}", 
sup.getTotalMemory(), sup.getTotalCPU());
+//intialize resource usages on node
+intializeResources();
+}
+}
+
+/**
+ * intializes resource usages on node
--- End diff --

`initializes`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49786777
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -103,92 +182,37 @@ public boolean isAlive() {
 }
 
 public boolean isTotallyFree() {
-return _topIdToUsedSlots.isEmpty();
+return getUsedSlots().isEmpty();
--- End diff --

Is this method used?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098881#comment-15098881
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49786777
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -103,92 +182,37 @@ public boolean isAlive() {
 }
 
 public boolean isTotallyFree() {
-return _topIdToUsedSlots.isEmpty();
+return getUsedSlots().isEmpty();
--- End diff --

Is this method used?


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49786868
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -103,92 +182,37 @@ public boolean isAlive() {
 }
 
 public boolean isTotallyFree() {
-return _topIdToUsedSlots.isEmpty();
+return getUsedSlots().isEmpty();
 }
 
 public int totalSlotsFree() {
-return _freeSlots.size();
+return getFreeSlots().size();
 }
 
 public int totalSlotsUsed() {
-int total = 0;
-for (Set slots : _topIdToUsedSlots.values()) {
-total += slots.size();
-}
-return total;
+return getUsedSlots().size();
 }
 
 public int totalSlots() {
-return totalSlotsFree() + totalSlotsUsed();
+return _slots.size();
 }
 
 public int totalSlotsUsed(String topId) {
-int total = 0;
-Set slots = _topIdToUsedSlots.get(topId);
-if (slots != null) {
-total = slots.size();
-}
-return total;
-}
-
-private void validateSlot(WorkerSlot ws) {
-if (!_nodeId.equals(ws.getNodeId())) {
-throw new IllegalArgumentException(
-"Trying to add a slot to the wrong node " + ws +
-" is not a part of " + _nodeId);
-}
-}
-
-void addOrphanedSlot(WorkerSlot ws) {
-if (_isAlive) {
-throw new IllegalArgumentException("Orphaned Slots " +
-"only are allowed on dead nodes.");
-}
-validateSlot(ws);
-if (_freeSlots.contains(ws)) {
-return;
-}
-for (Set used : _topIdToUsedSlots.values()) {
-if (used.contains(ws)) {
-return;
-}
-}
-_freeSlots.add(ws);
-}
-
-boolean assignInternal(WorkerSlot ws, String topId, boolean dontThrow) 
{
-validateSlot(ws);
-if (!_freeSlots.remove(ws)) {
-if (dontThrow) {
-return true;
-}
-throw new IllegalStateException("Assigning a slot that was not 
free " + ws);
-}
-Set usedSlots = _topIdToUsedSlots.get(topId);
-if (usedSlots == null) {
-usedSlots = new HashSet();
-_topIdToUsedSlots.put(topId, usedSlots);
-}
-usedSlots.add(ws);
-return false;
+return getUsedSlots(topId).size();
 }
 
 /**
  * Free all slots on this node.  This will update the Cluster too.
  */
-public void freeAllSlots() {
+ public void freeAllSlots() {
--- End diff --

revert?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098886#comment-15098886
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49786868
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -103,92 +182,37 @@ public boolean isAlive() {
 }
 
 public boolean isTotallyFree() {
-return _topIdToUsedSlots.isEmpty();
+return getUsedSlots().isEmpty();
 }
 
 public int totalSlotsFree() {
-return _freeSlots.size();
+return getFreeSlots().size();
 }
 
 public int totalSlotsUsed() {
-int total = 0;
-for (Set slots : _topIdToUsedSlots.values()) {
-total += slots.size();
-}
-return total;
+return getUsedSlots().size();
 }
 
 public int totalSlots() {
-return totalSlotsFree() + totalSlotsUsed();
+return _slots.size();
 }
 
 public int totalSlotsUsed(String topId) {
-int total = 0;
-Set slots = _topIdToUsedSlots.get(topId);
-if (slots != null) {
-total = slots.size();
-}
-return total;
-}
-
-private void validateSlot(WorkerSlot ws) {
-if (!_nodeId.equals(ws.getNodeId())) {
-throw new IllegalArgumentException(
-"Trying to add a slot to the wrong node " + ws +
-" is not a part of " + _nodeId);
-}
-}
-
-void addOrphanedSlot(WorkerSlot ws) {
-if (_isAlive) {
-throw new IllegalArgumentException("Orphaned Slots " +
-"only are allowed on dead nodes.");
-}
-validateSlot(ws);
-if (_freeSlots.contains(ws)) {
-return;
-}
-for (Set used : _topIdToUsedSlots.values()) {
-if (used.contains(ws)) {
-return;
-}
-}
-_freeSlots.add(ws);
-}
-
-boolean assignInternal(WorkerSlot ws, String topId, boolean dontThrow) 
{
-validateSlot(ws);
-if (!_freeSlots.remove(ws)) {
-if (dontThrow) {
-return true;
-}
-throw new IllegalStateException("Assigning a slot that was not 
free " + ws);
-}
-Set usedSlots = _topIdToUsedSlots.get(topId);
-if (usedSlots == null) {
-usedSlots = new HashSet();
-_topIdToUsedSlots.put(topId, usedSlots);
-}
-usedSlots.add(ws);
-return false;
+return getUsedSlots(topId).size();
 }
 
 /**
  * Free all slots on this node.  This will update the Cluster too.
  */
-public void freeAllSlots() {
+ public void freeAllSlots() {
--- End diff --

revert?


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49787051
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -247,19 +253,23 @@ private void freeMemory(double amount) {
 
 private void freeCPU(double amount) {
 LOG.debug("freeing {} CPU on node...avail CPU: {}", amount, 
getHostname(), _availCPU);
-if ((_availCPU + amount) > getAvailableCpuResources()) {
-LOG.warn("Freeing more CPU than there exists!");
+if ((_availCPU + amount) > getTotalCpuResources()) {
+LOG.warn("Freeing more CPU than there exists! CPU trying to 
free: {} Total CPU on Node: {}", (_availMemory + amount), 
getTotalCpuResources());
 return;
 }
 _availCPU += amount;
 }
 
+/**
+ * get the amount of memory used by a worker
+ */
 public double getMemoryUsedByWorker(WorkerSlot ws) {
 TopologyDetails topo = findTopologyUsingWorker(ws);
 if (topo == null) {
 return 0.0;
 }
 Collection execs = getExecutors(ws, _cluster);
+LOG.info("getMemoryUsedByWorker executors: {}", execs);
--- End diff --

Is it useful for this to be INFO level? This could be a lot of text to log.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098890#comment-15098890
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49787051
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/RAS_Node.java ---
@@ -247,19 +253,23 @@ private void freeMemory(double amount) {
 
 private void freeCPU(double amount) {
 LOG.debug("freeing {} CPU on node...avail CPU: {}", amount, 
getHostname(), _availCPU);
-if ((_availCPU + amount) > getAvailableCpuResources()) {
-LOG.warn("Freeing more CPU than there exists!");
+if ((_availCPU + amount) > getTotalCpuResources()) {
+LOG.warn("Freeing more CPU than there exists! CPU trying to 
free: {} Total CPU on Node: {}", (_availMemory + amount), 
getTotalCpuResources());
 return;
 }
 _availCPU += amount;
 }
 
+/**
+ * get the amount of memory used by a worker
+ */
 public double getMemoryUsedByWorker(WorkerSlot ws) {
 TopologyDetails topo = findTopologyUsingWorker(ws);
 if (topo == null) {
 return 0.0;
 }
 Collection execs = getExecutors(ws, _cluster);
+LOG.info("getMemoryUsedByWorker executors: {}", execs);
--- End diff --

Is it useful for this to be INFO level? This could be a lot of text to log.


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098902#comment-15098902
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49787766
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/DefaultResourceAwareStrategy.java
 ---
@@ -121,7 +116,7 @@ public SchedulingResult schedule(TopologyDetails td) {
 td.getTaskResourceReqList(exec), entry.getKey() });
 WorkerSlot targetSlot = this.findWorkerForExec(exec, 
td, schedulerAssignmentMap);
 if (targetSlot != null) {
-RAS_Node targetNode = 
this.idToNode(targetSlot.getNodeId());
+NodeDetails targetNode = 
this.idToNode(targetSlot.getNodeId());
--- End diff --

It seems there is a chunk of duplicate code here and in the for loop below. 
 We are adding one line to both, but maybe it would be better to pull them out 
into a helper method to avoid accidental mistakes in the future?


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49788091
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/IStrategy.java
 ---
@@ -36,7 +31,7 @@
 /**
  * initialize prior to scheduling
  */
-public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes);
+public void prepare(ClusterStateData clusterStateData);
--- End diff --

minor: public not needed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49787766
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/DefaultResourceAwareStrategy.java
 ---
@@ -121,7 +116,7 @@ public SchedulingResult schedule(TopologyDetails td) {
 td.getTaskResourceReqList(exec), entry.getKey() });
 WorkerSlot targetSlot = this.findWorkerForExec(exec, 
td, schedulerAssignmentMap);
 if (targetSlot != null) {
-RAS_Node targetNode = 
this.idToNode(targetSlot.getNodeId());
+NodeDetails targetNode = 
this.idToNode(targetSlot.getNodeId());
--- End diff --

It seems there is a chunk of duplicate code here and in the for loop below. 
 We are adding one line to both, but maybe it would be better to pull them out 
into a helper method to avoid accidental mistakes in the future?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098904#comment-15098904
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49788091
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/resource/strategies/scheduling/IStrategy.java
 ---
@@ -36,7 +31,7 @@
 /**
  * initialize prior to scheduling
  */
-public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes);
+public void prepare(ClusterStateData clusterStateData);
--- End diff --

minor: public not needed


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49783662
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj ---
@@ -701,11 +701,7 @@
;; making a map from node+port to WorkerSlot with 
allocated resources
node+port->slot (into {} (for [[[node port] 
[mem-on-heap mem-off-heap cpu]] worker->resources]
   {[node port]
-   (doto (WorkerSlot. node 
port)
- (.allocateResource
-   mem-on-heap
-   mem-off-heap
-   cpu))}))
+   (doto (WorkerSlot. node 
port mem-on-heap mem-off-heap cpu))}))
--- End diff --

Can we remove the `doto` here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....

2016-01-14 Thread Parth-Brahmbhatt
Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/936#issuecomment-171777228
  
@roshannaik Thanks for the patch. I have merged this request to master. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098839#comment-15098839
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49783662
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj ---
@@ -701,11 +701,7 @@
;; making a map from node+port to WorkerSlot with 
allocated resources
node+port->slot (into {} (for [[[node port] 
[mem-on-heap mem-off-heap cpu]] worker->resources]
   {[node port]
-   (doto (WorkerSlot. node 
port)
- (.allocateResource
-   mem-on-heap
-   mem-off-heap
-   cpu))}))
+   (doto (WorkerSlot. node 
port mem-on-heap mem-off-heap cpu))}))
--- End diff --

Can we remove the `doto` here?


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread jerrypeng
Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49783732
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj ---
@@ -701,11 +701,7 @@
;; making a map from node+port to WorkerSlot with 
allocated resources
node+port->slot (into {} (for [[[node port] 
[mem-on-heap mem-off-heap cpu]] worker->resources]
   {[node port]
-   (doto (WorkerSlot. node 
port)
- (.allocateResource
-   mem-on-heap
-   mem-off-heap
-   cpu))}))
+   (doto (WorkerSlot. node 
port mem-on-heap mem-off-heap cpu))}))
--- End diff --

will remove


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1199) Create HDFS Spout

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098840#comment-15098840
 ] 

ASF GitHub Bot commented on STORM-1199:
---

Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/936#issuecomment-171777228
  
@roshannaik Thanks for the patch. I have merged this request to master. 


> Create HDFS Spout
> -
>
> Key: STORM-1199
> URL: https://issues.apache.org/jira/browse/STORM-1199
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, 
> hdfs-spout.1.patch
>
>
> Create an HDFS spout so that Storm can suck in data from files in a HDFS 
> directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1199) Create HDFS Spout

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098838#comment-15098838
 ] 

ASF GitHub Bot commented on STORM-1199:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/936


> Create HDFS Spout
> -
>
> Key: STORM-1199
> URL: https://issues.apache.org/jira/browse/STORM-1199
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, 
> hdfs-spout.1.patch
>
>
> Create an HDFS spout so that Storm can suck in data from files in a HDFS 
> directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (STORM-1199) Create HDFS Spout

2016-01-14 Thread Parth Brahmbhatt (JIRA)

 [ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Parth Brahmbhatt resolved STORM-1199.
-
   Resolution: Fixed
Fix Version/s: 1.0.0

> Create HDFS Spout
> -
>
> Key: STORM-1199
> URL: https://issues.apache.org/jira/browse/STORM-1199
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Fix For: 1.0.0
>
> Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, 
> hdfs-spout.1.patch
>
>
> Create an HDFS spout so that Storm can suck in data from files in a HDFS 
> directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098841#comment-15098841
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49783732
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj ---
@@ -701,11 +701,7 @@
;; making a map from node+port to WorkerSlot with 
allocated resources
node+port->slot (into {} (for [[[node port] 
[mem-on-heap mem-off-heap cpu]] worker->resources]
   {[node port]
-   (doto (WorkerSlot. node 
port)
- (.allocateResource
-   mem-on-heap
-   mem-off-heap
-   cpu))}))
+   (doto (WorkerSlot. node 
port mem-on-heap mem-off-heap cpu))}))
--- End diff --

will remove


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49783928
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java ---
@@ -662,22 +662,50 @@ public void setStatusMap(Map 
statusMap) {
 this.status.putAll(statusMap);
 }
 
-public void setResources(String topologyId, Double[] resources) {
-this.resources.put(topologyId, resources);
+/**
+ * Set the amount of resources used used by a topology. Used for 
displaying resource information on the UI
+ * @param topologyId
+ * @param resources describes the resources requested and assigned to 
topology in the following format in an array:
+ *  {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, 
assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
+ */
+public void setTopologyResources(String topologyId, Double[] 
resources) {
--- End diff --

Can we replace `Double[]` with a class that is appropriate to the data?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098846#comment-15098846
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49783928
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java ---
@@ -662,22 +662,50 @@ public void setStatusMap(Map 
statusMap) {
 this.status.putAll(statusMap);
 }
 
-public void setResources(String topologyId, Double[] resources) {
-this.resources.put(topologyId, resources);
+/**
+ * Set the amount of resources used used by a topology. Used for 
displaying resource information on the UI
+ * @param topologyId
+ * @param resources describes the resources requested and assigned to 
topology in the following format in an array:
+ *  {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, 
assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
+ */
+public void setTopologyResources(String topologyId, Double[] 
resources) {
--- End diff --

Can we replace `Double[]` with a class that is appropriate to the data?


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098855#comment-15098855
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49784882
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/SchedulerAssignment.java ---
@@ -55,4 +56,6 @@
 public Set getExecutors();
 
 public Set getSlots();
+
+public Map 
getSlotToExecutors();
--- End diff --

very minor: The interface is public, so it's redundant to mark each method 
declaration as public.


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49784882
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/SchedulerAssignment.java ---
@@ -55,4 +56,6 @@
 public Set getExecutors();
 
 public Set getSlots();
+
+public Map 
getSlotToExecutors();
--- End diff --

very minor: The interface is public, so it's redundant to mark each method 
declaration as public.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Updated] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread Boyang Jerry Peng (JIRA)

 [ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Jerry Peng updated STORM-1450:
-
Description: 
Code refactored:

1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  Each 
RAS_Node will now be initialized with a map of all its assignments.  Each 
RAS_Node will also figure out resources used and available. Removed unnecessary 
functions.

2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
modify it

3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
that the semantics of what a scheduling strategy should do will be more clear.  
Each scheduling strategy shouldn't be actually assigning anything.  The 
strategy should only calculate a scheduling.

Bug fixes:

1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
The function updateSupervisorResources was placed in the wrong place

2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
was done.





  was:
Code refactored:

1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  Each 
RAS_Node will now be initialized with a map of all its assignments.  Each 
RAS_Node will also figure out resources used and available. 

2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
modify it

3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
that the semantics of what a scheduling strategy should do will be more clear.  
Each scheduling strategy shouldn't be actually assigning anything.  The 
strategy should only calculate a scheduling.

Bug fixes:

1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
The function updateSupervisorResources was placed in the wrong place

2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
was done.






> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1199) Create HDFS Spout

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098799#comment-15098799
 ] 

ASF GitHub Bot commented on STORM-1199:
---

Github user roshannaik commented on the pull request:

https://github.com/apache/storm/pull/936#issuecomment-171771850
  
As per request from @harshach, I have opened up STORM-1474 for addressing 
the pending minor review comments. Looks like @harshach is away, 
@Parth-Brahmbhatt  could you take a look into committing it ?


> Create HDFS Spout
> -
>
> Key: STORM-1199
> URL: https://issues.apache.org/jira/browse/STORM-1199
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, 
> hdfs-spout.1.patch
>
>
> Create an HDFS spout so that Storm can suck in data from files in a HDFS 
> directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....

2016-01-14 Thread roshannaik
Github user roshannaik commented on the pull request:

https://github.com/apache/storm/pull/936#issuecomment-171771850
  
As per request from @harshach, I have opened up STORM-1474 for addressing 
the pending minor review comments. Looks like @harshach is away, 
@Parth-Brahmbhatt  could you take a look into committing it ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49781622
  
--- Diff: 
storm-core/test/jvm/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -939,6 +945,76 @@ public void TestEvictTopologyFromItself() {
  * If users are above his or her guarantee, check if topology eviction 
works correct
  */
 @Test
+public void Test() {
--- End diff --

Rename?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098807#comment-15098807
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49781622
  
--- Diff: 
storm-core/test/jvm/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -939,6 +945,76 @@ public void TestEvictTopologyFromItself() {
  * If users are above his or her guarantee, check if topology eviction 
works correct
  */
 @Test
+public void Test() {
--- End diff --

Rename?


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1450] - Fix bugs and refactor code in R...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49781708
  
--- Diff: 
storm-core/test/jvm/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -1083,6 +1159,15 @@ public void TestOverGuaranteeEviction() {
 Assert.assertEquals("# of pending topologies", 0, 
rs.getUser("bobby").getTopologiesPending().size());
 Assert.assertEquals("# of attempted topologies", 0, 
rs.getUser("bobby").getTopologiesAttempted().size());
 Assert.assertEquals("# of invalid topologies", 0, 
rs.getUser("bobby").getTopologiesInvalid().size());
+
+LOG.info("Assignments: {}", cluster.getAssignments());
+for (Map.Entry entry : 
cluster.getAssignments().entrySet()) {
+LOG.info("Topology id: {}", entry.getKey());
+for(WorkerSlot target: entry.getValue().getSlots()) {
+LOG.info("target resources onheap: {} offheap: {} cpu: 
{}", target.getAllocatedMemOnHeap(), target.getAllocatedMemOffHeap(), 
target.getAllocatedCpu());
+}
+
+}
--- End diff --

Revert?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098808#comment-15098808
 ] 

ASF GitHub Bot commented on STORM-1450:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1016#discussion_r49781708
  
--- Diff: 
storm-core/test/jvm/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -1083,6 +1159,15 @@ public void TestOverGuaranteeEviction() {
 Assert.assertEquals("# of pending topologies", 0, 
rs.getUser("bobby").getTopologiesPending().size());
 Assert.assertEquals("# of attempted topologies", 0, 
rs.getUser("bobby").getTopologiesAttempted().size());
 Assert.assertEquals("# of invalid topologies", 0, 
rs.getUser("bobby").getTopologiesInvalid().size());
+
+LOG.info("Assignments: {}", cluster.getAssignments());
+for (Map.Entry entry : 
cluster.getAssignments().entrySet()) {
+LOG.info("Topology id: {}", entry.getKey());
+for(WorkerSlot target: entry.getValue().getSlots()) {
+LOG.info("target resources onheap: {} offheap: {} cpu: 
{}", target.getAllocatedMemOnHeap(), target.getAllocatedMemOffHeap(), 
target.getAllocatedCpu());
+}
+
+}
--- End diff --

Revert?


> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Code refactored:
> 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes.  
> Each RAS_Node will now be initialized with a map of all its assignments.  
> Each RAS_Node will also figure out resources used and available. Removed 
> unnecessary functions.
> 2.  Made WorkerSlot immutable so that a scheduling strategy won't mistakenly 
> modify it
> 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so 
> that the semantics of what a scheduling strategy should do will be more 
> clear.  Each scheduling strategy shouldn't be actually assigning anything.  
> The strategy should only calculate a scheduling.
> Bug fixes:
> 1. Minor bug in displaying the assigned resources for a supervisor on the UI. 
> The function updateSupervisorResources was placed in the wrong place
> 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that 
> was done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1199) Create HDFS Spout

2016-01-14 Thread Roshan Naik (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098948#comment-15098948
 ] 

Roshan Naik commented on STORM-1199:


Thanks all for your input/feedback/reviews ..  they were very useful.

> Create HDFS Spout
> -
>
> Key: STORM-1199
> URL: https://issues.apache.org/jira/browse/STORM-1199
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Fix For: 1.0.0
>
> Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, 
> hdfs-spout.1.patch
>
>
> Create an HDFS spout so that Storm can suck in data from files in a HDFS 
> directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098958#comment-15098958
 ] 

ASF GitHub Bot commented on STORM-1452:
---

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1012#issuecomment-171793561
  
Not really specific to this pull request, but I missed it before when the 
profiling functionality was originally added.

If profiling is enabled, we turn on the "Commercial Features" of the JVM:

`worker.profiler.childopts: "-XX:+UnlockCommercialFeatures 
-XX:+FlightRecorder`

The current Oracle Java license [1] states:

>Some of the packages described in the Installation of Oracle Java SE 
Product Editions section above install Commercial Features that are restricted 
to Oracle Java SE Advanced, Oracle Java SE Advanced Desktop and/or Oracle Java 
SE Suite. **This means that even if you download an Oracle Java SE package for 
free under the Java BCLA, you must separately license from Oracle (and pay the 
appropriate license fee) the right to use any Commercial Features, described in 
Table 1-1 below, included in these packages.**

If my interpretation of the license is correct, that means users must have 
purchased a license from Orace for each node that has profiling enabled. (Read 
the license [1] for more information and examples.)

I think that's probably okay, but we may want to document that somewhere 
(`defaults.yaml` maybe?), so users don't accidentally run afoul of Oracle's 
licensing terms.

[1] http://www.oracle.com/technetwork/java/javase/terms/products/index.html


> Worker "profiler" actions broken by default
> ---
>
> Key: STORM-1452
> URL: https://issues.apache.org/jira/browse/STORM-1452
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> * The profiler script flight.bash is not packaged by default.
> * The default options enable the Oracle specific flight-recorder that 
> requires a support subscription.
> The option to enable the profiler should not be enabled by default.  Other 
> actions such as worker restart, debugging, and heap can remain enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...

2016-01-14 Thread ptgoetz
Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1012#issuecomment-171793561
  
Not really specific to this pull request, but I missed it before when the 
profiling functionality was originally added.

If profiling is enabled, we turn on the "Commercial Features" of the JVM:

`worker.profiler.childopts: "-XX:+UnlockCommercialFeatures 
-XX:+FlightRecorder`

The current Oracle Java license [1] states:

>Some of the packages described in the Installation of Oracle Java SE 
Product Editions section above install Commercial Features that are restricted 
to Oracle Java SE Advanced, Oracle Java SE Advanced Desktop and/or Oracle Java 
SE Suite. **This means that even if you download an Oracle Java SE package for 
free under the Java BCLA, you must separately license from Oracle (and pay the 
appropriate license fee) the right to use any Commercial Features, described in 
Table 1-1 below, included in these packages.**

If my interpretation of the license is correct, that means users must have 
purchased a license from Orace for each node that has profiling enabled. (Read 
the license [1] for more information and examples.)

I think that's probably okay, but we may want to document that somewhere 
(`defaults.yaml` maybe?), so users don't accidentally run afoul of Oracle's 
licensing terms.

[1] http://www.oracle.com/technetwork/java/javase/terms/products/index.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] storm pull request: STORM-1466: Move the org.apache.thrift7 namesp...

2016-01-14 Thread d2r
Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1007#discussion_r49791863
  
--- Diff: storm-core/pom.xml ---
@@ -618,7 +618,7 @@
 
 org.apache.thrift
 
-
org.apache.thrift7
+
org.apache.storm.thrift
--- End diff --

We should remove the old comment about backward compatibility.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1466) Move the org.apache.thrift7 namespace to something correct/sensible

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098970#comment-15098970
 ] 

ASF GitHub Bot commented on STORM-1466:
---

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/1007#discussion_r49791863
  
--- Diff: storm-core/pom.xml ---
@@ -618,7 +618,7 @@
 
 org.apache.thrift
 
-
org.apache.thrift7
+
org.apache.storm.thrift
--- End diff --

We should remove the old comment about backward compatibility.


> Move the org.apache.thrift7 namespace to something correct/sensible
> ---
>
> Key: STORM-1466
> URL: https://issues.apache.org/jira/browse/STORM-1466
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
>
> Thrift is still shaded to org.apache.thrift7 even though we are not using 
> Thrift 7.
> We should also add something for temporary backwards compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (STORM-1475) storm-elasticsearch should support ES 2.X

2016-01-14 Thread Aaron Dossett (JIRA)
Aaron Dossett created STORM-1475:


 Summary: storm-elasticsearch should support ES 2.X
 Key: STORM-1475
 URL: https://issues.apache.org/jira/browse/STORM-1475
 Project: Apache Storm
  Issue Type: Improvement
  Components: storm-elasticsearch
Affects Versions: 1.0.0
Reporter: Aaron Dossett






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (STORM-1475) storm-elasticsearch should support ES 2.X

2016-01-14 Thread Aaron Dossett (JIRA)

 [ 
https://issues.apache.org/jira/browse/STORM-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron Dossett reassigned STORM-1475:


Assignee: Aaron Dossett

> storm-elasticsearch should support ES 2.X
> -
>
> Key: STORM-1475
> URL: https://issues.apache.org/jira/browse/STORM-1475
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-elasticsearch
>Affects Versions: 1.0.0
>Reporter: Aaron Dossett
>Assignee: Aaron Dossett
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...

2016-01-14 Thread ptgoetz
Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1011#issuecomment-171796975
  
Personally, I don't think we need separate pull requests for each branch 
yet (master and 1.x have not diverged enough to cause conflicts), as it kind of 
confuses things in terms of +1s and comments. It might be easier to add a note 
requesting that it also be applied to another branch, e.g. "I would also like 
to see this applied to the X branch."

If for some reason the patch couldn't be applied to the other branch(es), 
the committer doing the merge could either resolve the conflicts or request an 
additional pull request against the conflicted branch.

Thoughts?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication

2016-01-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098979#comment-15098979
 ] 

ASF GitHub Bot commented on STORM-1470:
---

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1011#issuecomment-171796975
  
Personally, I don't think we need separate pull requests for each branch 
yet (master and 1.x have not diverged enough to cause conflicts), as it kind of 
confuses things in terms of +1s and comments. It might be easier to add a note 
requesting that it also be applied to another branch, e.g. "I would also like 
to see this applied to the X branch."

If for some reason the patch couldn't be applied to the other branch(es), 
the committer doing the merge could either resolve the conflicts or request an 
additional pull request against the conflicted branch.

Thoughts?


> org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
> --
>
> Key: STORM-1470
> URL: https://issues.apache.org/jira/browse/STORM-1470
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> {noformat}
> 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for 
> /favicon.ico
> java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64
> at 
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We already shade commons-codec:commons-codec, but we don't apply that shading 
> to org.apache.hadoop:hadoop-auth.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)