date:20170221

[GitHub] storm issue #1954: STORM-2372: Pacemaker client doesn't clean up heartbeats ...

2017-02-21 Thread knusbaum

Github user knusbaum commented on the issue:

https://github.com/apache/storm/pull/1954
  
Need to test this one 1.* branches are tested.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

2017-02-21 Thread HeartSaVioR

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1939
  
@pasalkarsachin1 
Merged into master via 313e960bedde3b2c47520a74c391425d79271727, and 1.x 
branch via 4fc55b33504d446ef192a25e5164f861e1495291.
Could you close this PR? Thanks in advance.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1957: STORM-2373: HDFS Spout should support multiple ign...

2017-02-21 Thread pasalkarsachin1

GitHub user pasalkarsachin1 opened a pull request:

https://github.com/apache/storm/pull/1957

STORM-2373: HDFS Spout should support multiple ignore extensions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pasalkarsachin1/storm STORM-2373

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1957.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1957


commit c15c647b9cc001b03f030762a13b4e64e2e5cbf3
Author: Sachin Pasalkar 
Date:   2017-02-22T06:24:18Z

STORM-2373: HDFS Spout should support multiple ignore extensions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

2017-02-21 Thread pasalkarsachin1

Github user pasalkarsachin1 commented on the issue:

https://github.com/apache/storm/pull/1939
  
@HeartSaVioR  Can you merge this so I will close this PR :) Thanks in 
advance


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1942: STORM-2358: Update storm hdfs spout to remove specific im...

2017-02-21 Thread pasalkarsachin1

Github user pasalkarsachin1 commented on the issue:

https://github.com/apache/storm/pull/1942
  
@HeartSaVioR  @roshannaik @arunmahadevan  are you OK with this approach?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1934: STORM-2333: CGroup memory and CPU metrics

2017-02-21 Thread jerrypeng

Github user jerrypeng commented on the issue:

https://github.com/apache/storm/pull/1934
  
thanks @revans2 for taking the time to refactor some of the code!  I 
reviewed it once more.  If there is an example of how to user the metrics in 
the docs I think that would be extremely helpful to users.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1934: STORM-2333: CGroup memory and CPU metrics

2017-02-21 Thread jerrypeng

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1934#discussion_r102382536
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/cgroup/CGroupMemoryUsage.java ---
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.metric.cgroup;
+
+import java.util.Map;
+
+import org.apache.storm.container.cgroup.SubSystemType;
+import org.apache.storm.container.cgroup.core.CgroupCore;
+import org.apache.storm.container.cgroup.core.MemoryCore;
+
+/**
+ * Reports the current memory usage of the cgroup for this worker
+ */
+public class CGroupMemoryUsage extends CGroupMetricsBase {
+
+public CGroupMemoryUsage(Map conf) {
+super(conf, SubSystemType.memory);
+}
+
+@Override
+public Long getDataFrom(CgroupCore core) throws Exception {
--- End diff --

same question here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1934: STORM-2333: CGroup memory and CPU metrics

2017-02-21 Thread jerrypeng

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1934#discussion_r102382461
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/metric/cgroup/CGroupMemoryLimit.java ---
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.metric.cgroup;
+
+import java.util.Map;
+
+import org.apache.storm.container.cgroup.SubSystemType;
+import org.apache.storm.container.cgroup.core.CgroupCore;
+import org.apache.storm.container.cgroup.core.MemoryCore;
+
+/**
+ * Reports the current memory limit of the cgroup for this worker
+ */
+public class CGroupMemoryLimit extends CGroupMetricsBase {
+
+public CGroupMemoryLimit(Map conf) {
+super(conf, SubSystemType.memory);
+}
+
+@Override
+public Long getDataFrom(CgroupCore core) throws Exception {
--- End diff --

Same question here but also this function throws an Exception except for 
IOException


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1934: STORM-2333: CGroup memory and CPU metrics

2017-02-21 Thread jerrypeng

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1934#discussion_r102382308
  
--- Diff: storm-core/src/jvm/org/apache/storm/metric/cgroup/CGroupCpu.java 
---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.metric.cgroup;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.storm.container.cgroup.SubSystemType;
+import org.apache.storm.container.cgroup.core.CgroupCore;
+import org.apache.storm.container.cgroup.core.CpuacctCore;
+import org.apache.storm.container.cgroup.core.CpuacctCore.StatType;
+
+/**
+ * Report CPU used in the cgroup
+ */
+public class CGroupCpu extends CGroupMetricsBase> {
+long previousSystem = 0;
+long previousUser = 0;
+private int userHz = -1;
+
+public CGroupCpu(Map conf) {
+super(conf, SubSystemType.cpuacct);
+}
+
+public synchronized int getUserHZ() throws IOException {
+if (userHz < 0) {
+ProcessBuilder pb = new ProcessBuilder("getconf", "CLK_TCK");
+Process p = pb.start();
+BufferedReader in = new BufferedReader(new 
InputStreamReader(p.getInputStream()));
+String line = in.readLine().trim();
+userHz = Integer.valueOf(line);
+}
+return userHz;
+}
+
+@Override
+public Map getDataFrom(CgroupCore core) throws 
IOException {
+CpuacctCore cpu = (CpuacctCore) core;
+Map stat = cpu.getCpuStat();
--- End diff --

is there a reason why we not doing a try catch of an IOException here 
similar for cpu.getCpuShares() below but just propagating the exception?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1934: STORM-2333: CGroup memory and CPU metrics

2017-02-21 Thread jerrypeng

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/1934#discussion_r102381366
  
--- Diff: docs/cgroups_in_storm.md ---
@@ -43,13 +43,24 @@ group storm {
}
cpu {
}
+   memory {
+   }
+   cpuacct {
+   }
 }
 ```
 
 For a more detailed explanation of the format and configs for the 
cgconfig.conf file, please visit:
 
 
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch-Using_Control_Groups.html#The_cgconfig.conf_File
 
+To let storm manage the cgroups for indavidual workers you need to make 
sure that the resources you want to control are mounted under the same 
directory as in the example above.
+If they are not in the same directory the supervisor will throw an 
exception.
+
+The perm section needs to be configured so that the user the supervisor is 
running as can modify the group.
+
+If run as user is enabled so the supervisor spawns other processes as the 
user that launched the topology make sure that the permissions are such that 
indavidual users have read access but not write access.
--- End diff --

perhaps more clean 

if "run as user"is enabled so **that** the supervisor spawns other...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1956: Add Storm SQL docs to index page for 1.x branch

2017-02-21 Thread HeartSaVioR

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1956
  
It also needs to be addressed for 1.1.0, but since release vote is ongoing, 
I'll handle it manually if vote passes and site doc for 1.1.0 is up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1956: Add Storm SQL docs to index page for 1.x branch

2017-02-21 Thread HeartSaVioR

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1956

Add Storm SQL docs to index page for 1.x branch

Rearrange a bit and add documentations about some missing Storm SQL.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm 
documentation-add-storm-sql-to-index-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1956.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1956


commit 0cfffd2e833b94269bcae56b4e13c4980da6e5f1
Author: Jungtaek Lim 
Date:   2017-02-22T04:03:45Z

Add Storm SQL docs to index page for 1.x branch




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1955: Add Streams API and Storm SQL docs to index page

2017-02-21 Thread HeartSaVioR

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1955

Add Streams API and Storm SQL docs to index page

Rearrange a bit and add documentations about Streams API and some missing 
Storm SQL.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm 
documentation-add-streams-api-and-storm-sql-to-index

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1955.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1955


commit 9f0a157ed39605b7e374d702aee6fef19942602e
Author: Jungtaek Lim 
Date:   2017-02-22T03:46:20Z

Add Streams API and Storm SQL docs to index page




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

2017-02-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1945


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1945: [STORM-2367] Documentation for streams API

2017-02-21 Thread satishd

Github user satishd commented on the issue:

https://github.com/apache/storm/pull/1945
  
+1, Nice summary of streams API. Thanks @arunmahadevan 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1949: STORM-1290: port backtype.storm.local-state-test to java

2017-02-21 Thread HeartSaVioR

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1949
  
Could you create a new branch based on current master and work from there, 
and submit PR?
You need to pull master before starting.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1954: STORM-2372: Pacemaker client doesn't clean up hear...

2017-02-21 Thread knusbaum

GitHub user knusbaum opened a pull request:

https://github.com/apache/storm/pull/1954

STORM-2372: Pacemaker client doesn't clean up heartbeats properly. (2.x)

See #1952 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/knusbaum/incubator-storm master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1954.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1954






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1953: STORM-2372: Pacemaker client doesn't clean up hear...

2017-02-21 Thread knusbaum

GitHub user knusbaum opened a pull request:

https://github.com/apache/storm/pull/1953

STORM-2372: Pacemaker client doesn't clean up heartbeats properly. (1.0.x)

See #1952 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/knusbaum/incubator-storm 1.0.x-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1953.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1953


commit a6b9668c98b46a7acfb9d4f39c6e90342b2f41f7
Author: Kyle Nusbaum 
Date:   2017-02-21T20:18:31Z

Fixing pacemaker delete-path bug.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1952: STORM-2372: Pacemaker client doesn't clean up hear...

2017-02-21 Thread knusbaum

GitHub user knusbaum opened a pull request:

https://github.com/apache/storm/pull/1952

STORM-2372: Pacemaker client doesn't clean up heartbeats properly.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/knusbaum/incubator-storm 1.x-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1952.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1952


commit ebed1c8b01397b09f4083e66f574a25f9b7c585d
Author: Kyle Nusbaum 
Date:   2017-02-21T20:18:31Z

Fixing pacemaker delete-path bug.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1951: STORM-2371 Implementing new eventhub driver

2017-02-21 Thread rban1

GitHub user rban1 opened a pull request:

https://github.com/apache/storm/pull/1951

STORM-2371 Implementing new eventhub driver

Changing the undelying receiver implementation with the latest Eventhub 
receiver

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rban1/storm master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1951.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1951


commit 098272d753f406754ad17f4ba3ecd6a08881d82c
Author: Ranjan Banerjee 
Date:   2017-02-21T21:51:16Z

Implementing new eventhub driver




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1934: STORM-2333: CGroup memory and CPU metrics

2017-02-21 Thread revans2

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/1934
  
@jerrypeng Please take another look.  I refactored the code to reuse the 
CgroupManager and such.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

2017-02-21 Thread arunmahadevan

Github user arunmahadevan commented on a diff in the pull request:

https://github.com/apache/storm/pull/1945#discussion_r102246874
  
--- Diff: docs/Stream-API.md ---
@@ -0,0 +1,473 @@
+---
+title: Stream API Overview
+layout: documentation
+documentation: true
+---
+
+* [Concepts](#concepts)
+* [Stream Builder] (#streambuilder)
+* [Value mapper] (#valuemapper)
+* [Stream APIs](#streamapis)
+* [Basic transformations] (#basictransformations)
--- End diff --

updated


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Release Apache Storm 1.1.0 (RC1)

2017-02-21 Thread Bobby Evans

+1 ran some basic tests on a distributed cluster.  Built from source and all 
unit tests passed.


- Bobby

On Sunday, February 19, 2017, 8:38:59 PM CST, Xin Wang  
wrote:+1 Ran some tests in cluster mode. And I saw the js RAT issue has been
fixed by Jungtaek.

- Xin

2017-02-20 8:30 GMT+08:00 Jungtaek Lim :

> We've been introducing new changes via release note news. I'm not sure
> Taylor already wrote one, but if he didn't, I'm happy to simply describe
> it.
>
> Diff-ing docs directory between 1.0.x branch (or 1.0.3) and 1.x branch
> (1.1.0 rc1) might help to determine the improvement or new features.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 2월 19일 (일) 오전 2:15, Julien Nioche 님이
> 작성:
>
> > Hi guys,
> >
> > I can't find a description of how 1.1.x differs from 1.0.x, is that
> > available somewhere?
> >
> > BTW all the versions on JIRA are marked as 'unreleased'
> >
> > Thanks
> >
> > Julien
> >
> > On 18 February 2017 at 12:16, Jungtaek Lim  wrote:
> >
> > > +1 (binding)
> > >
> > > > source
> > >
> > > - verify file (signature, MD5, SHA)
> > > -- source, tar.gz : OK
> > > -- source, zip : OK
> > >
> > > - extract file
> > > -- source, tar.gz : OK
> > > -- source, zip : OK
> > >
> > > - diff-ing extracted files between tar.gz and zip : OK
> > >
> > > - build source with JDK 7
> > > -- source, tar.gz : seeing RAT issue, others are OK
> > >
> > > - build source dist
> > > -- source, tar.gz : OK
> > >
> > > - build binary dist
> > > -- source, tar.gz : OK
> > >
> > > > binary
> > >
> > > - verify file (signature, MD5, SHA)
> > > -- binary, tar.gz : OK
> > > -- binary, zip : OK
> > >
> > > - extract file
> > > -- binary, tar.gz :  OK
> > > -- binary, zip : OK
> > >
> > > - diff-ing extracted files between tar.gz and zip : OK
> > >
> > > - launch daemons : OK
> > > - run RollingTopWords (local) : OK
> > > - run RollingTopWords (remote) : OK
> > >  - activate / deactivate / rebalance / kill : OK
> > >  - logviewer (worker dir, daemon dir) : OK
> > >  - change log level : OK
> > >  - thread dump, heap dump, restart worker : OK
> > >  - log search : OK
> > > - run WordCountTopology (multilang, remote) : OK
> > > - run StatefulTopology (local) : OK
> > > - run StatefulTopology (remote) : OK
> > > - run StatefulWindowingTopology (remote) : OK
> > >
> > > - run Storm SQL features (with --artifacts) : OK (via testing Kafka &
> > > csv/tsv schema)
> > >
> > > I've seen RAT issue from newly added js files, but it's not about
> LICENSE
> > > issue, just exclusion from RAT plugin is missing. (I'll take care of it
> > > immediately.)
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 2월 18일 (토) 오전 4:42, P. Taylor Goetz 님이 작성:
> > >
> > > > This is a call to vote on releasing Apache Storm 1.1.0 (rc1)
> > > >
> > > > Full list of changes in this release:
> > > >
> > > >
> > > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > > plain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
> > > >
> > > > The tag/commit to be voted upon is v1.1.0:
> > > >
> > > >
> > > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> > >
> > 75b7d808d10674ff580beb031ae970fe94e97615;hb=
> 17a2017fb644e353fb2a0f5bf50d40
> > > 0ee28036ba
> > > >
> > > > The source archive being voted upon can be found here:
> > > >
> > > >
> > > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > > 1.0-rc1/apache-storm-1.1.0-src.tar.gz
> > > >
> > > > Other release files, signatures and digests can be found here:
> > > >
> > > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc1/
> > > >
> > > > The release artifacts are signed with the following key:
> > > >
> > > >
> > > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > > >
> > > > The Nexus staging repository for this release is:
> > > >
> > > > https://repository.apache.org/content/repositories/
> orgapachestorm-1045
> > > >
> > > > Please vote on releasing this package as Apache Storm 1.1.0.
> > > >
> > > > When voting, please list the actions taken to verify the release.
> > > >
> > > > This vote will be open for at least 72 hours.
> > > >
> > > > [ ] +1 Release this package as Apache Storm 1.1.0
> > > > [ ]  0 No opinion
> > > > [ ] -1 Do not release this package because...
> > > >
> > > > Thanks to everyone who contributed to this release.
> > > >
> > > > -Taylor
> > >
> >
> >
> >
> > --
> >
> > *Open Source Solutions for Text Engineering*
> >
> > http://www.digitalpebble.com
> > http://digitalpebble.blogspot.com/
> > #digitalpebble 
> >
>

[GitHub] storm issue #1950: STORM-2369 [storm-redis] Use binary type for State manage...

2017-02-21 Thread HeartSaVioR

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1950
  
Might need to provide migration tool to migrate Base64 encoded states to be 
conformed to current implementation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1950: STORM-2369 [storm-redis] Use binary type for State manage...

2017-02-21 Thread HeartSaVioR

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1950
  
Tested storm-starter's StatefulTopology with Redis Cluster nodes manually.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

2017-02-21 Thread satishd

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/1945#discussion_r102217060
  
--- Diff: docs/Stream-API.md ---
@@ -0,0 +1,473 @@
+---
+title: Stream API Overview
+layout: documentation
+documentation: true
+---
+
+* [Concepts](#concepts)
+* [Stream Builder] (#streambuilder)
+* [Value mapper] (#valuemapper)
+* [Stream APIs](#streamapis)
+* [Basic transformations] (#basictransformations)
--- End diff --

Some of the IDE MD plugins and tools are not generating links appropriately 
when there is a space. You can see the syntax 
[here](https://guides.github.com/features/mastering-markdown/) and [here] 
(https://en.support.wordpress.com/markdown-quick-reference/)

Even Github comments throws broken links when there is space between [] and 
(), you can observe above `here` links to see the difference. I guess it is 
better to address that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

2017-02-21 Thread satishd

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/1945#discussion_r102216885
  
--- Diff: docs/Stream-API.md ---
@@ -0,0 +1,473 @@
+---
+title: Stream API Overview
+layout: documentation
+documentation: true
+---
+
+* [Concepts](#concepts)
+* [Stream Builder] (#streambuilder)
--- End diff --

Some of the IDE MD plugins and tools are not generating links appropriately 
when there is a space. You can see the syntax 
[here](https://guides.github.com/features/mastering-markdown/) and [here] 
(https://en.support.wordpress.com/markdown-quick-reference/)

Even Github comments throws broken links when there is space between [] and 
(), you can observe above `here` links to see the difference. I guess it is 
better to address that.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1950: STORM-2369 [storm-redis] Use binary type for State...

2017-02-21 Thread HeartSaVioR

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1950

STORM-2369 [storm-redis] Use binary type for State management (1.x)

* introduce new command interface and relevant containers
  * new command interface will contain both binary and string commands in 
near future
* change RedisKeyValue to use byte[] for State key and value management 
instead of String
  * get rid of Base64 encode/decode on RedisEncoder, and also 
SafeEncoder.encode() in Jedis internal
* implement some utils for handling Map with byte[]: since byte[] is bad 
for key type of Map

NOTE: Currently I can't run or debug tests on IntelliJ for master branch 
(so worked on 1.x-branch). I also heard many folks are struggling, so might be 
better to address it soon.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-2369-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1950.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1950


commit 1c73ceeca7d78fde44cead5d7b9b5b8d0ad9e7e4
Author: Jungtaek Lim 
Date:   2017-02-21T06:35:58Z

STORM-2369 [storm-redis] Use binary type for State management

* introduce new command interface and relevant containers
  * new command interface will contain both binary and string commands in 
near future
* change RedisKeyValue to use byte[] for State key and value management 
instead of String
  * get rid of Base64 encode/decode on RedisEncoder, and also 
SafeEncoder.encode() in Jedis internal
* implement some utils for handling Map with byte[]: since byte[] is bad 
for key type of Map




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

2017-02-21 Thread arunmahadevan

Github user arunmahadevan commented on a diff in the pull request:

https://github.com/apache/storm/pull/1945#discussion_r102196464
  
--- Diff: docs/Stream-API.md ---
@@ -0,0 +1,473 @@
+---
+title: Stream API Overview
+layout: documentation
+documentation: true
+---
+
+* [Concepts](#concepts)
+* [Stream Builder] (#streambuilder)
+* [Value mapper] (#valuemapper)
+* [Stream APIs](#streamapis)
+* [Basic transformations] (#basictransformations)
+* [filter] (#filter)
+* [map] (#map)
+* [flatmap] (#flatmap)
+* [Windowing] (#windowing)
+* [Transformation to key-value pairs] (#keyvaluepairs)
+* [mapToPair] (#mapflatmaptopair)
+* [flatMapToPair] (#mapflatmaptopair)
+* [Aggregations] (#aggregations)
+* [aggregate] (#aggregatereduce)
+* [reduce] (#aggregatereduce)
+* [aggregateByKey] (#aggregatereducebykey)
+* [reduceByKey] (#aggregatereducebykey)
+* [groupByKey] (#groupbykey)
+* [countByKey] (#countbykey)
+* [Repartition](#repartition)
+* [Output operations](#outputoperations)
+* [print](#print)
+* [peek](#peek)
+* [forEach](#foreach)
+* [to](#to)
+* [Branch](#branching)
+* [Joins](#joins)
+* [State](#state)
+* [updateStateByKey](#updatestatebykey)
+* [stateQuery](#statequery)
+* [Guarantees](#guarantees)
+* [Example](#example)
+
+Historically Storm provided Spout and Bolt apis for expressing streaming 
computations. Though these apis are fairly simple to use, 
+there are no reusable constructs for expressing common streaming 
operations like filtering, transformations, windowing, joins, 
+aggregations and so on.
+
+Stream APIs build on top of the Storm's spouts and bolts to provide a 
typed API for expressing streaming computations and supports functional style 
operations such as map-reduce. 
+
+#  Concepts 
+
+Conceptually a `Stream` can be thought of as a stream of messages flowing 
through a pipeline. A `Stream` may be generated by reading messages out of a 
source like spout, or by transforming other streams. For example,
+
+```java
+// imports
+import org.apache.storm.streams.Stream;
+import org.apache.storm.streams.StreamBuilder;
+...
+
+StreamBuilder builder = new StreamBuilder();
+
+// a stream of sentences obtained from a source spout
+Stream sentences = builder.newStream(new 
RandomSentenceSpout()).map(tuple -> tuple.getString(0));
+
+// a stream of words obtained by transforming (splitting) the stream of 
sentences
+Stream words = sentences.flatMap(s -> Arrays.asList(s.split(" ")));
+
+// output operation that prints the words to console
+words.forEach(w -> System.out.println(w));
+```
+
+
+Most stream operations accept parameters that describe user-specified 
behavior typically via lambda expressions like `s -> Arrays.asList(s.split(" 
"))` as in the above example.
+
+A `Stream` supports two kinds of operations, 
+
+1. **Transformations** that produce another stream from the current stream 
 (like the `flatMap` operation in the example above) 
+1. **Output operations** that produce a result. (like the `forEach` 
operation in the example above).
+
+##  Stream Builder
+
+`StreamBuilder` provides the builder apis to create a new stream. 
Typically a spout forms the source of a stream.
+
+```java
+StreamBuilder builder = new StreamBuilder();
+Stream sentences = builder.newStream(new TestSentenceSpout());
+```
+
+The `StreamBuilder` tracks the overall pipeline of operations expressed 
via the Stream. One can then create the Storm topology
+via `build()` and submit it like a normal storm topology via 
`StormSubmitter`.
+
+```java
+StormSubmitter.submitTopologyWithProgressBar("test", new Config(), 
streamBuilder.build());
+```
+
+##  Value mapper
+
+Value mappers can be used to extract specific fields from the tuples 
emitted from a spout to produce a typed stream of values. Value mappers are 
passed as arguments to the `StreamBuilder.newStream`.
+
+```java
+StreamBuilder builder = new StreamBuilder();
+
+// extract the first field from the tuple to get a Stream of 
sentences
+Stream sentences = builder.newStream(new TestWordSpout(), new 
ValueMapper(0));
+```
+
+Storm provides strongly typed tuples via the `Pair` and Tuple classes 
(Tuple3 upto Tuple10). One can use a `TupleValueMapper` to produce a stream of 
typed tuples as shown below.
+
+```java
+// extract first three fields of the tuple emitted by the spout to produce 
a stream of typed tuples.
+Stream> stream = builder.newStream(new

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

2017-02-21 Thread arunmahadevan

Github user arunmahadevan commented on a diff in the pull request:

https://github.com/apache/storm/pull/1945#discussion_r102196192
  
--- Diff: docs/Stream-API.md ---
@@ -0,0 +1,473 @@
+---
+title: Stream API Overview
+layout: documentation
+documentation: true
+---
+
+* [Concepts](#concepts)
+* [Stream Builder] (#streambuilder)
--- End diff --

It seems the space does not matter. I am able to click on the links here - 
https://github.com/arunmahadevan/storm/blob/STORM-2367/docs/Stream-API.md


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

2017-02-21 Thread arunmahadevan

Github user arunmahadevan commented on a diff in the pull request:

https://github.com/apache/storm/pull/1945#discussion_r102196235
  
--- Diff: docs/Stream-API.md ---
@@ -0,0 +1,473 @@
+---
+title: Stream API Overview
+layout: documentation
+documentation: true
+---
+
+* [Concepts](#concepts)
+* [Stream Builder] (#streambuilder)
+* [Value mapper] (#valuemapper)
+* [Stream APIs](#streamapis)
+* [Basic transformations] (#basictransformations)
--- End diff --

It seems the space does not matter. I am able to click on the links here - 
https://github.com/arunmahadevan/storm/blob/STORM-2367/docs/Stream-API.md


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1954: STORM-2372: Pacemaker client doesn't clean up heartbeats ...

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

[GitHub] storm pull request #1957: STORM-2373: HDFS Spout should support multiple ign...

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

[GitHub] storm issue #1942: STORM-2358: Update storm hdfs spout to remove specific im...

[GitHub] storm issue #1934: STORM-2333: CGroup memory and CPU metrics

[GitHub] storm pull request #1934: STORM-2333: CGroup memory and CPU metrics

[GitHub] storm pull request #1934: STORM-2333: CGroup memory and CPU metrics

[GitHub] storm pull request #1934: STORM-2333: CGroup memory and CPU metrics

[GitHub] storm pull request #1934: STORM-2333: CGroup memory and CPU metrics

[GitHub] storm issue #1956: Add Storm SQL docs to index page for 1.x branch

[GitHub] storm pull request #1956: Add Storm SQL docs to index page for 1.x branch

[GitHub] storm pull request #1955: Add Streams API and Storm SQL docs to index page

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

[GitHub] storm issue #1945: [STORM-2367] Documentation for streams API

[GitHub] storm issue #1949: STORM-1290: port backtype.storm.local-state-test to java

[GitHub] storm pull request #1954: STORM-2372: Pacemaker client doesn't clean up hear...

[GitHub] storm pull request #1953: STORM-2372: Pacemaker client doesn't clean up hear...

[GitHub] storm pull request #1952: STORM-2372: Pacemaker client doesn't clean up hear...

[GitHub] storm pull request #1951: STORM-2371 Implementing new eventhub driver

[GitHub] storm issue #1934: STORM-2333: CGroup memory and CPU metrics

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

Re: [VOTE] Release Apache Storm 1.1.0 (RC1)

[GitHub] storm issue #1950: STORM-2369 [storm-redis] Use binary type for State manage...

[GitHub] storm issue #1950: STORM-2369 [storm-redis] Use binary type for State manage...

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

[GitHub] storm pull request #1950: STORM-2369 [storm-redis] Use binary type for State...

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

[GitHub] storm pull request #1945: [STORM-2367] Documentation for streams API

31 matches

Site Navigation

Mail list logo

Footer information