from:"sriharsha"

[3/3] storm git commit: Added STORM-2499 to CHANGELOG.

2017-05-19 Thread sriharsha

Added STORM-2499 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/e33c415e
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/e33c415e
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/e33c415e

Branch: refs/heads/master
Commit: e33c415e9cba89b6282b70af5c7f285071f62dfb
Parents: a8cf365
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri May 19 13:52:45 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri May 19 13:52:45 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/e33c415e/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 024f861..b2ed259 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ï»¿## 2.0.0
+ * STORM-2499: Add Serialization plugin for EventHub System Properties
  * STORM-2520: AutoHDFS should prefer cluster-wise hdfs kerberos principal
  * STORM-2519: Modify AbstractAutoCreds to look for configKeys in both nimbus 
and topology configs
  * STORM-2493: update documents to reflect the changes

[2/3] storm git commit: Merge branch 'eventhub4' of https://github.com/rban1/storm into STORM-2499

2017-05-19 Thread sriharsha

Merge branch 'eventhub4' of https://github.com/rban1/storm into STORM-2499


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/a8cf3653
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/a8cf3653
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/a8cf3653

Branch: refs/heads/master
Commit: a8cf3653bafb6b0f8b7c0e82e39ea619257671b3
Parents: 64e29f3 ca17c4f
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri May 19 13:51:40 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri May 19 13:51:40 2017 -0700

--
 external/storm-eventhubs/README.md |  4 
 .../eventhubs/spout/BinaryEventDataScheme.java | 17 ++---
 .../storm/eventhubs/spout/EventDataScheme.java |  6 ++
 .../storm/eventhubs/spout/FieldConstants.java  |  1 +
 4 files changed, 17 insertions(+), 11 deletions(-)
--

[1/3] storm git commit: STORM-2499: Add Serialization plugin for EventHub System Properties

2017-05-19 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 64e29f365 -> e33c415e9


STORM-2499: Add Serialization plugin for EventHub System Properties


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/ca17c4ff
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/ca17c4ff
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/ca17c4ff

Branch: refs/heads/master
Commit: ca17c4ff10231a5d93deb3d4ac934140ccec674d
Parents: 9755ff5
Author: Ranjan Banerjee 
Authored: Wed May 3 17:39:31 2017 -0700
Committer: Ranjan Banerjee 
Committed: Tue May 9 16:12:14 2017 -0700

--
 external/storm-eventhubs/README.md |  4 
 .../eventhubs/spout/BinaryEventDataScheme.java | 17 ++---
 .../storm/eventhubs/spout/EventDataScheme.java |  6 ++
 .../storm/eventhubs/spout/FieldConstants.java  |  1 +
 4 files changed, 17 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/ca17c4ff/external/storm-eventhubs/README.md
--
diff --git a/external/storm-eventhubs/README.md 
b/external/storm-eventhubs/README.md
index 1586ca8..681ab2d 100755
--- a/external/storm-eventhubs/README.md
+++ b/external/storm-eventhubs/README.md
@@ -32,6 +32,10 @@ We have included a simple EventHubs send client for testing 
purpose. You can run
[username] [password] [entityPath] [partitionId] [messageSize] 
[messageCount]
 If you want to send messages to all partitions, use "-1" as partitionId.
 
+### Serialization Scheme ###
+By default the serialization scheme is StringEventDataScheme where only 
the body of the eventdata is being sent.
+To have a more comprehensive Eventhub metadata exposure look into the 
BinaryEventDataScheme.
+
 ### Windows Azure Eventhubs ###
http://azure.microsoft.com/en-us/services/event-hubs/
 

http://git-wip-us.apache.org/repos/asf/storm/blob/ca17c4ff/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/BinaryEventDataScheme.java
--
diff --git 
a/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/BinaryEventDataScheme.java
 
b/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/BinaryEventDataScheme.java
index 99d764c..5c767e8 100644
--- 
a/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/BinaryEventDataScheme.java
+++ 
b/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/BinaryEventDataScheme.java
@@ -31,9 +31,11 @@ import java.util.Map;
 /**
  * An Event Data Scheme which deserializes message payload into the raw bytes.
  *
- * The resulting tuple would contain two items, the first being the message
+ * The resulting tuple would contain three items, the first being the message
  * bytes, and the second a map of properties that include metadata, which can 
be
- * used to determine who processes the message, and how it is processed.
+ * used to determine who processes the message, and how it is processed.The 
third is
+ * the system properties which exposes information like enqueue-time, offset 
and
+ * sequence number
  */
 public class BinaryEventDataScheme implements IEventDataScheme {
 
@@ -56,16 +58,17 @@ public class BinaryEventDataScheme implements 
IEventDataScheme {
throw new RuntimeException(e);
}
}
-   Map metaDataMap = eventData.getProperties().size() > 0 ? 
eventData.getProperties() : null;
+   Map metaDataMap =  eventData.getProperties();
+   Map systemMetaDataMap = eventData.getSystemProperties();
fieldContents.add(messageData);
-   if ( metaDataMap != null ) {
-   fieldContents.add(metaDataMap);
-   }
+   fieldContents.add(metaDataMap);
+   fieldContents.add(systemMetaDataMap);
return fieldContents;
}
 
@Override
public Fields getOutputFields() {
-   return new Fields(FieldConstants.Message, 
FieldConstants.META_DATA);
+   return new Fields(FieldConstants.Message, 
FieldConstants.META_DATA,
+   FieldConstants.SYSTEM_META_DATA);
}
 }

http://git-wip-us.apache.org/repos/asf/storm/blob/ca17c4ff/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/EventDataScheme.java
--
diff --git 
a/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/EventDataScheme.java

[1/3] storm git commit: STORM-2496 Dependency artifacts should be uploaded to blobstore with READ permission for all

2017-05-15 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 6f8bdccea -> c2950e490


STORM-2496 Dependency artifacts should be uploaded to blobstore with READ 
permission for all

* When uploading dependencies, set ACL properly so that it can be shared to 
other users as well
  * but allows only READ so that it can't be deleted from others


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/cca9ba71
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/cca9ba71
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/cca9ba71

Branch: refs/heads/1.x-branch
Commit: cca9ba71ea595b79c16842180029e8475bbcd096
Parents: 95ec555
Author: Jungtaek Lim 
Authored: Mon May 8 12:14:25 2017 +0900
Committer: Jungtaek Lim 
Committed: Mon May 8 13:06:35 2017 +0900

--
 .../apache/storm/dependency/DependencyUploader.java  | 13 ++---
 .../storm/dependency/DependencyUploaderTest.java | 15 ++-
 2 files changed, 24 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/cca9ba71/storm-core/src/jvm/org/apache/storm/dependency/DependencyUploader.java
--
diff --git 
a/storm-core/src/jvm/org/apache/storm/dependency/DependencyUploader.java 
b/storm-core/src/jvm/org/apache/storm/dependency/DependencyUploader.java
index 636f454..16a3f61 100644
--- a/storm-core/src/jvm/org/apache/storm/dependency/DependencyUploader.java
+++ b/storm-core/src/jvm/org/apache/storm/dependency/DependencyUploader.java
@@ -20,8 +20,10 @@ package org.apache.storm.dependency;
 import com.google.common.annotations.VisibleForTesting;
 import org.apache.storm.blobstore.AtomicOutputStream;
 import org.apache.storm.blobstore.BlobStoreUtils;
+import org.apache.storm.blobstore.BlobStoreAclHandler;
 import org.apache.storm.blobstore.ClientBlobStore;
 import org.apache.storm.generated.AccessControl;
+import org.apache.storm.generated.AccessControlType;
 import org.apache.storm.generated.AuthorizationException;
 import org.apache.storm.generated.KeyAlreadyExistsException;
 import org.apache.storm.generated.KeyNotFoundException;
@@ -146,9 +148,14 @@ public class DependencyUploader {
 // as a workaround, we call getBlobMeta() for all keys
 getBlobStore().getBlobMeta(key);
 } catch (KeyNotFoundException e) {
-// TODO: do we want to add ACL here?
-AtomicOutputStream blob = getBlobStore()
-.createBlob(key, new SettableBlobMeta(new 
ArrayList()));
+// set acl to below so that it can be shared by other users as 
well, but allows only read
+List acls = new ArrayList<>();
+acls.add(new AccessControl(AccessControlType.USER,
+BlobStoreAclHandler.READ | BlobStoreAclHandler.WRITE | 
BlobStoreAclHandler.ADMIN));
+acls.add(new AccessControl(AccessControlType.OTHER,
+BlobStoreAclHandler.READ));
+
+AtomicOutputStream blob = getBlobStore().createBlob(key, new 
SettableBlobMeta(acls));
 Files.copy(dependency.toPath(), blob);
 blob.close();
 

http://git-wip-us.apache.org/repos/asf/storm/blob/cca9ba71/storm-core/test/jvm/org/apache/storm/dependency/DependencyUploaderTest.java
--
diff --git 
a/storm-core/test/jvm/org/apache/storm/dependency/DependencyUploaderTest.java 
b/storm-core/test/jvm/org/apache/storm/dependency/DependencyUploaderTest.java
index 526e758..e5d9faf 100644
--- 
a/storm-core/test/jvm/org/apache/storm/dependency/DependencyUploaderTest.java
+++ 
b/storm-core/test/jvm/org/apache/storm/dependency/DependencyUploaderTest.java
@@ -20,14 +20,17 @@ package org.apache.storm.dependency;
 import com.google.common.collect.Lists;
 import com.google.common.io.Files;
 import org.apache.storm.blobstore.AtomicOutputStream;
-import org.apache.storm.blobstore.BlobStoreUtils;
+import org.apache.storm.blobstore.BlobStoreAclHandler;
 import org.apache.storm.blobstore.ClientBlobStore;
+import org.apache.storm.generated.AccessControl;
+import org.apache.storm.generated.AccessControlType;
 import org.apache.storm.generated.KeyNotFoundException;
 import org.apache.storm.generated.ReadableBlobMeta;
 import org.apache.storm.generated.SettableBlobMeta;
 import org.junit.After;
 import org.junit.Before;
 import org.junit.Test;
+import org.mockito.ArgumentCaptor;
 import org.mockito.invocation.InvocationOnMock;
 import org.mockito.stubbing.Answer;
 
@@ -171,6 +174,16 @@ public class DependencyUploaderTest {
 
 assertTrue(counter.get() > 0);
 verify(mockOutputStream).close();
+
+ArgumentCaptor blobMetaArgumentCaptor =

[2/3] storm git commit: Merge branch 'STORM-2496-1.x' of https://github.com/HeartSaVioR/storm into 1.x-branch

2017-05-15 Thread sriharsha

Merge branch 'STORM-2496-1.x' of https://github.com/HeartSaVioR/storm into 
1.x-branch


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/3785dead
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/3785dead
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/3785dead

Branch: refs/heads/1.x-branch
Commit: 3785dead38b4be93c04b95b6431729472b5390f9
Parents: 6f8bdcc cca9ba7
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Mon May 15 08:14:49 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Mon May 15 08:14:49 2017 -0700

--
 .../apache/storm/dependency/DependencyUploader.java  | 13 ++---
 .../storm/dependency/DependencyUploaderTest.java | 15 ++-
 2 files changed, 24 insertions(+), 4 deletions(-)
--

[3/3] storm git commit: Added STORM-2496 to CHANGELOG.

2017-05-15 Thread sriharsha

Added STORM-2496 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/c2950e49
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/c2950e49
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/c2950e49

Branch: refs/heads/1.x-branch
Commit: c2950e490c95cb56c5a04088a1e07754ad5929fe
Parents: 3785dea
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Mon May 15 08:15:38 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Mon May 15 08:15:38 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/c2950e49/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9290990..662c191 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ï»¿## 1.1.1
+ * STORM-2496: Dependency artifacts should be uploaded to blobstore with READ 
permission for all
  * STORM-2512: Make constructor public and add one more builder constructor
  * STORM-2505: Spout to support topic compaction
  * STORM-2498: Fix Download Full File link

storm git commit: Added STORM-2496 to CHANGELOG.

2017-05-15 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master dd6b48708 -> 013edc967


Added STORM-2496 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/013edc96
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/013edc96
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/013edc96

Branch: refs/heads/master
Commit: 013edc967ff7b290bf5e384b343715c522fb2c1b
Parents: dd6b487
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Mon May 15 08:13:33 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Mon May 15 08:13:33 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/013edc96/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1a08e8a..5ff28bd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -245,6 +245,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.1
+ * STORM-2496: Dependency artifacts should be uploaded to blobstore with READ 
permission for all
  * STORM-2512: Make constructor public and add one more builder constructor
  * STORM-2482: Refactor the Storm auto credential plugins to be more usable
  * STORM-2343: New Kafka spout can stop emitting tuples if more than 
maxUncommittedOffsets tuples fail at once.

[2/2] storm git commit: Merge branch 'STORM-2496' of https://github.com/HeartSaVioR/storm

2017-05-15 Thread sriharsha

Merge branch 'STORM-2496' of https://github.com/HeartSaVioR/storm


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/dd6b4870
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/dd6b4870
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/dd6b4870

Branch: refs/heads/master
Commit: dd6b4870844a6854644968000198871c7b82a3f8
Parents: 7043dea f56ab44
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Mon May 15 08:12:02 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Mon May 15 08:12:02 2017 -0700

--
 .../apache/storm/dependency/DependencyUploader.java   | 13 ++---
 .../storm/dependency/DependencyUploaderTest.java  | 14 ++
 2 files changed, 24 insertions(+), 3 deletions(-)
--

[1/2] storm git commit: STORM-2496 Dependency artifacts should be uploaded to blobstore with READ permission for all

2017-05-15 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 7043dea8e -> dd6b48708


STORM-2496 Dependency artifacts should be uploaded to blobstore with READ 
permission for all

* When uploading dependencies, set ACL properly so that it can be shared to 
other users as well
  * but allows only READ so that it can't be deleted from others


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/f56ab447
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/f56ab447
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/f56ab447

Branch: refs/heads/master
Commit: f56ab4477e28c500575934c987780478bc1319b7
Parents: 5b270d3
Author: Jungtaek Lim 
Authored: Mon May 8 12:14:25 2017 +0900
Committer: Jungtaek Lim 
Committed: Mon May 8 12:25:30 2017 +0900

--
 .../apache/storm/dependency/DependencyUploader.java   | 13 ++---
 .../storm/dependency/DependencyUploaderTest.java  | 14 ++
 2 files changed, 24 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/f56ab447/storm-client/src/jvm/org/apache/storm/dependency/DependencyUploader.java
--
diff --git 
a/storm-client/src/jvm/org/apache/storm/dependency/DependencyUploader.java 
b/storm-client/src/jvm/org/apache/storm/dependency/DependencyUploader.java
index 6fac380..3514a86 100644
--- a/storm-client/src/jvm/org/apache/storm/dependency/DependencyUploader.java
+++ b/storm-client/src/jvm/org/apache/storm/dependency/DependencyUploader.java
@@ -19,8 +19,10 @@ package org.apache.storm.dependency;
 
 import com.google.common.annotations.VisibleForTesting;
 import org.apache.storm.blobstore.AtomicOutputStream;
+import org.apache.storm.blobstore.BlobStoreAclHandler;
 import org.apache.storm.blobstore.ClientBlobStore;
 import org.apache.storm.generated.AccessControl;
+import org.apache.storm.generated.AccessControlType;
 import org.apache.storm.generated.AuthorizationException;
 import org.apache.storm.generated.KeyAlreadyExistsException;
 import org.apache.storm.generated.KeyNotFoundException;
@@ -145,9 +147,14 @@ public class DependencyUploader {
 // as a workaround, we call getBlobMeta() for all keys
 getBlobStore().getBlobMeta(key);
 } catch (KeyNotFoundException e) {
-// TODO: do we want to add ACL here?
-AtomicOutputStream blob = getBlobStore()
-.createBlob(key, new SettableBlobMeta(new 
ArrayList()));
+// set acl to below so that it can be shared by other users as 
well, but allows only read
+List acls = new ArrayList<>();
+acls.add(new AccessControl(AccessControlType.USER,
+BlobStoreAclHandler.READ | BlobStoreAclHandler.WRITE | 
BlobStoreAclHandler.ADMIN));
+acls.add(new AccessControl(AccessControlType.OTHER,
+BlobStoreAclHandler.READ));
+
+AtomicOutputStream blob = getBlobStore().createBlob(key, new 
SettableBlobMeta(acls));
 Files.copy(dependency.toPath(), blob);
 blob.close();
 

http://git-wip-us.apache.org/repos/asf/storm/blob/f56ab447/storm-client/test/jvm/org/apache/storm/dependency/DependencyUploaderTest.java
--
diff --git 
a/storm-client/test/jvm/org/apache/storm/dependency/DependencyUploaderTest.java 
b/storm-client/test/jvm/org/apache/storm/dependency/DependencyUploaderTest.java
index 0ea9479..90d6d4d 100644
--- 
a/storm-client/test/jvm/org/apache/storm/dependency/DependencyUploaderTest.java
+++ 
b/storm-client/test/jvm/org/apache/storm/dependency/DependencyUploaderTest.java
@@ -20,13 +20,17 @@ package org.apache.storm.dependency;
 import com.google.common.collect.Lists;
 import com.google.common.io.Files;
 import org.apache.storm.blobstore.AtomicOutputStream;
+import org.apache.storm.blobstore.BlobStoreAclHandler;
 import org.apache.storm.blobstore.ClientBlobStore;
+import org.apache.storm.generated.AccessControl;
+import org.apache.storm.generated.AccessControlType;
 import org.apache.storm.generated.KeyNotFoundException;
 import org.apache.storm.generated.ReadableBlobMeta;
 import org.apache.storm.generated.SettableBlobMeta;
 import org.junit.After;
 import org.junit.Before;
 import org.junit.Test;
+import org.mockito.ArgumentCaptor;
 import org.mockito.invocation.InvocationOnMock;
 import org.mockito.stubbing.Answer;
 
@@ -170,6 +174,16 @@ public class DependencyUploaderTest {
 
 assertTrue(counter.get() > 0);
 verify(mockOutputStream).close();
+
+ArgumentCaptor blobMetaArgumentCaptor = 
ArgumentCaptor.forClass(SettableBlobMeta.class);
+verify(mockBlobStore).createBlob(anyString(),

[1/2] storm git commit: STORM-2512: Make constructor public and add one more builder constructor

2017-05-14 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master fac6e0b4f -> fd9b25ba1


STORM-2512: Make constructor public and add one more builder constructor


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/9c616fd5
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/9c616fd5
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/9c616fd5

Branch: refs/heads/master
Commit: 9c616fd5ee887e2cf17582b4e80adc3b82158c37
Parents: fac6e0b
Author: Priyank 
Authored: Fri May 12 23:32:08 2017 -0700
Committer: Priyank 
Committed: Fri May 12 23:45:17 2017 -0700

--
 .../java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java  | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/9c616fd5/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
--
diff --git 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
index 5c4c4f5..e89ef93 100644
--- 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
+++ 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
@@ -107,7 +107,11 @@ public class KafkaSpoutConfig implements 
Serializable {
 private KafkaSpoutRetryService retryService = DEFAULT_RETRY_SERVICE;
 private long partitionRefreshPeriodMs = 
DEFAULT_PARTITION_REFRESH_PERIOD_MS;
 private boolean emitNullTuples = false;
-
+
+public Builder(String bootstrapServers, String ... topics) {
+this(bootstrapServers, (SerializableDeserializer) null, 
(SerializableDeserializer) null, new NamedSubscription(topics));
+}
+
 public Builder(String bootstrapServers, SerializableDeserializer 
keyDes, SerializableDeserializer valDes, String ... topics) {
 this(bootstrapServers, keyDes, valDes, new 
NamedSubscription(topics));
 }
@@ -435,7 +439,7 @@ public class KafkaSpoutConfig implements Serializable 
{
 private final long partitionRefreshPeriodMs;
 private final boolean emitNullTuples;
 
-private KafkaSpoutConfig(Builder builder) {
+public KafkaSpoutConfig(Builder builder) {
 this.kafkaProps = setDefaultsAndGetKafkaProps(builder.kafkaProps);
 this.subscription = builder.subscription;
 this.translator = builder.translator;

[2/2] storm git commit: Added STORM-2512 to CHANGELOG.

2017-05-14 Thread sriharsha

Added STORM-2512 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/fd9b25ba
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/fd9b25ba
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/fd9b25ba

Branch: refs/heads/master
Commit: fd9b25ba1fb6fff3c61da2a0af6f6551677a2426
Parents: 9c616fd
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Sun May 14 08:58:33 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Sun May 14 08:58:33 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/fd9b25ba/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0fd2142..7fc17df 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -244,6 +244,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.1
+ * STORM-2512: Make constructor public and add one more builder constructor
  * STORM-2482: Refactor the Storm auto credential plugins to be more usable
  * STORM-2343: New Kafka spout can stop emitting tuples if more than 
maxUncommittedOffsets tuples fail at once.
  * STORM-2488: The UI user Must be HTTP

[1/2] storm git commit: STORM-2512: Make constructor public and add one more builder constructor

2017-05-14 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 979153be8 -> 6f8bdccea


STORM-2512: Make constructor public and add one more builder constructor


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/5ff7865c
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/5ff7865c
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/5ff7865c

Branch: refs/heads/1.x-branch
Commit: 5ff7865cf0b86f40e99b54e789fa60b8843191aa
Parents: 979153b
Author: Priyank 
Authored: Fri May 12 23:32:08 2017 -0700
Committer: Priyank 
Committed: Fri May 12 23:32:08 2017 -0700

--
 .../java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java  | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/5ff7865c/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
--
diff --git 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
index 5f8071f..e1c6c96 100644
--- 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
+++ 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
@@ -107,7 +107,11 @@ public class KafkaSpoutConfig implements 
Serializable {
 private KafkaSpoutRetryService retryService = DEFAULT_RETRY_SERVICE;
 private long partitionRefreshPeriodMs = 
DEFAULT_PARTITION_REFRESH_PERIOD_MS;
 private boolean emitNullTuples = false;
-
+
+public Builder(String bootstrapServers, String ... topics) {
+this(bootstrapServers, (SerializableDeserializer) null, 
(SerializableDeserializer) null, new NamedSubscription(topics));
+}
+
 public Builder(String bootstrapServers, SerializableDeserializer 
keyDes, SerializableDeserializer valDes, String ... topics) {
 this(bootstrapServers, keyDes, valDes, new 
NamedSubscription(topics));
 }
@@ -435,7 +439,7 @@ public class KafkaSpoutConfig implements Serializable 
{
 private final long partitionRefreshPeriodMs;
 private final boolean emitNullTuples;
 
-private KafkaSpoutConfig(Builder builder) {
+public KafkaSpoutConfig(Builder builder) {
 this.kafkaProps = setDefaultsAndGetKafkaProps(builder.kafkaProps);
 this.subscription = builder.subscription;
 this.translator = builder.translator;

[2/2] storm git commit: Added STORM-2512 to CHANGELOG.

2017-05-14 Thread sriharsha

Added STORM-2512 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/6f8bdcce
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/6f8bdcce
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/6f8bdcce

Branch: refs/heads/1.x-branch
Commit: 6f8bdccea3383de9ac5d784180d0d25a721652f5
Parents: 5ff7865
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Sun May 14 08:52:18 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Sun May 14 08:52:18 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/6f8bdcce/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index daf20a4..9290990 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ï»¿## 1.1.1
+ * STORM-2512: Make constructor public and add one more builder constructor
  * STORM-2505: Spout to support topic compaction
  * STORM-2498: Fix Download Full File link
  * STORM-2191: shorten classpaths by using wildcards

[2/2] storm git commit: Added STORM-2505 to CHANGELOG.

2017-05-12 Thread sriharsha

Added STORM-2505 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/fac6e0b4
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/fac6e0b4
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/fac6e0b4

Branch: refs/heads/master
Commit: fac6e0b4fa3509afdba3f921a9e4b762909c7e5f
Parents: 440f1b5
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri May 12 22:19:17 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri May 12 22:19:17 2017 -0700

--
 CHANGELOG.md | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/fac6e0b4/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 955c17b..0fd2142 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -252,6 +252,9 @@
  * STORM-2429: Properly validate supervisor.scheduler.meta
  * STORM-2451: windows storm.cmd does not set log4j2 config file correctly by 
default
  * STORM-2379: ElasticSearch connector changes to use new APIs.
+ * STORM-2505: Spout to support topic compaction
+
+
 
 ## 1.1.0
  * STORM-2416: Release Packaging Improvements

[1/2] storm git commit: [STORM-2505] Spout to support topic compaction

2017-05-12 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 9755ff547 -> fac6e0b4f


[STORM-2505] Spout to support topic compaction

[STORM-2505] Maintaining a emitted set in OffsetManager to handle the voids in 
the topic

[STORM-2505] Handling NPE in Boxed Long to primitive type comparison

[STORM-2505] Rephrased the log message when a non contiguous offset is acked by 
the spout

[STORM-2505] Updated comment

[STORM-2505] Renamed the methods ack/emit to addToAckMsgs and addToEmitMsgs in 
OffsetManager


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/440f1b5d
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/440f1b5d
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/440f1b5d

Branch: refs/heads/master
Commit: 440f1b5d3dd6194feed81a7da21b63b51cde6544
Parents: 9755ff5
Author: Vivek Mittal 
Authored: Mon May 8 10:43:36 2017 +0530
Committer: Vivek Mittal 
Committed: Wed May 10 08:45:41 2017 +0530

--
 .../apache/storm/kafka/spout/KafkaSpout.java| 21 -
 .../kafka/spout/internal/OffsetManager.java | 47 +---
 2 files changed, 53 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/440f1b5d/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
--
diff --git 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
index 164d7a7..310902e 100644
--- 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
+++ 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
@@ -77,7 +77,7 @@ public class KafkaSpout extends BaseRichSpout {
 private transient boolean initialized;  // 
Flag indicating that the spout is still undergoing initialization process.
 // Initialization is only complete after the first call to  
KafkaSpoutConsumerRebalanceListener.onPartitionsAssigned()
 
-private transient Map acked; // 
Tuples that were successfully acked. These tuples will be committed 
periodically when the commit timer expires, or after a consumer rebalance, or 
during close/deactivate
+private transient Map offsetManagers;// 
Tuples that were successfully acked/emitted. These tuples will be committed 
periodically when the commit timer expires, or after a consumer rebalance, or 
during close/deactivate
 private transient Set emitted; // 
Tuples that have been emitted but that are "on the wire", i.e. pending being 
acked or failed. Not used if it's AutoCommitMode
 private transient Iterator> waitingToEmit; // 
Records that have been polled and are queued to be emitted in the nextTuple() 
call. One record is emitted per nextTuple()
 private transient long numUncommittedOffsets;   // 
Number of offsets that have been polled and emitted but not yet been committed. 
Not used if auto commit mode is enabled.
@@ -117,7 +117,7 @@ public class KafkaSpout extends BaseRichSpout {
 }
 refreshSubscriptionTimer = new Timer(TIMER_DELAY_MS, 
kafkaSpoutConfig.getPartitionRefreshPeriodMs(), TimeUnit.MILLISECONDS);
 
-acked = new HashMap<>();
+offsetManagers = new HashMap<>();
 emitted = new HashSet<>();
 waitingToEmit = Collections.emptyListIterator();
 
@@ -147,7 +147,7 @@ public class KafkaSpout extends BaseRichSpout {
 
 private void initialize(Collection partitions) {
 if (!consumerAutoCommitMode) {
-acked.keySet().retainAll(partitions);   // remove from acked 
all partitions that are no longer assigned to this spout
+offsetManagers.keySet().retainAll(partitions);   // remove 
from acked all partitions that are no longer assigned to this spout
 }
 
 retryService.retainAll(partitions);
@@ -205,8 +205,8 @@ public class KafkaSpout extends BaseRichSpout {
 
 private void setAcked(TopicPartition tp, long fetchOffset) {
 // If this partition was previously assigned to this spout, leave the 
acked offsets as they were to resume where it left off
-if (!consumerAutoCommitMode && !acked.containsKey(tp)) {
-acked.put(tp, new OffsetManager(tp, fetchOffset));
+if (!consumerAutoCommitMode && !offsetManagers.containsKey(tp)) {
+offsetManagers.put(tp, new OffsetManager(tp, fetchOffset));
 }
 }
 
@@ -319,7 +319,7 @@ public class KafkaSpout extends

[3/3] storm git commit: Added STORM-2505 to CHANGELOG.

2017-05-12 Thread sriharsha

Added STORM-2505 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/979153be
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/979153be
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/979153be

Branch: refs/heads/1.x-branch
Commit: 979153be8ae09ecdc368d10a5aa01db7685ef02c
Parents: 958f53f
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri May 12 22:04:18 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri May 12 22:04:18 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/979153be/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3d9e053..daf20a4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ï»¿## 1.1.1
+ * STORM-2505: Spout to support topic compaction
  * STORM-2498: Fix Download Full File link
  * STORM-2191: shorten classpaths by using wildcards
  * STORM-2482: Refactor the Storm auto credential plugins to be more usable

[2/3] storm git commit: Merge branch '1.x-branch' of https://github.com/vivekmittal/storm into 1.x-branch

2017-05-12 Thread sriharsha

Merge branch '1.x-branch' of https://github.com/vivekmittal/storm into 
1.x-branch


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/958f53f0
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/958f53f0
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/958f53f0

Branch: refs/heads/1.x-branch
Commit: 958f53f013363128189c0dd022d8d2bdec8e1078
Parents: ea7e7f7 a2dde20
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri May 12 22:03:32 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri May 12 22:03:32 2017 -0700

--
 .../apache/storm/kafka/spout/KafkaSpout.java| 21 -
 .../kafka/spout/internal/OffsetManager.java | 47 +---
 2 files changed, 53 insertions(+), 15 deletions(-)
--

[1/3] storm git commit: [STORM-2505] Spout to support topic compaction

2017-05-12 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch ea7e7f754 -> 979153be8


[STORM-2505] Spout to support topic compaction

[STORM-2505] Maintaining a emitted set in OffsetManager to handle the voids in 
the topic

[STORM-2505] Handling NPE in Boxed Long to primitive type comparison

[STORM-2505] Rephrased the log message when a non contiguous offset is acked by 
the spout

[STORM-2505] Updated comment

[STORM-2505] Renamed the methods ack/emit to addToAckMsgs and addToEmitMsgs in 
OffsetManager


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/a2dde20c
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/a2dde20c
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/a2dde20c

Branch: refs/heads/1.x-branch
Commit: a2dde20cc4313ec373946e0845742eabdce67017
Parents: 95ec555
Author: Vivek Mittal 
Authored: Mon May 8 10:43:36 2017 +0530
Committer: Vivek Mittal 
Committed: Wed May 10 11:10:27 2017 +0530

--
 .../apache/storm/kafka/spout/KafkaSpout.java| 21 -
 .../kafka/spout/internal/OffsetManager.java | 47 +---
 2 files changed, 53 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/a2dde20c/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
--
diff --git 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
index 47c305b..32542b9 100644
--- 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
+++ 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
@@ -77,7 +77,7 @@ public class KafkaSpout extends BaseRichSpout {
 private transient boolean initialized;  // 
Flag indicating that the spout is still undergoing initialization process.
 // Initialization is only complete after the first call to  
KafkaSpoutConsumerRebalanceListener.onPartitionsAssigned()
 
-private transient Map acked; // 
Tuples that were successfully acked. These tuples will be committed 
periodically when the commit timer expires, or after a consumer rebalance, or 
during close/deactivate
+private transient Map offsetManagers;// 
Tuples that were successfully acked/emitted. These tuples will be committed 
periodically when the commit timer expires, or after a consumer rebalance, or 
during close/deactivate
 private transient Set emitted; // 
Tuples that have been emitted but that are "on the wire", i.e. pending being 
acked or failed. Not used if it's AutoCommitMode
 private transient Iterator> waitingToEmit; // 
Records that have been polled and are queued to be emitted in the nextTuple() 
call. One record is emitted per nextTuple()
 private transient long numUncommittedOffsets;   // 
Number of offsets that have been polled and emitted but not yet been committed. 
Not used if auto commit mode is enabled.
@@ -117,7 +117,7 @@ public class KafkaSpout extends BaseRichSpout {
 }
 refreshSubscriptionTimer = new Timer(TIMER_DELAY_MS, 
kafkaSpoutConfig.getPartitionRefreshPeriodMs(), TimeUnit.MILLISECONDS);
 
-acked = new HashMap<>();
+offsetManagers = new HashMap<>();
 emitted = new HashSet<>();
 waitingToEmit = Collections.emptyListIterator();
 
@@ -147,7 +147,7 @@ public class KafkaSpout extends BaseRichSpout {
 
 private void initialize(Collection partitions) {
 if (!consumerAutoCommitMode) {
-acked.keySet().retainAll(partitions);   // remove from acked 
all partitions that are no longer assigned to this spout
+offsetManagers.keySet().retainAll(partitions);   // remove 
from acked all partitions that are no longer assigned to this spout
 }
 
 retryService.retainAll(partitions);
@@ -205,8 +205,8 @@ public class KafkaSpout extends BaseRichSpout {
 
 private void setAcked(TopicPartition tp, long fetchOffset) {
 // If this partition was previously assigned to this spout, leave the 
acked offsets as they were to resume where it left off
-if (!consumerAutoCommitMode && !acked.containsKey(tp)) {
-acked.put(tp, new OffsetManager(tp, fetchOffset));
+if (!consumerAutoCommitMode && !offsetManagers.containsKey(tp)) {
+offsetManagers.put(tp, new OffsetManager(tp, fetchOffset));
 }
 }
 
@@ -319,7 +319,7 @@ public class KafkaSpout

storm git commit: Added STORM-2482 to CHANGELOG.

2017-05-02 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 4f240f266 -> 95ec555e8


Added STORM-2482 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/95ec555e
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/95ec555e
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/95ec555e

Branch: refs/heads/1.x-branch
Commit: 95ec555e8e9ce91797076ef6000c56c43b6c0a72
Parents: 4f240f2
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue May 2 14:57:23 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue May 2 14:57:23 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/95ec555e/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 62a7b3c..ffce6b7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ï»¿## 1.1.1
+ * STORM-2482: Refactor the Storm auto credential plugins to be more usable
  * STORM-2343: New Kafka spout can stop emitting tuples if more than 
maxUncommittedOffsets tuples fail at once.
  * STORM-2486: Prevent cd from printing target directory to avoid breaking 
classpath
  * STORM-2488: The UI user Must be HTTP.

[1/4] storm git commit: [STORM-2482] Refactor the Storm auto credential plugins to be more usable

2017-05-02 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch e56dd29bd -> 4f240f266


http://git-wip-us.apache.org/repos/asf/storm/blob/19ea744d/pom.xml
--
diff --git a/pom.xml b/pom.xml
index bf54381..7e2c41a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -304,6 +304,7 @@
 storm-rename-hack
 
 
+external/storm-autocreds
 external/storm-kafka
 external/storm-hdfs
 external/storm-hbase

http://git-wip-us.apache.org/repos/asf/storm/blob/19ea744d/storm-dist/binary/src/main/assembly/binary.xml
--
diff --git a/storm-dist/binary/src/main/assembly/binary.xml 
b/storm-dist/binary/src/main/assembly/binary.xml
index 78e0778..7124abd 100644
--- a/storm-dist/binary/src/main/assembly/binary.xml
+++ b/storm-dist/binary/src/main/assembly/binary.xml
@@ -316,6 +316,14 @@
 storm*jar
 
 
+   
+
+
${project.basedir}/../../external/storm-autocreds/target/app-assembler/repo
+external/storm-autocreds
+
+*jar
+
+

[4/4] storm git commit: Merge branch 'STORM-2482-1.x' of https://github.com/arunmahadevan/storm into 1.x-branch

2017-05-02 Thread sriharsha

Merge branch 'STORM-2482-1.x' of https://github.com/arunmahadevan/storm into 
1.x-branch


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/4f240f26
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/4f240f26
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/4f240f26

Branch: refs/heads/1.x-branch
Commit: 4f240f2667059866f6a982844a2c5b9d83f33bcb
Parents: e56dd29 6d08e29
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue May 2 09:51:06 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue May 2 09:51:06 2017 -0700

--
 docs/SECURITY.md|  18 +-
 docs/storm-hbase.md |  37 ++-
 docs/storm-hdfs.md  |  43 ++-
 external/storm-autocreds/pom.xml| 103 +++
 .../apache/storm/common/AbstractAutoCreds.java  | 248 
 .../apache/storm/hbase/security/AutoHBase.java  | 179 
 .../storm/hbase/security/HBaseSecurityUtil.java |  73 +
 .../apache/storm/hdfs/security/AutoHDFS.java| 216 ++
 .../storm/hdfs/security/HdfsSecurityUtil.java   |  69 +
 external/storm-hbase/pom.xml|   5 +
 .../apache/storm/hbase/security/AutoHBase.java  | 243 
 .../storm/hbase/security/HBaseSecurityUtil.java |  72 -
 external/storm-hdfs/pom.xml |   5 +
 .../storm/hdfs/bolt/AbstractHdfsBolt.java   |   2 +-
 .../storm/hdfs/common/security/AutoHDFS.java| 281 ---
 .../hdfs/common/security/HdfsSecurityUtil.java  |  67 -
 .../org/apache/storm/hdfs/spout/HdfsSpout.java  |   2 +-
 .../apache/storm/hdfs/trident/HdfsState.java|   2 +-
 pom.xml |   1 +
 storm-dist/binary/src/main/assembly/binary.xml  |   8 +
 20 files changed, 981 insertions(+), 693 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/4f240f26/docs/SECURITY.md
--

[2/4] storm git commit: [STORM-2482] Refactor the Storm auto credential plugins to be more usable

2017-05-02 Thread sriharsha

[STORM-2482] Refactor the Storm auto credential plugins to be more usable

1. Create a new storm module storm-autocreds
2. Move AutoHDFS and AutoHBase to storm-autocreds
3. Refactor code and accepts config keys for customizing the hadoop 
configuration for the plugins
4. Package the auto cred and dependency jars in the storm binary and deploy to 
lib-autocreds


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/19ea744d
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/19ea744d
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/19ea744d

Branch: refs/heads/1.x-branch
Commit: 19ea744d0aac59187058693fbec7d9a80b470f7c
Parents: 5849e69
Author: Arun Mahadevan 
Authored: Tue Apr 18 13:29:56 2017 +0530
Committer: Arun Mahadevan 
Committed: Fri Apr 21 17:42:42 2017 +0530

--
 docs/SECURITY.md|  18 +-
 docs/storm-hbase.md |  35 ++-
 docs/storm-hdfs.md  |  43 ++-
 external/storm-autocreds/pom.xml| 103 +++
 .../apache/storm/common/AbstractAutoCreds.java  | 250 +
 .../apache/storm/hbase/security/AutoHBase.java  | 179 
 .../storm/hbase/security/HBaseSecurityUtil.java |  73 +
 .../apache/storm/hdfs/security/AutoHDFS.java| 216 ++
 .../storm/hdfs/security/HdfsSecurityUtil.java   |  69 +
 external/storm-hbase/pom.xml|   5 +
 .../apache/storm/hbase/security/AutoHBase.java  | 243 
 .../storm/hbase/security/HBaseSecurityUtil.java |  72 -
 external/storm-hdfs/pom.xml |   5 +
 .../storm/hdfs/bolt/AbstractHdfsBolt.java   |   2 +-
 .../storm/hdfs/common/security/AutoHDFS.java| 281 ---
 .../hdfs/common/security/HdfsSecurityUtil.java  |  67 -
 .../org/apache/storm/hdfs/spout/HdfsSpout.java  |   2 +-
 .../apache/storm/hdfs/trident/HdfsState.java|   2 +-
 pom.xml |   1 +
 storm-dist/binary/src/main/assembly/binary.xml  |   8 +
 20 files changed, 982 insertions(+), 692 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/19ea744d/docs/SECURITY.md
--
diff --git a/docs/SECURITY.md b/docs/SECURITY.md
index a9d0d7f..0acb237 100644
--- a/docs/SECURITY.md
+++ b/docs/SECURITY.md
@@ -423,16 +423,18 @@ nimbus.impersonation.acl:
 
 ### Automatic Credentials Push and Renewal
 Individual topologies have the ability to push credentials (tickets and 
tokens) to workers so that they can access secure services.  Exposing this to 
all of the users can be a pain for them.
-To hide this from them in the common case plugins can be used to populate the 
credentials, unpack them on the other side into a java Subject, and also allow 
Nimbus to renew the credentials if needed.
-These are controlled by the following configs. topology.auto-credentials is a 
list of java plugins, all of which must implement IAutoCredentials interface, 
that populate the credentials on gateway 
-and unpack them on the worker side. On a kerberos secure cluster they should 
be set by default to point to org.apache.storm.security.auth.kerberos.AutoTGT.  
-nimbus.credential.renewers.classes should also be set to this value so that 
nimbus can periodically renew the TGT on behalf of the user.
+To hide this from them in the common case plugins can be used to populate the 
credentials, unpack them on the other side into a java Subject, and also allow 
Nimbus to renew the credentials if needed. These are controlled by the 
following configs.
+ 
+`topology.auto-credentials` is a list of java plugins, all of which must 
implement the `IAutoCredentials` interface, that populate the credentials on 
gateway 
+and unpack them on the worker side. On a kerberos secure cluster they should 
be set by default to point to `org.apache.storm.security.auth.kerberos.AutoTGT`
+
+`nimbus.credential.renewers.classes` should also be set to 
`org.apache.storm.security.auth.kerberos.AutoTGT` so that nimbus can 
periodically renew the TGT on behalf of the user.
 
-nimbus.credential.renewers.freq.secs controls how often the renewer will poll 
to see if anything needs to be renewed, but the default should be fine.
+`nimbus.credential.renewers.freq.secs` controls how often the renewer will 
poll to see if anything needs to be renewed, but the default should be fine.
 
-In addition Nimbus itself can be used to get credentials on behalf of the user 
submitting topologies. This can be configures using 
nimbus.autocredential.plugins.classes which is a list 
-of fully qualified class names ,all of which must implement 
INimbusCredentialPlugin.  Nimbus will invoke the populateCredentials method of 
all

[3/4] storm git commit: [STORM-2482] addressing review comments

2017-05-02 Thread sriharsha

[STORM-2482] addressing review comments


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/6d08e29e
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/6d08e29e
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/6d08e29e

Branch: refs/heads/1.x-branch
Commit: 6d08e29e5577f31696190664a1b94cb9e4bad25d
Parents: 19ea744
Author: Arun Mahadevan 
Authored: Thu Apr 27 18:17:19 2017 +0530
Committer: Arun Mahadevan 
Committed: Thu Apr 27 18:24:19 2017 +0530

--
 docs/storm-hbase.md |  2 +-
 .../java/org/apache/storm/common/AbstractAutoCreds.java | 12 +---
 2 files changed, 6 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/6d08e29e/docs/storm-hbase.md
--
diff --git a/docs/storm-hbase.md b/docs/storm-hbase.md
index 8074244..f8d955e 100644
--- a/docs/storm-hbase.md
+++ b/docs/storm-hbase.md
@@ -98,7 +98,7 @@ topology submission, nimbus will impersonate the topology 
submitter user and acq
 topology submitter user. If topology was started with 
topology.auto-credentials set to AutoHBase, nimbus will push the
 delegation tokens to all the workers for your topology and the hbase 
bolt/state will authenticate with these tokens.
 
-As nimbus is impersonating topology submitter user, you need to ensure the 
user specified in storm.kerberos.principal 
+As nimbus is impersonating topology submitter user, you need to ensure the 
user specified in hbase.kerberos.principal 
 has permissions to acquire tokens on behalf of other users. To achieve this 
you need to follow configuration directions 
 listed on this link
 

http://git-wip-us.apache.org/repos/asf/storm/blob/6d08e29e/external/storm-autocreds/src/main/java/org/apache/storm/common/AbstractAutoCreds.java
--
diff --git 
a/external/storm-autocreds/src/main/java/org/apache/storm/common/AbstractAutoCreds.java
 
b/external/storm-autocreds/src/main/java/org/apache/storm/common/AbstractAutoCreds.java
index 816e263..cc374c9 100644
--- 
a/external/storm-autocreds/src/main/java/org/apache/storm/common/AbstractAutoCreds.java
+++ 
b/external/storm-autocreds/src/main/java/org/apache/storm/common/AbstractAutoCreds.java
@@ -74,7 +74,7 @@ public abstract class AbstractAutoCreds implements 
IAutoCredentials, ICredential
 @Override
 public void populateCredentials(Map credentials, Map conf) 
{
 try {
-if (configKeys != null) {
+if (!configKeys.isEmpty()) {
 Map updatedConf = updateConfigs(conf);
 for (String configKey : configKeys) {
 credentials.put(getCredentialKey(configKey),
@@ -92,11 +92,9 @@ public abstract class AbstractAutoCreds implements 
IAutoCredentials, ICredential
 
 private Map updateConfigs(Map topologyConf) {
 Map res = new HashMap<>(topologyConf);
-if (configKeys != null) {
-for (String configKey : configKeys) {
-if (!res.containsKey(configKey) && 
configMap.containsKey(configKey)) {
-res.put(configKey, configMap.get(configKey));
-}
+for (String configKey : configKeys) {
+if (!res.containsKey(configKey) && 
configMap.containsKey(configKey)) {
+res.put(configKey, configMap.get(configKey));
 }
 }
 return res;
@@ -134,7 +132,7 @@ public abstract class AbstractAutoCreds implements 
IAutoCredentials, ICredential
 
 protected Set> getCredentials(Map credentials) {
 Set> res = new HashSet<>();
-if (configKeys != null) {
+if (!configKeys.isEmpty()) {
 for (String configKey : configKeys) {
 Credentials cred = doGetCredentials(credentials, configKey);
 if (cred != null) {

[3/3] storm git commit: Added STORM-2343 to CHANGELOG.

2017-05-01 Thread sriharsha

Added STORM-2343 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/e56dd29b
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/e56dd29b
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/e56dd29b

Branch: refs/heads/1.x-branch
Commit: e56dd29bdaa6775a8bbb8336e60978234e298ead
Parents: 979b5b0
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Mon May 1 13:51:16 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Mon May 1 13:51:16 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/e56dd29b/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 48e8391..62a7b3c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ï»¿## 1.1.1
+ * STORM-2343: New Kafka spout can stop emitting tuples if more than 
maxUncommittedOffsets tuples fail at once.
  * STORM-2486: Prevent cd from printing target directory to avoid breaking 
classpath
  * STORM-2488: The UI user Must be HTTP.
  * STORM-2481: Upgrade Aether version to resolve Aether bug BUG-451566

[1/3] storm git commit: STORM-2343: Fix new Kafka spout stopping processing if more than maxUncommittedOffsets tuples fail at once

2017-05-01 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch a49ad1854 -> e56dd29bd


STORM-2343: Fix new Kafka spout stopping processing if more than 
maxUncommittedOffsets tuples fail at once


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/fa6a2bd7
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/fa6a2bd7
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/fa6a2bd7

Branch: refs/heads/1.x-branch
Commit: fa6a2bd7f3ae273c8f312b60f4a87f357b1d8b6f
Parents: 609fe10
Author: Stig Rohde DÃ¸ssing 
Authored: Sat Apr 8 08:26:42 2017 +0200
Committer: Stig Rohde DÃ¸ssing 
Committed: Sat Apr 8 08:26:42 2017 +0200

--
 .../apache/storm/kafka/spout/KafkaSpout.java|  26 +-
 .../storm/kafka/spout/KafkaSpoutConfig.java |  10 +-
 .../KafkaSpoutRetryExponentialBackoff.java  |  78 --
 .../kafka/spout/KafkaSpoutRetryService.java |  10 +-
 .../storm/kafka/spout/KafkaSpoutEmitTest.java   | 256 +++
 .../kafka/spout/KafkaSpoutRebalanceTest.java|  11 +-
 .../kafka/spout/MaxUncommittedOffsetTest.java   | 245 ++
 .../kafka/spout/SingleTopicKafkaSpoutTest.java  |  45 +++-
 .../SingleTopicKafkaSpoutConfiguration.java |  19 +-
 9 files changed, 633 insertions(+), 67 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/fa6a2bd7/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
--
diff --git 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
index 207ba23..0bfcfea 100644
--- 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
+++ 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
@@ -31,6 +31,7 @@ import java.util.Iterator;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Map.Entry;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 
@@ -79,7 +80,7 @@ public class KafkaSpout extends BaseRichSpout {
 private transient Map acked; // 
Tuples that were successfully acked. These tuples will be committed 
periodically when the commit timer expires, or after a consumer rebalance, or 
during close/deactivate
 private transient Set emitted; // 
Tuples that have been emitted but that are "on the wire", i.e. pending being 
acked or failed. Not used if it's AutoCommitMode
 private transient Iterator> waitingToEmit; // 
Records that have been polled and are queued to be emitted in the nextTuple() 
call. One record is emitted per nextTuple()
-private transient long numUncommittedOffsets;   // 
Number of offsets that have been polled and emitted but not yet been committed. 
Not used if it's AutoCommitMode
+private transient long numUncommittedOffsets;   // 
Number of offsets that have been polled and emitted but not yet been committed. 
Not used if auto commit mode is enabled.
 private transient Timer refreshSubscriptionTimer;   // 
Triggers when a subscription should be refreshed
 private transient TopologyContext context;
 
@@ -250,9 +251,13 @@ public class KafkaSpout extends BaseRichSpout {
 
 private boolean poll() {
 final int maxUncommittedOffsets = 
kafkaSpoutConfig.getMaxUncommittedOffsets();
-final boolean poll = !waitingToEmit()
-&& (numUncommittedOffsets < maxUncommittedOffsets || 
consumerAutoCommitMode);
-
+final int readyMessageCount = retryService.readyMessageCount();
+final boolean poll = !waitingToEmit() &&
+//Check that the number of uncommitted, nonretriable tuples is 
less than the maxUncommittedOffsets limit
+//Accounting for retriable tuples this way still guarantees that 
the limit is followed on a per partition basis, and prevents locking up the 
spout when there are too many retriable tuples
+(numUncommittedOffsets - readyMessageCount < maxUncommittedOffsets 
||
+consumerAutoCommitMode);
+
 if (!poll) {
 if (waitingToEmit()) {
 LOG.debug("Not polling. Tuples waiting to be emitted. [{}] 
uncommitted offsets across all topic partitions", numUncommittedOffsets);
@@ -290,15 +295,11 @@ public class KafkaSpout extends BaseRichSpout {
 }
 
 private void doSeekRetriableTopicPartitions() {
-final Set retriableTopicPartitions = 
retryService.retriableTopicPartitions();
+final Map

[2/3] storm git commit: Merge branch 'STORM-2343-1.x' of https://github.com/srdo/storm into 1.x-branch

2017-05-01 Thread sriharsha

Merge branch 'STORM-2343-1.x' of https://github.com/srdo/storm into 1.x-branch


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/979b5b03
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/979b5b03
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/979b5b03

Branch: refs/heads/1.x-branch
Commit: 979b5b03955dad00e7db657c4fc083478b520b5f
Parents: a49ad18 fa6a2bd
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Mon May 1 13:50:11 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Mon May 1 13:50:11 2017 -0700

--
 .../apache/storm/kafka/spout/KafkaSpout.java|  26 +-
 .../storm/kafka/spout/KafkaSpoutConfig.java |  10 +-
 .../KafkaSpoutRetryExponentialBackoff.java  |  78 --
 .../kafka/spout/KafkaSpoutRetryService.java |  10 +-
 .../storm/kafka/spout/KafkaSpoutEmitTest.java   | 256 +++
 .../kafka/spout/KafkaSpoutRebalanceTest.java|  11 +-
 .../kafka/spout/MaxUncommittedOffsetTest.java   | 245 ++
 .../kafka/spout/SingleTopicKafkaSpoutTest.java  |  45 +++-
 .../SingleTopicKafkaSpoutConfiguration.java |  19 +-
 9 files changed, 633 insertions(+), 67 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/979b5b03/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
--

[2/3] storm git commit: Merge branch 'STORM-2343' of https://github.com/srdo/storm

2017-05-01 Thread sriharsha

Merge branch 'STORM-2343' of https://github.com/srdo/storm


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/d39e3a77
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/d39e3a77
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/d39e3a77

Branch: refs/heads/master
Commit: d39e3a778873b36af9c120a71c81b883cefc674b
Parents: a4afacd d98a5b8
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Mon May 1 12:55:31 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Mon May 1 12:55:31 2017 -0700

--
 .../apache/storm/kafka/spout/KafkaSpout.java|  26 +-
 .../storm/kafka/spout/KafkaSpoutConfig.java |  10 +-
 .../KafkaSpoutRetryExponentialBackoff.java  |  73 --
 .../kafka/spout/KafkaSpoutRetryService.java |  10 +-
 .../storm/kafka/spout/KafkaSpoutEmitTest.java   | 250 +++
 .../kafka/spout/KafkaSpoutRebalanceTest.java|  11 +-
 .../kafka/spout/MaxUncommittedOffsetTest.java   | 240 ++
 .../kafka/spout/SingleTopicKafkaSpoutTest.java  |  56 +++--
 .../SingleTopicKafkaSpoutConfiguration.java |  19 +-
 9 files changed, 616 insertions(+), 79 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/d39e3a77/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
--

[1/3] storm git commit: STORM-2343: Fix new Kafka spout stopping processing if more than maxUncommittedOffsets tuples fail at once

2017-05-01 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master a4afacd96 -> 2c597e562


STORM-2343: Fix new Kafka spout stopping processing if more than 
maxUncommittedOffsets tuples fail at once


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/d98a5b8b
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/d98a5b8b
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/d98a5b8b

Branch: refs/heads/master
Commit: d98a5b8baaac3595df6d1f43a0f84a33b57de546
Parents: 1850dd5
Author: Stig Rohde DÃ¸ssing 
Authored: Sat Apr 8 08:23:43 2017 +0200
Committer: Stig Rohde DÃ¸ssing 
Committed: Sat Apr 8 08:23:43 2017 +0200

--
 .../apache/storm/kafka/spout/KafkaSpout.java|  26 +-
 .../storm/kafka/spout/KafkaSpoutConfig.java |  10 +-
 .../KafkaSpoutRetryExponentialBackoff.java  |  73 --
 .../kafka/spout/KafkaSpoutRetryService.java |  10 +-
 .../storm/kafka/spout/KafkaSpoutEmitTest.java   | 250 +++
 .../kafka/spout/KafkaSpoutRebalanceTest.java|  11 +-
 .../kafka/spout/MaxUncommittedOffsetTest.java   | 240 ++
 .../kafka/spout/SingleTopicKafkaSpoutTest.java  |  56 +++--
 .../SingleTopicKafkaSpoutConfiguration.java |  19 +-
 9 files changed, 616 insertions(+), 79 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/d98a5b8b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
--
diff --git 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
index 51cdb38..6098119 100644
--- 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
+++ 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
@@ -31,6 +31,7 @@ import java.util.Iterator;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Map.Entry;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 
@@ -79,7 +80,7 @@ public class KafkaSpout extends BaseRichSpout {
 private transient Map acked; // 
Tuples that were successfully acked. These tuples will be committed 
periodically when the commit timer expires, or after a consumer rebalance, or 
during close/deactivate
 private transient Set emitted; // 
Tuples that have been emitted but that are "on the wire", i.e. pending being 
acked or failed. Not used if it's AutoCommitMode
 private transient Iterator> waitingToEmit; // 
Records that have been polled and are queued to be emitted in the nextTuple() 
call. One record is emitted per nextTuple()
-private transient long numUncommittedOffsets;   // 
Number of offsets that have been polled and emitted but not yet been committed. 
Not used if it's AutoCommitMode
+private transient long numUncommittedOffsets;   // 
Number of offsets that have been polled and emitted but not yet been committed. 
Not used if auto commit mode is enabled.
 private transient Timer refreshSubscriptionTimer;   // 
Triggers when a subscription should be refreshed
 private transient TopologyContext context;
 
@@ -250,9 +251,13 @@ public class KafkaSpout extends BaseRichSpout {
 
 private boolean poll() {
 final int maxUncommittedOffsets = 
kafkaSpoutConfig.getMaxUncommittedOffsets();
-final boolean poll = !waitingToEmit()
-&& (numUncommittedOffsets < maxUncommittedOffsets || 
consumerAutoCommitMode);
-
+final int readyMessageCount = retryService.readyMessageCount();
+final boolean poll = !waitingToEmit() &&
+//Check that the number of uncommitted, nonretriable tuples is 
less than the maxUncommittedOffsets limit
+//Accounting for retriable tuples this way still guarantees that 
the limit is followed on a per partition basis, and prevents locking up the 
spout when there are too many retriable tuples
+(numUncommittedOffsets - readyMessageCount < maxUncommittedOffsets 
||
+consumerAutoCommitMode);
+
 if (!poll) {
 if (waitingToEmit()) {
 LOG.debug("Not polling. Tuples waiting to be emitted. [{}] 
uncommitted offsets across all topic partitions", numUncommittedOffsets);
@@ -290,15 +295,11 @@ public class KafkaSpout extends BaseRichSpout {
 }
 
 private void doSeekRetriableTopicPartitions() {
-final Set retriableTopicPartitions = 
retryService.retriableTopicPartitions();
+final Map

[3/3] storm git commit: Added STORM-2343 to CHANGELOG.

2017-05-01 Thread sriharsha

Added STORM-2343 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/2c597e56
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/2c597e56
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/2c597e56

Branch: refs/heads/master
Commit: 2c597e56255ad86a1920b64b6153d6972d37179c
Parents: d39e3a7
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Mon May 1 12:57:08 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Mon May 1 12:57:08 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/2c597e56/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b447269..a3f44f0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -241,6 +241,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.1
+ * STORM-2343: New Kafka spout can stop emitting tuples if more than 
maxUncommittedOffsets tuples fail at once.
  * STORM-2488: The UI user Must be HTTP
  * STORM-2315: New kafka spout can't commit offset when ack is disabled
  * STORM-2467: Use explicit charset when decoding from array backed buffer

[2/2] storm git commit: Added STORM-2488 to CHANGELOG.

2017-04-25 Thread sriharsha

Added STORM-2488 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/ebcfb1ef
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/ebcfb1ef
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/ebcfb1ef

Branch: refs/heads/1.x-branch
Commit: ebcfb1ef27b551ef7c19320a66719fbaafc464db
Parents: a8b2391
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Apr 25 20:42:23 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Apr 25 20:42:23 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/ebcfb1ef/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3130079..af54a8f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ï»¿## 1.1.1
+ * STORM-2488: The UI user Must be HTTP.
  * STORM-2481: Upgrade Aether version to resolve Aether bug BUG-451566
  * STORM-2435: Logging in storm.js inconsistent to console.log and does not 
support log levels
  * STORM-2315: New kafka spout can't commit offset when ack is disabled

[1/2] storm git commit: Merge branch 'patch-1' of https://github.com/MichealShin/storm

2017-04-25 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 5849e69f1 -> ebcfb1ef2


Merge branch 'patch-1' of https://github.com/MichealShin/storm


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/a8b23913
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/a8b23913
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/a8b23913

Branch: refs/heads/1.x-branch
Commit: a8b23913e2e21506e13fcfdffdbb4a5a2d71c46f
Parents: 5849e69
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Apr 25 20:40:08 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Apr 25 20:41:54 2017 -0700

--
 docs/SECURITY.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/a8b23913/docs/SECURITY.md
--
diff --git a/docs/SECURITY.md b/docs/SECURITY.md
index a9d0d7f..66566ce 100644
--- a/docs/SECURITY.md
+++ b/docs/SECURITY.md
@@ -74,6 +74,7 @@ ui.filter.params:
"kerberos.name.rules": 
"RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/ 
RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT"
 ```
 make sure to create a principal 'HTTP/{hostname}' (here hostname should be the 
one where UI daemon runs
+Be aware that the UI user *MUST* be HTTP.
 
 Once configured users needs to do kinit before accessing UI.
 Ex:

[3/3] storm git commit: Added STORM-2488 to CHANGELOG.

2017-04-25 Thread sriharsha

Added STORM-2488 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/41ab0172
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/41ab0172
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/41ab0172

Branch: refs/heads/master
Commit: 41ab01726664c4a67552c2327466ab1290f2dad6
Parents: 0d690c4
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Apr 25 20:40:57 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Apr 25 20:40:57 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/41ab0172/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5c8ffae..2a9ef3d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -238,6 +238,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.1
+ * STORM-2488: The UI user Must be HTTP
  * STORM-2315: New kafka spout can't commit offset when ack is disabled
  * STORM-2467: Use explicit charset when decoding from array backed buffer
  * STORM-2429: Properly validate supervisor.scheduler.meta

[1/3] storm git commit: The UI user Must be HTTP

2017-04-25 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 7e183c010 -> 41ab01726


The UI user Must be HTTP

The UI user Must be HTTP. Otherwise, the UI page can not be authorized

Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/8c3b9797
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/8c3b9797
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/8c3b9797

Branch: refs/heads/master
Commit: 8c3b9797101b313c31f8af5d585eb43c88d54dd0
Parents: bfd1006
Author: MichealShin 
Authored: Mon Apr 24 09:50:39 2017 +0800
Committer: GitHub 
Committed: Mon Apr 24 09:50:39 2017 +0800

--
 docs/SECURITY.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/8c3b9797/docs/SECURITY.md
--
diff --git a/docs/SECURITY.md b/docs/SECURITY.md
index a9d0d7f..66566ce 100644
--- a/docs/SECURITY.md
+++ b/docs/SECURITY.md
@@ -74,6 +74,7 @@ ui.filter.params:
"kerberos.name.rules": 
"RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/ 
RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT"
 ```
 make sure to create a principal 'HTTP/{hostname}' (here hostname should be the 
one where UI daemon runs
+Be aware that the UI user *MUST* be HTTP.
 
 Once configured users needs to do kinit before accessing UI.
 Ex:

[2/3] storm git commit: Merge branch 'patch-1' of https://github.com/MichealShin/storm

2017-04-25 Thread sriharsha

Merge branch 'patch-1' of https://github.com/MichealShin/storm


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/0d690c41
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/0d690c41
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/0d690c41

Branch: refs/heads/master
Commit: 0d690c4105654da266eb726c2d23459fac3bb219
Parents: 7e183c0 8c3b979
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Apr 25 20:40:08 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Apr 25 20:40:08 2017 -0700

--
 docs/SECURITY.md | 1 +
 1 file changed, 1 insertion(+)
--

[3/3] storm git commit: Added STORM-2371 to CHANGELOG.

2017-04-18 Thread sriharsha

Added STORM-2371 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/2307b802
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/2307b802
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/2307b802

Branch: refs/heads/master
Commit: 2307b8023c947a408aac6d522d18e13434c0f57e
Parents: 940096a
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Apr 18 14:08:22 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Apr 18 14:08:22 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/2307b802/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 116c9b3..5f9d118 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ï»¿## 2.0.0
+ * STORM-2371: Replace existing AMQP eventhub client with the lastest one from 
Microsoft eventhubs
  * STORM-2475: Fix parsing of host:port to deal with IPv6 addresses
  * STORM-832: Allow config validation to be used by plugins/etc.
  * STORM-2471: Add metric for thread count

[4/4] storm git commit: Added STORM-2379 to CHANGELOG.

2017-04-18 Thread sriharsha

Added STORM-2379 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/f794d723
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/f794d723
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/f794d723

Branch: refs/heads/master
Commit: f794d72306d040249427000ba2ea7eae5b647cf7
Parents: 55297b0
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Apr 18 12:25:50 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Apr 18 12:25:50 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/f794d723/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 63a4aa1..116c9b3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -238,6 +238,7 @@
  * STORM-2467: Use explicit charset when decoding from array backed buffer
  * STORM-2429: Properly validate supervisor.scheduler.meta
  * STORM-2451: windows storm.cmd does not set log4j2 config file correctly by 
default
+ * STORM-2379: ElasticSearch connector changes to use new APIs.
 
 ## 1.1.0
  * STORM-2416: Release Packaging Improvements

[2/4] storm git commit: STORM-2379: update for Elasticsearch 2.

2017-04-18 Thread sriharsha

STORM-2379: update for Elasticsearch 2.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/2b4565c6
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/2b4565c6
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/2b4565c6

Branch: refs/heads/master
Commit: 2b4565c668c16c366e086214d11f3ec2e1d4769c
Parents: 1a17f74
Author: Heather McCartney 
Authored: Mon Mar 13 13:16:27 2017 +
Committer: Heather McCartney 
Committed: Thu Apr 13 21:30:18 2017 +0100

--
 examples/storm-elasticsearch-examples/pom.xml   |  10 ++
 .../elasticsearch/bolt/EsIndexTopology.java |   3 +-
 .../storm/elasticsearch/common/EsTestUtil.java  |  14 +-
 .../trident/TridentEsTopology.java  |   3 +-
 external/storm-elasticsearch/pom.xml|  44 ++-
 .../DefaultEsLookupResultOutput.java|  62 +
 .../elasticsearch/ElasticsearchGetRequest.java  |  36 -
 .../elasticsearch/EsLookupResultOutput.java |   5 +-
 .../elasticsearch/bolt/AbstractEsBolt.java  |  53 +---
 .../storm/elasticsearch/bolt/EsIndexBolt.java   |  31 +++--
 .../storm/elasticsearch/bolt/EsLookupBolt.java  |  49 ---
 .../elasticsearch/bolt/EsPercolateBolt.java |  41 --
 .../common/DefaultEsTupleMapper.java|  20 +++
 .../storm/elasticsearch/common/EsConfig.java| 130 +--
 .../elasticsearch/common/EsTupleMapper.java |   9 ++
 .../common/StormElasticSearchClient.java|  37 +++---
 .../common/TransportAddresses.java  |  72 --
 .../apache/storm/elasticsearch/doc/Index.java   |  69 ++
 .../storm/elasticsearch/doc/IndexDoc.java   |  43 ++
 .../storm/elasticsearch/doc/IndexItem.java  |  91 +
 .../storm/elasticsearch/doc/IndexItemDoc.java   |  42 ++
 .../apache/storm/elasticsearch/doc/Shards.java  |  63 +
 .../storm/elasticsearch/doc/SourceDoc.java  |  43 ++
 .../response/BulkIndexResponse.java |  80 
 .../elasticsearch/response/LookupResponse.java  |  63 +
 .../response/PercolateResponse.java |  85 
 .../storm/elasticsearch/trident/EsState.java|  72 ++
 .../elasticsearch/trident/EsStateFactory.java   |  15 +--
 .../bolt/AbstractEsBoltIntegrationTest.java |  68 +++---
 .../elasticsearch/bolt/AbstractEsBoltTest.java  |  15 ++-
 .../elasticsearch/bolt/EsIndexBoltTest.java |  44 ---
 .../bolt/EsLookupBoltIntegrationTest.java   |  75 +++
 .../elasticsearch/bolt/EsLookupBoltTest.java|  67 +-
 .../elasticsearch/bolt/EsPercolateBoltTest.java |  62 ++---
 .../elasticsearch/common/EsConfigTest.java  |  60 -
 .../storm/elasticsearch/common/EsTestUtil.java  |  99 --
 .../common/TransportAddressesTest.java  |  81 
 .../trident/EsStateFactoryTest.java |   2 +-
 .../elasticsearch/trident/EsStateTest.java  |  98 ++
 .../src/test/resources/log4j2.xml   |  33 +
 40 files changed, 1424 insertions(+), 565 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/2b4565c6/examples/storm-elasticsearch-examples/pom.xml
--
diff --git a/examples/storm-elasticsearch-examples/pom.xml 
b/examples/storm-elasticsearch-examples/pom.xml
index 64c8229..23a717d 100644
--- a/examples/storm-elasticsearch-examples/pom.xml
+++ b/examples/storm-elasticsearch-examples/pom.xml
@@ -26,6 +26,10 @@
 
 
 storm-elasticsearch-examples
+
+2.4.4
+
+
 
 
 org.apache.storm
@@ -38,6 +42,12 @@
 storm-elasticsearch
 ${project.version}
 
+
+org.elasticsearch
+elasticsearch
+${elasticsearch.test.version}
+${provided.scope}
+
 
 
 

http://git-wip-us.apache.org/repos/asf/storm/blob/2b4565c6/examples/storm-elasticsearch-examples/src/main/java/org/apache/storm/elasticsearch/bolt/EsIndexTopology.java
--
diff --git 
a/examples/storm-elasticsearch-examples/src/main/java/org/apache/storm/elasticsearch/bolt/EsIndexTopology.java
 
b/examples/storm-elasticsearch-examples/src/main/java/org/apache/storm/elasticsearch/bolt/EsIndexTopology.java
index 3cd2bc8..c7ec7d0 100644
--- 
a/examples/storm-elasticsearch-examples/src/main/java/org/apache/storm/elasticsearch/bolt/EsIndexTopology.java
+++ 
b/examples/storm-elasticsearch-examples/src/main/java/org/apache/storm/elasticsearch/bolt/EsIndexTopology.java
@@ -24,7 +24,6 @@ import java.util.concurrent.ConcurrentHashMap;
 import org.apache.storm.Config;
 import

[1/4] storm git commit: STORM-2379: update for Elasticsearch 2.

2017-04-18 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master c38d7950a -> f794d7230


http://git-wip-us.apache.org/repos/asf/storm/blob/2b4565c6/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsIndexBoltTest.java
--
diff --git 
a/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsIndexBoltTest.java
 
b/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsIndexBoltTest.java
index 9f7592d..ebed80e 100644
--- 
a/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsIndexBoltTest.java
+++ 
b/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsIndexBoltTest.java
@@ -17,27 +17,43 @@
  */
 package org.apache.storm.elasticsearch.bolt;
 
-import org.apache.storm.testing.IntegrationTest;
-import org.apache.storm.tuple.Tuple;
+import static org.mockito.Mockito.verify;
+
 import org.apache.storm.elasticsearch.common.EsConfig;
 import org.apache.storm.elasticsearch.common.EsTestUtil;
-import org.apache.storm.elasticsearch.common.EsTupleMapper;
-import org.elasticsearch.action.count.CountResponse;
+import org.apache.storm.testing.IntegrationTest;
+import org.apache.storm.tuple.Tuple;
+import org.elasticsearch.action.search.SearchResponse;
 import org.elasticsearch.index.query.TermQueryBuilder;
 import org.junit.Assert;
 import org.junit.Test;
 import org.junit.experimental.categories.Category;
 
-import static org.mockito.Mockito.verify;
-
 @Category(IntegrationTest.class)
 public class EsIndexBoltTest extends 
AbstractEsBoltIntegrationTest {
 
 @Test
 public void testEsIndexBolt()
 throws Exception {
-String index = "index1";
-String type = "type1";
+Tuple tuple = createTestTuple(index, type);
+
+bolt.execute(tuple);
+
+verify(outputCollector).ack(tuple);
+
+
node.client().admin().indices().prepareRefresh(index).execute().actionGet();
+SearchResponse resp = node.client().prepareSearch(index)
+.setQuery(new TermQueryBuilder("_type", type))
+.setSize(0)
+.execute().actionGet();
+
+Assert.assertEquals(1, resp.getHits().getTotalHits());
+}
+
+@Test
+public void indexMissing()
+throws Exception {
+String index = "missing";
 
 Tuple tuple = createTestTuple(index, type);
 
@@ -46,23 +62,21 @@ public class EsIndexBoltTest extends 
AbstractEsBoltIntegrationTest
 verify(outputCollector).ack(tuple);
 
 
node.client().admin().indices().prepareRefresh(index).execute().actionGet();
-CountResponse resp = node.client().prepareCount(index)
+SearchResponse resp = node.client().prepareSearch(index)
 .setQuery(new TermQueryBuilder("_type", type))
+.setSize(0)
 .execute().actionGet();
 
-Assert.assertEquals(1, resp.getCount());
+Assert.assertEquals(1, resp.getHits().getTotalHits());
 }
 
 private Tuple createTestTuple(String index, String type) {
-String source = "{\"user\":\"user1\"}";
-String id = "docId";
-return EsTestUtil.generateTestTuple(source, index, type, id);
+return EsTestUtil.generateTestTuple(source, index, type, documentId);
 }
 
 @Override
 protected EsIndexBolt createBolt(EsConfig esConfig) {
-EsTupleMapper tupleMapper = EsTestUtil.generateDefaultTupleMapper();
-return new EsIndexBolt(esConfig, tupleMapper);
+return new EsIndexBolt(esConfig);
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/storm/blob/2b4565c6/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsLookupBoltIntegrationTest.java
--
diff --git 
a/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsLookupBoltIntegrationTest.java
 
b/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsLookupBoltIntegrationTest.java
index 038dcce..c266069 100644
--- 
a/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsLookupBoltIntegrationTest.java
+++ 
b/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/bolt/EsLookupBoltIntegrationTest.java
@@ -17,68 +17,46 @@
  */
 package org.apache.storm.elasticsearch.bolt;
 
+import static org.hamcrest.CoreMatchers.is;
+import static org.junit.Assert.assertThat;
+import static org.mockito.Matchers.any;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+
+import org.apache.storm.elasticsearch.common.EsConfig;
+import org.apache.storm.elasticsearch.common.EsTestUtil;
 import org.apache.storm.testing.IntegrationTest;
-import org.apache.storm.tuple.Fields;
-import org.apache.storm.tuple.ITuple;
 import org.apache.storm.tuple.Tuple;
 import

[3/4] storm git commit: Merge branch 'STORM-2379' of https://github.com/hmcc/storm

2017-04-18 Thread sriharsha

Merge branch 'STORM-2379' of https://github.com/hmcc/storm


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/55297b01
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/55297b01
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/55297b01

Branch: refs/heads/master
Commit: 55297b01d67981572f4705b210a904e560e5ac6d
Parents: c38d795 2b4565c
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Apr 18 10:50:15 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Apr 18 10:50:15 2017 -0700

--
 examples/storm-elasticsearch-examples/pom.xml   |  10 ++
 .../elasticsearch/bolt/EsIndexTopology.java |   3 +-
 .../storm/elasticsearch/common/EsTestUtil.java  |  14 +-
 .../trident/TridentEsTopology.java  |   3 +-
 external/storm-elasticsearch/pom.xml|  44 ++-
 .../DefaultEsLookupResultOutput.java|  62 +
 .../elasticsearch/ElasticsearchGetRequest.java  |  36 -
 .../elasticsearch/EsLookupResultOutput.java |   5 +-
 .../elasticsearch/bolt/AbstractEsBolt.java  |  53 +---
 .../storm/elasticsearch/bolt/EsIndexBolt.java   |  31 +++--
 .../storm/elasticsearch/bolt/EsLookupBolt.java  |  49 ---
 .../elasticsearch/bolt/EsPercolateBolt.java |  41 --
 .../common/DefaultEsTupleMapper.java|  20 +++
 .../storm/elasticsearch/common/EsConfig.java| 130 +--
 .../elasticsearch/common/EsTupleMapper.java |   9 ++
 .../common/StormElasticSearchClient.java|  37 +++---
 .../common/TransportAddresses.java  |  72 --
 .../apache/storm/elasticsearch/doc/Index.java   |  69 ++
 .../storm/elasticsearch/doc/IndexDoc.java   |  43 ++
 .../storm/elasticsearch/doc/IndexItem.java  |  91 +
 .../storm/elasticsearch/doc/IndexItemDoc.java   |  42 ++
 .../apache/storm/elasticsearch/doc/Shards.java  |  63 +
 .../storm/elasticsearch/doc/SourceDoc.java  |  43 ++
 .../response/BulkIndexResponse.java |  80 
 .../elasticsearch/response/LookupResponse.java  |  63 +
 .../response/PercolateResponse.java |  85 
 .../storm/elasticsearch/trident/EsState.java|  72 ++
 .../elasticsearch/trident/EsStateFactory.java   |  15 +--
 .../bolt/AbstractEsBoltIntegrationTest.java |  68 +++---
 .../elasticsearch/bolt/AbstractEsBoltTest.java  |  15 ++-
 .../elasticsearch/bolt/EsIndexBoltTest.java |  44 ---
 .../bolt/EsLookupBoltIntegrationTest.java   |  75 +++
 .../elasticsearch/bolt/EsLookupBoltTest.java|  67 +-
 .../elasticsearch/bolt/EsPercolateBoltTest.java |  62 ++---
 .../elasticsearch/common/EsConfigTest.java  |  60 -
 .../storm/elasticsearch/common/EsTestUtil.java  |  99 --
 .../common/TransportAddressesTest.java  |  81 
 .../trident/EsStateFactoryTest.java |   2 +-
 .../elasticsearch/trident/EsStateTest.java  |  98 ++
 .../src/test/resources/log4j2.xml   |  33 +
 40 files changed, 1424 insertions(+), 565 deletions(-)
--

[2/3] storm git commit: Merge branch 'Apache_master_STORM-2407_KafkaTridentMissesData' of https://github.com/hmcl/storm-apache

2017-03-20 Thread sriharsha

Merge branch 'Apache_master_STORM-2407_KafkaTridentMissesData' of 
https://github.com/hmcl/storm-apache


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/c2c9559f
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/c2c9559f
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/c2c9559f

Branch: refs/heads/master
Commit: c2c9559f0cac2049a83b026209610bd99dcf6a8e
Parents: f5a362a cd3b930
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Mon Mar 20 10:27:13 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Mon Mar 20 10:27:13 2017 -0700

--
 .../trident/OpaqueTridentEventHubEmitter.java   |  7 +-
 .../spout/trident/KafkaTridentSpoutEmitter.java | 94 +++-
 .../spout/trident/KafkaTridentSpoutOpaque.java  |  8 +-
 .../kafka/trident/TridentKafkaEmitter.java  |  7 +-
 .../spout/IOpaquePartitionedTridentSpout.java   | 29 +-
 .../OpaquePartitionedTridentSpoutExecutor.java  | 15 ++--
 .../topology/state/TransactionalState.java  |  4 +
 7 files changed, 74 insertions(+), 90 deletions(-)
--

[1/3] storm git commit: STORM-2407: KafkaTridentSpoutOpaque Doesn't Poll Data From All Topic-Partitions When Parallelism Hint Not a Multiple Total Topic-Partitions - Introduce logic to poll data from

2017-03-20 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master f5a362a6e -> 17783d5e0


STORM-2407: KafkaTridentSpoutOpaque Doesn't Poll Data From All Topic-Partitions 
When Parallelism Hint Not a Multiple Total Topic-Partitions
 - Introduce logic to poll data from the topic partitions assigned to each  task


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/cd3b9309
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/cd3b9309
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/cd3b9309

Branch: refs/heads/master
Commit: cd3b93093b3a3ef85e9acffc4cd350ec1e3640de
Parents: 943bef6
Author: Hugo Louro 
Authored: Fri Mar 10 13:13:31 2017 -0800
Committer: Hugo Louro 
Committed: Wed Mar 15 20:52:16 2017 -0700

--
 .../trident/OpaqueTridentEventHubEmitter.java   |  7 +-
 .../spout/trident/KafkaTridentSpoutEmitter.java | 94 +++-
 .../spout/trident/KafkaTridentSpoutOpaque.java  |  8 +-
 .../kafka/trident/TridentKafkaEmitter.java  |  7 +-
 .../spout/IOpaquePartitionedTridentSpout.java   | 29 +-
 .../OpaquePartitionedTridentSpoutExecutor.java  | 15 ++--
 .../topology/state/TransactionalState.java  |  4 +
 7 files changed, 74 insertions(+), 90 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/cd3b9309/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/trident/OpaqueTridentEventHubEmitter.java
--
diff --git 
a/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/trident/OpaqueTridentEventHubEmitter.java
 
b/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/trident/OpaqueTridentEventHubEmitter.java
index ae21ab3..5b6b642 100755
--- 
a/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/trident/OpaqueTridentEventHubEmitter.java
+++ 
b/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/trident/OpaqueTridentEventHubEmitter.java
@@ -17,16 +17,15 @@
  
***/
 package org.apache.storm.eventhubs.trident;
 
-import java.util.List;
-import java.util.Map;
-
 import org.apache.storm.eventhubs.spout.EventHubSpoutConfig;
 import org.apache.storm.eventhubs.spout.IEventHubReceiverFactory;
-
 import org.apache.storm.trident.operation.TridentCollector;
 import org.apache.storm.trident.spout.IOpaquePartitionedTridentSpout;
 import org.apache.storm.trident.topology.TransactionAttempt;
 
+import java.util.List;
+import java.util.Map;
+
 /**
  * A thin wrapper of TransactionalTridentEventHubEmitter for 
OpaqueTridentEventHubSpout
  */

http://git-wip-us.apache.org/repos/asf/storm/blob/cd3b9309/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
--
diff --git 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
index 79dfc60..8607853 100644
--- 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
+++ 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
@@ -35,7 +35,6 @@ import org.slf4j.LoggerFactory;
 
 import java.io.Serializable;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.HashSet;
@@ -71,7 +70,7 @@ public class KafkaTridentSpoutEmitter implements 
IOpaquePartitionedTrident
 
 private TopologyContext topologyContext;
 
-public KafkaTridentSpoutEmitter(KafkaTridentSpoutManager 
kafkaManager, TopologyContext topologyContext, Timer refreshSubscriptionTimer) {
+public KafkaTridentSpoutEmitter(KafkaTridentSpoutManager 
kafkaManager, TopologyContext topologyContext, Timer refreshSubscriptionTimer) {
 this.kafkaConsumer = 
kafkaManager.createAndSubscribeKafkaConsumer(topologyContext);
 this.kafkaManager = kafkaManager;
 this.topologyContext = topologyContext;
@@ -87,14 +86,14 @@ public class KafkaTridentSpoutEmitter implements 
IOpaquePartitionedTrident
 /**
  * Creates instance of this class with default 500 millisecond refresh 
subscription timer
  */
-public KafkaTridentSpoutEmitter(KafkaTridentSpoutManager 
kafkaManager, TopologyContext topologyContext) {
+public KafkaTridentSpoutEmitter(KafkaTridentSpoutManager 
kafkaManager, TopologyContext topologyContext) {
 this(kafkaManager, topologyContext, new Timer(500,

[3/3] storm git commit: STORM-2407: CHANGELOG.

2017-03-20 Thread sriharsha

STORM-2407: CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/17783d5e
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/17783d5e
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/17783d5e

Branch: refs/heads/master
Commit: 17783d5e061b887b87a03c18a0485e6d0f6f2baa
Parents: c2c9559
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Mon Mar 20 10:57:29 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Mon Mar 20 10:57:29 2017 -0700

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/17783d5e/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0d4b553..c40a18e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -210,6 +210,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.0
+ * STORM-2407: KafkaTridentSpoutOpaque Doesn't Poll Data From All 
Topic-Partitions When Parallelism Hint Not a Multiple Total Topic-Partitions.
  * STORM-2414: Skip checking meta's ACL when subject has write privileges for 
any blobs
  * STORM-2038: Disable symlinks with a config option
  * STORM-2408: build failed if storm.kafka.client.version = 0.10.2.0

[1/2] storm git commit: STORM-2407: KafkaTridentSpoutOpaque Doesn't Poll Data From All Topic-Partitions When Parallelism Hint Not a Multiple Total Topic-Partitions - Introduce logic to poll data from

2017-03-15 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 1deb9ab95 -> 262029b15


STORM-2407: KafkaTridentSpoutOpaque Doesn't Poll Data From All Topic-Partitions 
When Parallelism Hint Not a Multiple Total Topic-Partitions
 - Introduce logic to poll data from the topic partitions assigned to each  task


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/b8885411
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/b8885411
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/b8885411

Branch: refs/heads/1.x-branch
Commit: b88854110099edb778d5a906ff1f838737b673a3
Parents: 2a99f61
Author: Hugo Louro 
Authored: Fri Mar 10 15:13:31 2017 -0600
Committer: Hugo Louro 
Committed: Fri Mar 10 17:55:02 2017 -0600

--
 .../trident/OpaqueTridentEventHubEmitter.java   | 20 -
 .../spout/trident/KafkaTridentSpoutEmitter.java | 94 +++-
 .../spout/trident/KafkaTridentSpoutOpaque.java  |  8 +-
 .../kafka/trident/TridentKafkaEmitter.java  | 19 ++--
 .../spout/IOpaquePartitionedTridentSpout.java   | 19 +++-
 .../OpaquePartitionedTridentSpoutExecutor.java  | 15 ++--
 .../topology/state/TransactionalState.java  |  4 +
 7 files changed, 89 insertions(+), 90 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/b8885411/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/trident/OpaqueTridentEventHubEmitter.java
--
diff --git 
a/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/trident/OpaqueTridentEventHubEmitter.java
 
b/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/trident/OpaqueTridentEventHubEmitter.java
index ae21ab3..20375a2 100755
--- 
a/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/trident/OpaqueTridentEventHubEmitter.java
+++ 
b/external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/trident/OpaqueTridentEventHubEmitter.java
@@ -17,16 +17,16 @@
  
***/
 package org.apache.storm.eventhubs.trident;
 
-import java.util.List;
-import java.util.Map;
-
 import org.apache.storm.eventhubs.spout.EventHubSpoutConfig;
 import org.apache.storm.eventhubs.spout.IEventHubReceiverFactory;
-
 import org.apache.storm.trident.operation.TridentCollector;
 import org.apache.storm.trident.spout.IOpaquePartitionedTridentSpout;
 import org.apache.storm.trident.topology.TransactionAttempt;
 
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
 /**
  * A thin wrapper of TransactionalTridentEventHubEmitter for 
OpaqueTridentEventHubSpout
  */
@@ -63,6 +63,18 @@ public class OpaqueTridentEventHubEmitter implements 
IOpaquePartitionedTridentSp
   }
 
   @Override
+  public List getPartitionsForTask(int taskId, int numTasks, 
Partitions allPartitionInfo) {
+final List orderedPartitions = 
getOrderedPartitions(allPartitionInfo);
+final List taskPartitions = new ArrayList<>(orderedPartitions 
== null ? 0 : orderedPartitions.size());
+if (orderedPartitions != null) {
+  for (int i = taskId; i < orderedPartitions.size(); i += numTasks) {
+taskPartitions.add(orderedPartitions.get(i));
+  }
+}
+return taskPartitions;
+  }
+
+  @Override
   public void refreshPartitions(List partitionList) {
 transactionalEmitter.refreshPartitions(partitionList);
   }

http://git-wip-us.apache.org/repos/asf/storm/blob/b8885411/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
--
diff --git 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
index 79dfc60..8607853 100644
--- 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
+++ 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
@@ -35,7 +35,6 @@ import org.slf4j.LoggerFactory;
 
 import java.io.Serializable;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.HashSet;
@@ -71,7 +70,7 @@ public class KafkaTridentSpoutEmitter implements 
IOpaquePartitionedTrident
 
 private TopologyContext topologyContext;
 
-public KafkaTridentSpoutEmitter(KafkaTridentSpoutManager 
kafkaManager, TopologyContext topologyContext, Timer refreshSubscriptionTimer) {
+public KafkaTridentSpoutEmitter(KafkaTridentSpoutManager 
kafkaManager,

[2/2] storm git commit: Merge branch '1.x-branch_STORM-2407_KafkaTridentMissesData' of https://github.com/hmcl/storm-apache into STORM-2407

2017-03-15 Thread sriharsha

Merge branch '1.x-branch_STORM-2407_KafkaTridentMissesData' of 
https://github.com/hmcl/storm-apache into STORM-2407


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/262029b1
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/262029b1
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/262029b1

Branch: refs/heads/1.x-branch
Commit: 262029b15f558969ffcc90658f43238b0d1dc31f
Parents: 1deb9ab b888541
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Wed Mar 15 12:45:28 2017 -0700
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Wed Mar 15 12:45:28 2017 -0700

--
 .../trident/OpaqueTridentEventHubEmitter.java   | 20 -
 .../spout/trident/KafkaTridentSpoutEmitter.java | 94 +++-
 .../spout/trident/KafkaTridentSpoutOpaque.java  |  8 +-
 .../kafka/trident/TridentKafkaEmitter.java  | 19 ++--
 .../spout/IOpaquePartitionedTridentSpout.java   | 19 +++-
 .../OpaquePartitionedTridentSpoutExecutor.java  | 15 ++--
 .../topology/state/TransactionalState.java  |  4 +
 7 files changed, 89 insertions(+), 90 deletions(-)
--

[1/4] storm git commit: Merge branch 'STORM-1772' of https://github.com/roshannaik/storm into STORM-1772

2017-03-07 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master fc69a1cb4 -> 4fcd37829


http://git-wip-us.apache.org/repos/asf/storm/blob/5c59e063/storm-perf/src/main/sampledata/randomwords.txt
--
diff --git a/storm-perf/src/main/sampledata/randomwords.txt 
b/storm-perf/src/main/sampledata/randomwords.txt
new file mode 100644
index 000..1b800cf
--- /dev/null
+++ b/storm-perf/src/main/sampledata/randomwords.txt
@@ -0,0 +1,14049 @@
+Hester starosta Bassaris guanajuatite pinulus debromination
+outhue pamphlet Socraticism tomorrowness masa Bertat Vichyite  
+epauliere magnificently Mormyrus circumzenithal sapphiric Homoiousian 
misthread
+Pithecolobium saguran hemimelus epauliere symbiogenetically
+chronist cervisial euphemious topline sandbox pneumonalgia Babylonism sawdust 
by   
+figured impressor theologicopolitical scrat arsenide cockal roughcast 
introducer playfellowship
+potentness Tsonecan ovopyriform inertly Bertat acocotl whitlowwort 
+Gothish Pishquow corona interruptedness glossing molecule preagitate   
+seelful unscourged swangy Mycogone bogydom nummi homotransplant unburnt
+guitarist cockal Coniferae cylindric ovopyriform nonmanufacture monilioid 
hypochondriacism 
+underogating impairment genii Ludgatian palaeotheriodont marten
+euphonym zoonitic cervisial Bertat guanajuatite Vaishnavism nonsuppressed 
subsequentially  
+scotale semiangle epididymitis bestill nonutilitarian lifter pyrocatechol 
obolus   
+scrat unpremonished ungrave undeterring winterproof zanyism Itea tonsure   
+antideflation rivethead seeingness snare refasten repealableness monstership 
bacillite 
+biodynamics antiadiaphorist strander undangered Hester 
+overwoven fallacious yawler seelful helminthagogic 
+quad putative whittle placatory vinny nebular sialadenitis 
+epauliere elemicin greave relaster pyroacetic nectopod nonexecutive weism 
dunkadoo 
+sarcologist tetragynian cubit frictionlessly magnetooptics undangered 
glyphography vinny   
+bladderwort unobservantness unswanlike terrestrially botchedly ultraobscure
+papery mesymnion embryotic Bulanda pumpkinification Bassaris cromlech ungrave  
+Semecarpus commotion familist uncontradictableness manny vinny 
+eurythermal enation oratorship cretaceous hypoid   
+biopsic yote ineunt subangulated mutter barkometer 
+manny signifier brag heavenful Homoiousian undeterring michigan
+ultratense Protestantize angina zenick swearingly unexplicit   
+monstership unrepealably shibuichi enhedge pachydermous bozal rechar 
lineamental   
+thermochemically nonutilitarian uninterpleaded Florissant ineunt jharal 
aneurism stachyuraceous
+neuromimesis reperuse magnificently scyphostoma manganosiderite
+merciful becomma unrealize Vaishnavism superindifference trillium soorkee 
rechar   
+lienteria Orbitolina Protestantize Gothish molossic nonsuppressed 
nonutilitarian   
+cyanoguanidine feasibleness unstipulated thermoresistant slait Inger 
characinoid   
+transudatory seizing ungreat almud monander abstractionism socioromantic   
+sequestrum sequestrum widdle angina bacillite  
+tum beatable seraphism bozal cattimandoo skyshine sombreroed emir  
+circular strander diplomatize veterinarian inertly prolificy prepavement   
+taver sequestrum corbel classificational mangonism strander swacking   
+unrevolting Sebastian spiranthic hepatorrhaphy oflete subangulated 
subtransverse   
+unharmed myesthesia trophonema bozal immatchable   
+uninhabitedness eristically Eryon allegedly daytime periclitation phallaceous  
+phytonic yote Serrifera nonmanufacture Animalivora 
+bozal gunshop ornithodelphous Tamil chorograph Quakerishly outwealth laryngic 
hepatorrhaphy
+pinulus uninhabitedness endotheliomyoma unachievable Confervales ladhood 
chilblain 
+unachievable nonmanufacture rave Prosobranchiata dermorhynchous minniebush 
oinomancy overinstruct ventricous   
+precostal ununiformly mastication figured idiotize antihero
+Kenipsim beneficent blurredness Dawsonia twinling phallaceous impugnation 
dermorhynchous   
+packsack zanyism breadwinner Megaluridae uncarefully zenick dishpan squdge 
+dialoguer theologicopolitical subfoliar mangonism mustafina champer
+Consolamentum exploiter imperceptivity nectopod breadwinner floatability   
+heavenful theologicopolitical pyrocatechol ferrogoslarite infestation 
dermorhynchous lampyrine 
+shallowish crystallographical uninhabitedness cocksuredom yawler hysterolysis 
genii bromic 
+antalgol chronographic chasmy proboscidiform craglike stiffish 
+marshiness Scanic adz uninterpleaded trailmaking stormy impairment Dunlop 
pinulus  
+mechanist helminthagogic semantician ununiformly incomprehensible equiconvex 
adatom transcorporeal sertularian 
+migrainoid bacillite

[4/4] storm git commit: Added STORM-1772 to CHANGELOG.

2017-03-07 Thread sriharsha

Added STORM-1772 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/4fcd3782
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/4fcd3782
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/4fcd3782

Branch: refs/heads/master
Commit: 4fcd37829a63feda5a8e42b96239cf05c49bf08d
Parents: 5c59e06
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Mar 7 19:56:37 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Mar 7 19:56:37 2017 -0800

--
 CHANGELOG.md   | 1 +
 storm-perf/pom.xml | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/4fcd3782/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index babd3a6..2122b38 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -205,6 +205,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.0
+ * STORM-1772: Adding Perf module with topologies for measuring performance
  * STORM-2391: Move HdfsSpoutTopology from storm-starter to storm-hdfs-examples
  * STORM-2389: Avoid instantiating Event Logger when 
topology.eventlogger.executors=0
  * STORM-2386: Fail-back Blob deletion also fails in 
BlobSynchronizer.syncBlobs.

http://git-wip-us.apache.org/repos/asf/storm/blob/4fcd3782/storm-perf/pom.xml
--
diff --git a/storm-perf/pom.xml b/storm-perf/pom.xml
index ab54bb7..626495c 100644
--- a/storm-perf/pom.xml
+++ b/storm-perf/pom.xml
@@ -22,7 +22,7 @@
 
 storm
 org.apache.storm
-1.1.0-SNAPSHOT
+2.0.0-SNAPSHOT
 ..
 
 
@@ -119,4 +119,4 @@
 
 
 
-
\ No newline at end of file
+

[2/4] storm git commit: Merge branch 'STORM-1772' of https://github.com/roshannaik/storm into STORM-1772

2017-03-07 Thread sriharsha

http://git-wip-us.apache.org/repos/asf/storm/blob/5c59e063/storm-perf/src/main/java/org/apache/storm/perf/utils/BasicMetricsCollector.java
--
diff --git 
a/storm-perf/src/main/java/org/apache/storm/perf/utils/BasicMetricsCollector.java
 
b/storm-perf/src/main/java/org/apache/storm/perf/utils/BasicMetricsCollector.java
new file mode 100755
index 000..686f9da
--- /dev/null
+++ 
b/storm-perf/src/main/java/org/apache/storm/perf/utils/BasicMetricsCollector.java
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.storm.perf.utils;
+
+import org.apache.storm.LocalCluster;
+import org.apache.storm.generated.Nimbus;
+import org.apache.storm.utils.NimbusClient;
+import org.apache.storm.utils.Utils;
+import org.apache.log4j.Logger;
+
+import java.io.PrintWriter;
+import java.util.*;
+
+
+public class BasicMetricsCollector  {
+
+private LocalCluster localCluster = null;
+private Nimbus.Client client = null;
+private PrintWriter dataWriter;
+private long startTime=0;
+
+public enum MetricsItem {
+TOPOLOGY_STATS,
+XSFER_RATE,
+SPOUT_THROUGHPUT,
+SPOUT_LATENCY,
+ALL
+}
+
+
+/* headers */
+public static final String TIME = "elapsed (sec)";
+public static final String TIME_FORMAT = "%d";
+public static final String TOTAL_SLOTS = "total_slots";
+public static final String USED_SLOTS = "used_slots";
+public static final String WORKERS = "workers";
+public static final String TASKS = "tasks";
+public static final String EXECUTORS = "executors";
+public static final String TRANSFERRED = "transferred (messages)";
+public static final String XSFER_RATE = "transfer rate (messages/s)";
+public static final String SPOUT_EXECUTORS = "spout_executors";
+public static final String SPOUT_TRANSFERRED = "spout_transferred 
(messages)";
+public static final String SPOUT_ACKED = "spout_acks";
+public static final String SPOUT_THROUGHPUT = "spout_throughput (acks/s)";
+public static final String SPOUT_AVG_COMPLETE_LATENCY = 
"spout_avg_complete_latency(ms)";
+public static final String SPOUT_AVG_LATENCY_FORMAT = "%.1f";
+public static final String SPOUT_MAX_COMPLETE_LATENCY = 
"spout_max_complete_latency(ms)";
+public static final String SPOUT_MAX_LATENCY_FORMAT = "%.1f";
+private static final Logger LOG = 
Logger.getLogger(BasicMetricsCollector.class);
+final MetricsCollectorConfig config;
+//final StormTopology topology;
+final Set header = new LinkedHashSet();
+final Map metrics = new HashMap();
+int lineNumber = 0;
+
+final boolean collectTopologyStats;
+final boolean collectExecutorStats;
+final boolean collectThroughput;
+
+final boolean collectSpoutThroughput;
+final boolean collectSpoutLatency;
+
+private MetricsSample lastSample;
+private MetricsSample curSample;
+private double maxLatency = 0;
+
+boolean first = true;
+
+public BasicMetricsCollector(Nimbus.Client client, String topoName, Map 
stormConfig) {
+this(topoName, stormConfig);
+this.client = client;
+this.localCluster = null;
+}
+
+public BasicMetricsCollector(LocalCluster localCluster, String topoName, 
Map stormConfig) {
+this(topoName, stormConfig);
+this.client = null;
+this.localCluster = localCluster;
+}
+
+private BasicMetricsCollector(String topoName, Map stormConfig) {
+Set items = getMetricsToCollect();
+this.config = new MetricsCollectorConfig(topoName, stormConfig);
+collectTopologyStats = collectTopologyStats(items);
+collectExecutorStats = collectExecutorStats(items);
+collectThroughput = collectThroughput(items);
+collectSpoutThroughput = collectSpoutThroughput(items);
+collectSpoutLatency = collectSpoutLatency(items);
+dataWriter = new PrintWriter(System.err);
+}
+
+
+private Set  getMetricsToCollect() {
+Set result = new HashSet<>();
+result.add(MetricsItem.ALL);
+return result;
+}
+
+public

[3/4] storm git commit: Merge branch 'STORM-1772' of https://github.com/roshannaik/storm into STORM-1772

2017-03-07 Thread sriharsha

Merge branch 'STORM-1772' of https://github.com/roshannaik/storm into STORM-1772


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/5c59e063
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/5c59e063
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/5c59e063

Branch: refs/heads/master
Commit: 5c59e0632108847a0ba7dacf43eff9ac477b00d7
Parents: fc69a1c
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Mar 7 13:03:00 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Mar 7 19:56:08 2017 -0800

--
 pom.xml | 3 +
 .../final-package/src/main/assembly/binary.xml  |29 +
 storm-perf/README.markdown  |50 +
 storm-perf/pom.xml  |   122 +
 .../main/conf/ConstSpoutIdBoltNullBoltTopo.yaml |22 +
 .../src/main/conf/ConstSpoutNullBoltTopo.yaml   |22 +
 .../src/main/conf/FileReadWordCountTopo.yaml|23 +
 .../src/main/conf/HdfsSpoutNullBoltTopo.yaml|25 +
 storm-perf/src/main/conf/KafkaHdfsTopo.yaml |26 +
 .../src/main/conf/KafkaSpoutNullBoltTopo.yaml   |23 +
 .../src/main/conf/StrGenSpoutHdfsBoltTopo.yaml  |25 +
 .../perf/ConstSpoutIdBoltNullBoltTopo.java  |   101 +
 .../storm/perf/ConstSpoutNullBoltTopo.java  |   107 +
 .../apache/storm/perf/ConstSpoutOnlyTopo.java   |74 +
 .../storm/perf/FileReadWordCountTopo.java   |96 +
 .../storm/perf/HdfsSpoutNullBoltTopo.java   |   101 +
 .../org/apache/storm/perf/KafkaHdfsTopo.java|   168 +
 .../storm/perf/KafkaSpoutNullBoltTopo.java  |   114 +
 .../storm/perf/StrGenSpoutHdfsBoltTopo.java |   154 +
 .../org/apache/storm/perf/bolt/CountBolt.java   |58 +
 .../org/apache/storm/perf/bolt/DevNullBolt.java |47 +
 .../java/org/apache/storm/perf/bolt/IdBolt.java |49 +
 .../storm/perf/bolt/SplitSentenceBolt.java  |58 +
 .../org/apache/storm/perf/spout/ConstSpout.java |70 +
 .../apache/storm/perf/spout/FileReadSpout.java  |   141 +
 .../apache/storm/perf/spout/StringGenSpout.java |93 +
 .../storm/perf/utils/BasicMetricsCollector.java |   309 +
 .../org/apache/storm/perf/utils/Helper.java |   133 +
 .../apache/storm/perf/utils/IdentityBolt.java   |51 +
 .../apache/storm/perf/utils/MetricsSample.java  |   248 +
 storm-perf/src/main/sampledata/randomwords.txt  | 14049 +
 31 files changed, 16591 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/5c59e063/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 523c1a0..e5c47f5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -352,6 +352,7 @@
 examples/storm-elasticsearch-examples
 examples/storm-mqtt-examples
 examples/storm-pmml-examples
+storm-perf
 
 
 
@@ -439,6 +440,7 @@
 
**/src/ui/public/js/js-yaml.min.js
 
**/src/main/resources/Audit.50.csv
 
**/src/main/resources/KNIME_PMML_4.1_Examples_single_audit_logreg.xml
+**/src/main/sampledata/**
 
 
 
**/dependency-reduced-pom.xml
@@ -457,6 +459,7 @@
 
 
 
integration-test/config/.vagrant/**
+
 
 
 

http://git-wip-us.apache.org/repos/asf/storm/blob/5c59e063/storm-dist/binary/final-package/src/main/assembly/binary.xml
--
diff --git a/storm-dist/binary/final-package/src/main/assembly/binary.xml 
b/storm-dist/binary/final-package/src/main/assembly/binary.xml
index b58752e..a6aab07 100644
--- a/storm-dist/binary/final-package/src/main/assembly/binary.xml
+++ b/storm-dist/binary/final-package/src/main/assembly/binary.xml
@@ -510,6 +510,35 @@
 
 
 
+
+${project.basedir}/../../storm-perf/target
+perf
+
+storm*jar
+
+
+
+
${project.basedir}/../../storm-perf/src/main/conf
+perf/conf
+
+*yaml
+
+
+
+
${project.basedir}/../../storm-perf/src/main/sampledata
+perf/resources
+
+*
+
+
+
+${project.basedir}/../../storm-perf/
+perf
+
+README.*
+
+
+
 
 
 

http://git-wip-us.apache.org/repos/asf/storm/blob/5c59e063/storm-perf/REA

[2/5] storm git commit: STORM-1772 Adding top level storm-perf module with perf topos

2017-03-07 Thread sriharsha

http://git-wip-us.apache.org/repos/asf/storm/blob/08dd24b2/storm-perf/src/main/java/org/apache/storm/perf/utils/BasicMetricsCollector.java
--
diff --git 
a/storm-perf/src/main/java/org/apache/storm/perf/utils/BasicMetricsCollector.java
 
b/storm-perf/src/main/java/org/apache/storm/perf/utils/BasicMetricsCollector.java
new file mode 100755
index 000..686f9da
--- /dev/null
+++ 
b/storm-perf/src/main/java/org/apache/storm/perf/utils/BasicMetricsCollector.java
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.storm.perf.utils;
+
+import org.apache.storm.LocalCluster;
+import org.apache.storm.generated.Nimbus;
+import org.apache.storm.utils.NimbusClient;
+import org.apache.storm.utils.Utils;
+import org.apache.log4j.Logger;
+
+import java.io.PrintWriter;
+import java.util.*;
+
+
+public class BasicMetricsCollector  {
+
+private LocalCluster localCluster = null;
+private Nimbus.Client client = null;
+private PrintWriter dataWriter;
+private long startTime=0;
+
+public enum MetricsItem {
+TOPOLOGY_STATS,
+XSFER_RATE,
+SPOUT_THROUGHPUT,
+SPOUT_LATENCY,
+ALL
+}
+
+
+/* headers */
+public static final String TIME = "elapsed (sec)";
+public static final String TIME_FORMAT = "%d";
+public static final String TOTAL_SLOTS = "total_slots";
+public static final String USED_SLOTS = "used_slots";
+public static final String WORKERS = "workers";
+public static final String TASKS = "tasks";
+public static final String EXECUTORS = "executors";
+public static final String TRANSFERRED = "transferred (messages)";
+public static final String XSFER_RATE = "transfer rate (messages/s)";
+public static final String SPOUT_EXECUTORS = "spout_executors";
+public static final String SPOUT_TRANSFERRED = "spout_transferred 
(messages)";
+public static final String SPOUT_ACKED = "spout_acks";
+public static final String SPOUT_THROUGHPUT = "spout_throughput (acks/s)";
+public static final String SPOUT_AVG_COMPLETE_LATENCY = 
"spout_avg_complete_latency(ms)";
+public static final String SPOUT_AVG_LATENCY_FORMAT = "%.1f";
+public static final String SPOUT_MAX_COMPLETE_LATENCY = 
"spout_max_complete_latency(ms)";
+public static final String SPOUT_MAX_LATENCY_FORMAT = "%.1f";
+private static final Logger LOG = 
Logger.getLogger(BasicMetricsCollector.class);
+final MetricsCollectorConfig config;
+//final StormTopology topology;
+final Set header = new LinkedHashSet();
+final Map metrics = new HashMap();
+int lineNumber = 0;
+
+final boolean collectTopologyStats;
+final boolean collectExecutorStats;
+final boolean collectThroughput;
+
+final boolean collectSpoutThroughput;
+final boolean collectSpoutLatency;
+
+private MetricsSample lastSample;
+private MetricsSample curSample;
+private double maxLatency = 0;
+
+boolean first = true;
+
+public BasicMetricsCollector(Nimbus.Client client, String topoName, Map 
stormConfig) {
+this(topoName, stormConfig);
+this.client = client;
+this.localCluster = null;
+}
+
+public BasicMetricsCollector(LocalCluster localCluster, String topoName, 
Map stormConfig) {
+this(topoName, stormConfig);
+this.client = null;
+this.localCluster = localCluster;
+}
+
+private BasicMetricsCollector(String topoName, Map stormConfig) {
+Set items = getMetricsToCollect();
+this.config = new MetricsCollectorConfig(topoName, stormConfig);
+collectTopologyStats = collectTopologyStats(items);
+collectExecutorStats = collectExecutorStats(items);
+collectThroughput = collectThroughput(items);
+collectSpoutThroughput = collectSpoutThroughput(items);
+collectSpoutLatency = collectSpoutLatency(items);
+dataWriter = new PrintWriter(System.err);
+}
+
+
+private Set  getMetricsToCollect() {
+Set result = new HashSet<>();
+result.add(MetricsItem.ALL);
+return result;
+}
+
+public

[3/5] storm git commit: STORM-1772 Adding top level storm-perf module with perf topos

2017-03-07 Thread sriharsha

STORM-1772 Adding top level storm-perf module with perf topos

Adding ConstSpoutIDBoltNullBoltTopo. Bug fix in ConstSpoutNullBoltTopo

Adding documentation. Refinements to how perf topos are run. Packaging fixes

Addressing rat checks


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/08dd24b2
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/08dd24b2
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/08dd24b2

Branch: refs/heads/1.x-branch
Commit: 08dd24b2d4d7dda0d59012f602405c0ad9a1cafd
Parents: 6ee5ad0
Author: Roshan Naik 
Authored: Wed Feb 22 22:42:22 2017 -0800
Committer: Roshan Naik 
Committed: Fri Mar 3 17:14:21 2017 -0800

--
 pom.xml | 3 +
 storm-dist/binary/src/main/assembly/binary.xml  |29 +
 storm-perf/README.markdown  |50 +
 storm-perf/pom.xml  |   122 +
 .../main/conf/ConstSpoutIdBoltNullBoltTopo.yaml |22 +
 .../src/main/conf/ConstSpoutNullBoltTopo.yaml   |22 +
 .../src/main/conf/FileReadWordCountTopo.yaml|23 +
 .../src/main/conf/HdfsSpoutNullBoltTopo.yaml|25 +
 storm-perf/src/main/conf/KafkaHdfsTopo.yaml |26 +
 .../src/main/conf/KafkaSpoutNullBoltTopo.yaml   |23 +
 .../src/main/conf/StrGenSpoutHdfsBoltTopo.yaml  |25 +
 .../perf/ConstSpoutIdBoltNullBoltTopo.java  |   101 +
 .../storm/perf/ConstSpoutNullBoltTopo.java  |   107 +
 .../apache/storm/perf/ConstSpoutOnlyTopo.java   |74 +
 .../storm/perf/FileReadWordCountTopo.java   |96 +
 .../storm/perf/HdfsSpoutNullBoltTopo.java   |   101 +
 .../org/apache/storm/perf/KafkaHdfsTopo.java|   168 +
 .../storm/perf/KafkaSpoutNullBoltTopo.java  |   114 +
 .../storm/perf/StrGenSpoutHdfsBoltTopo.java |   154 +
 .../org/apache/storm/perf/bolt/CountBolt.java   |58 +
 .../org/apache/storm/perf/bolt/DevNullBolt.java |47 +
 .../java/org/apache/storm/perf/bolt/IdBolt.java |49 +
 .../storm/perf/bolt/SplitSentenceBolt.java  |58 +
 .../org/apache/storm/perf/spout/ConstSpout.java |70 +
 .../apache/storm/perf/spout/FileReadSpout.java  |   141 +
 .../apache/storm/perf/spout/StringGenSpout.java |93 +
 .../storm/perf/utils/BasicMetricsCollector.java |   309 +
 .../org/apache/storm/perf/utils/Helper.java |   133 +
 .../apache/storm/perf/utils/IdentityBolt.java   |51 +
 .../apache/storm/perf/utils/MetricsSample.java  |   248 +
 storm-perf/src/main/sampledata/randomwords.txt  | 14049 +
 31 files changed, 16591 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/08dd24b2/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 38cbe52..da2b555 100644
--- a/pom.xml
+++ b/pom.xml
@@ -331,6 +331,7 @@
 examples/storm-elasticsearch-examples
 examples/storm-mqtt-examples
 examples/storm-pmml-examples
+storm-perf
 
 
 
@@ -418,6 +419,7 @@
 
**/src/ui/public/js/js-yaml.min.js
 
**/src/main/resources/Audit.50.csv
 
**/src/main/resources/KNIME_PMML_4.1_Examples_single_audit_logreg.xml
+**/src/main/sampledata/**
 
 
 
**/dependency-reduced-pom.xml
@@ -436,6 +438,7 @@
 
 
 
integration-test/config/.vagrant/**
+
 
 
 

http://git-wip-us.apache.org/repos/asf/storm/blob/08dd24b2/storm-dist/binary/src/main/assembly/binary.xml
--
diff --git a/storm-dist/binary/src/main/assembly/binary.xml 
b/storm-dist/binary/src/main/assembly/binary.xml
index 40c2905..8bd62d2 100644
--- a/storm-dist/binary/src/main/assembly/binary.xml
+++ b/storm-dist/binary/src/main/assembly/binary.xml
@@ -503,6 +503,35 @@
 
 
 
+
+${project.basedir}/../../storm-perf/target
+perf
+
+storm*jar
+
+
+
+
${project.basedir}/../../storm-perf/src/main/conf
+perf/conf
+
+*yaml
+
+
+
+
${project.basedir}/../../storm-perf/src/main/sampledata
+perf/resources
+
+*
+
+
+
+${project.basedir}/../../storm-perf/
+perf
+
+README.*
+
+
+

[5/5] storm git commit: Added STORM-1772 to CHANGELOG.

2017-03-07 Thread sriharsha

Added STORM-1772 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/ef3ff329
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/ef3ff329
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/ef3ff329

Branch: refs/heads/1.x-branch
Commit: ef3ff3290a603d57e39f024450e056dfdd1666fa
Parents: dd9f1c8
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Mar 7 13:58:15 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Mar 7 13:58:15 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/ef3ff329/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5b93de1..a02530d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ï»¿## 1.1.0
+ * STORM-1772: Adding Perf module with topologies for measuring performance
  * STORM-2395: storm.cmd supervisor calls the wrong class name
  * STORM-2391: Move HdfsSpoutTopology from storm-starter to storm-hdfs-examples
  * STORM-2389: Avoid instantiating Event Logger when 
topology.eventlogger.executors=0

[4/5] storm git commit: Merge branch 'STORM-1772' of https://github.com/roshannaik/storm into STORM-1772

2017-03-07 Thread sriharsha

Merge branch 'STORM-1772' of https://github.com/roshannaik/storm into STORM-1772


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/dd9f1c8b
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/dd9f1c8b
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/dd9f1c8b

Branch: refs/heads/1.x-branch
Commit: dd9f1c8b8071401f708493469476cc5c7280b604
Parents: 718be34 08dd24b
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Mar 7 13:03:00 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Mar 7 13:03:00 2017 -0800

--
 pom.xml | 3 +
 storm-dist/binary/src/main/assembly/binary.xml  |29 +
 storm-perf/README.markdown  |50 +
 storm-perf/pom.xml  |   122 +
 .../main/conf/ConstSpoutIdBoltNullBoltTopo.yaml |22 +
 .../src/main/conf/ConstSpoutNullBoltTopo.yaml   |22 +
 .../src/main/conf/FileReadWordCountTopo.yaml|23 +
 .../src/main/conf/HdfsSpoutNullBoltTopo.yaml|25 +
 storm-perf/src/main/conf/KafkaHdfsTopo.yaml |26 +
 .../src/main/conf/KafkaSpoutNullBoltTopo.yaml   |23 +
 .../src/main/conf/StrGenSpoutHdfsBoltTopo.yaml  |25 +
 .../perf/ConstSpoutIdBoltNullBoltTopo.java  |   101 +
 .../storm/perf/ConstSpoutNullBoltTopo.java  |   107 +
 .../apache/storm/perf/ConstSpoutOnlyTopo.java   |74 +
 .../storm/perf/FileReadWordCountTopo.java   |96 +
 .../storm/perf/HdfsSpoutNullBoltTopo.java   |   101 +
 .../org/apache/storm/perf/KafkaHdfsTopo.java|   168 +
 .../storm/perf/KafkaSpoutNullBoltTopo.java  |   114 +
 .../storm/perf/StrGenSpoutHdfsBoltTopo.java |   154 +
 .../org/apache/storm/perf/bolt/CountBolt.java   |58 +
 .../org/apache/storm/perf/bolt/DevNullBolt.java |47 +
 .../java/org/apache/storm/perf/bolt/IdBolt.java |49 +
 .../storm/perf/bolt/SplitSentenceBolt.java  |58 +
 .../org/apache/storm/perf/spout/ConstSpout.java |70 +
 .../apache/storm/perf/spout/FileReadSpout.java  |   141 +
 .../apache/storm/perf/spout/StringGenSpout.java |93 +
 .../storm/perf/utils/BasicMetricsCollector.java |   309 +
 .../org/apache/storm/perf/utils/Helper.java |   133 +
 .../apache/storm/perf/utils/IdentityBolt.java   |51 +
 .../apache/storm/perf/utils/MetricsSample.java  |   248 +
 storm-perf/src/main/sampledata/randomwords.txt  | 14049 +
 31 files changed, 16591 insertions(+)
--

[1/5] storm git commit: STORM-1772 Adding top level storm-perf module with perf topos

2017-03-07 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 718be3443 -> ef3ff3290


http://git-wip-us.apache.org/repos/asf/storm/blob/08dd24b2/storm-perf/src/main/sampledata/randomwords.txt
--
diff --git a/storm-perf/src/main/sampledata/randomwords.txt 
b/storm-perf/src/main/sampledata/randomwords.txt
new file mode 100644
index 000..1b800cf
--- /dev/null
+++ b/storm-perf/src/main/sampledata/randomwords.txt
@@ -0,0 +1,14049 @@
+Hester starosta Bassaris guanajuatite pinulus debromination
+outhue pamphlet Socraticism tomorrowness masa Bertat Vichyite  
+epauliere magnificently Mormyrus circumzenithal sapphiric Homoiousian 
misthread
+Pithecolobium saguran hemimelus epauliere symbiogenetically
+chronist cervisial euphemious topline sandbox pneumonalgia Babylonism sawdust 
by   
+figured impressor theologicopolitical scrat arsenide cockal roughcast 
introducer playfellowship
+potentness Tsonecan ovopyriform inertly Bertat acocotl whitlowwort 
+Gothish Pishquow corona interruptedness glossing molecule preagitate   
+seelful unscourged swangy Mycogone bogydom nummi homotransplant unburnt
+guitarist cockal Coniferae cylindric ovopyriform nonmanufacture monilioid 
hypochondriacism 
+underogating impairment genii Ludgatian palaeotheriodont marten
+euphonym zoonitic cervisial Bertat guanajuatite Vaishnavism nonsuppressed 
subsequentially  
+scotale semiangle epididymitis bestill nonutilitarian lifter pyrocatechol 
obolus   
+scrat unpremonished ungrave undeterring winterproof zanyism Itea tonsure   
+antideflation rivethead seeingness snare refasten repealableness monstership 
bacillite 
+biodynamics antiadiaphorist strander undangered Hester 
+overwoven fallacious yawler seelful helminthagogic 
+quad putative whittle placatory vinny nebular sialadenitis 
+epauliere elemicin greave relaster pyroacetic nectopod nonexecutive weism 
dunkadoo 
+sarcologist tetragynian cubit frictionlessly magnetooptics undangered 
glyphography vinny   
+bladderwort unobservantness unswanlike terrestrially botchedly ultraobscure
+papery mesymnion embryotic Bulanda pumpkinification Bassaris cromlech ungrave  
+Semecarpus commotion familist uncontradictableness manny vinny 
+eurythermal enation oratorship cretaceous hypoid   
+biopsic yote ineunt subangulated mutter barkometer 
+manny signifier brag heavenful Homoiousian undeterring michigan
+ultratense Protestantize angina zenick swearingly unexplicit   
+monstership unrepealably shibuichi enhedge pachydermous bozal rechar 
lineamental   
+thermochemically nonutilitarian uninterpleaded Florissant ineunt jharal 
aneurism stachyuraceous
+neuromimesis reperuse magnificently scyphostoma manganosiderite
+merciful becomma unrealize Vaishnavism superindifference trillium soorkee 
rechar   
+lienteria Orbitolina Protestantize Gothish molossic nonsuppressed 
nonutilitarian   
+cyanoguanidine feasibleness unstipulated thermoresistant slait Inger 
characinoid   
+transudatory seizing ungreat almud monander abstractionism socioromantic   
+sequestrum sequestrum widdle angina bacillite  
+tum beatable seraphism bozal cattimandoo skyshine sombreroed emir  
+circular strander diplomatize veterinarian inertly prolificy prepavement   
+taver sequestrum corbel classificational mangonism strander swacking   
+unrevolting Sebastian spiranthic hepatorrhaphy oflete subangulated 
subtransverse   
+unharmed myesthesia trophonema bozal immatchable   
+uninhabitedness eristically Eryon allegedly daytime periclitation phallaceous  
+phytonic yote Serrifera nonmanufacture Animalivora 
+bozal gunshop ornithodelphous Tamil chorograph Quakerishly outwealth laryngic 
hepatorrhaphy
+pinulus uninhabitedness endotheliomyoma unachievable Confervales ladhood 
chilblain 
+unachievable nonmanufacture rave Prosobranchiata dermorhynchous minniebush 
oinomancy overinstruct ventricous   
+precostal ununiformly mastication figured idiotize antihero
+Kenipsim beneficent blurredness Dawsonia twinling phallaceous impugnation 
dermorhynchous   
+packsack zanyism breadwinner Megaluridae uncarefully zenick dishpan squdge 
+dialoguer theologicopolitical subfoliar mangonism mustafina champer
+Consolamentum exploiter imperceptivity nectopod breadwinner floatability   
+heavenful theologicopolitical pyrocatechol ferrogoslarite infestation 
dermorhynchous lampyrine 
+shallowish crystallographical uninhabitedness cocksuredom yawler hysterolysis 
genii bromic 
+antalgol chronographic chasmy proboscidiform craglike stiffish 
+marshiness Scanic adz uninterpleaded trailmaking stormy impairment Dunlop 
pinulus  
+mechanist helminthagogic semantician ununiformly incomprehensible equiconvex 
adatom transcorporeal sertularian 
+migrainoid

[2/3] storm git commit: Merge branch 'STORM-2386-1.x' of https://github.com/HeartSaVioR/storm into STORM-2386-1.x

2017-03-03 Thread sriharsha

Merge branch 'STORM-2386-1.x' of https://github.com/HeartSaVioR/storm into 
STORM-2386-1.x


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/6082b78a
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/6082b78a
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/6082b78a

Branch: refs/heads/1.x-branch
Commit: 6082b78a026e2323385e39a7420951b4835e79f1
Parents: 31c2b71 9e6d331
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri Mar 3 14:13:10 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri Mar 3 14:13:10 2017 -0800

--
 .../src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
--

[1/3] storm git commit: STORM-2386 Fail-back Blob deletion also fails in BlobSynchronizer.syncBlobs

2017-03-03 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 31c2b71b8 -> 6ee5ad076


STORM-2386 Fail-back Blob deletion also fails in BlobSynchronizer.syncBlobs


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/9e6d331a
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/9e6d331a
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/9e6d331a

Branch: refs/heads/1.x-branch
Commit: 9e6d331adcc4781f56bd6a257b37be4822263754
Parents: d1e8c69
Author: Jungtaek Lim 
Authored: Tue Feb 28 12:23:54 2017 +0900
Committer: Jungtaek Lim 
Committed: Tue Feb 28 12:26:54 2017 +0900

--
 .../src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/9e6d331a/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
--
diff --git 
a/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java 
b/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
index c6647ac..3321bcf 100644
--- a/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
+++ b/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
@@ -85,9 +85,7 @@ public class BlobSynchronizer {
 BlobStoreUtils.createStateInZookeeper(conf, key, 
nimbusInfo);
 }
 } catch (KeyNotFoundException e) {
-LOG.debug("Detected deletion for the key {} - deleting the 
blob instead", key);
-// race condition with a delete, delete the blob in key 
instead
-blobStore.deleteBlob(key, 
BlobStoreUtils.getNimbusSubject());
+LOG.debug("Detected deletion for the key {} while 
downloading - skipping download", key);
 }
 }
 if (zkClient !=null) {

[3/3] storm git commit: Added STORM-2386 to CHANGELOG.

2017-03-03 Thread sriharsha

Added STORM-2386 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/6ee5ad07
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/6ee5ad07
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/6ee5ad07

Branch: refs/heads/1.x-branch
Commit: 6ee5ad076276b84cb9eaf22bdada1e00cf7dd892
Parents: 6082b78
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri Mar 3 14:14:23 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri Mar 3 14:14:23 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/6ee5ad07/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bc04fde..31dc488 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ï»¿## 1.1.0
+ * STORM-2386: Fail-back Blob deletion also fails in 
BlobSynchronizer.syncBlobs.
  * STORM-2388: JoinBolt breaks compilation against JDK 7
  * STORM-2374: Storm Kafka Client Test Topologies Must be Serializable
  * STORM-2372: Pacemaker client doesn't clean up heartbeats properly

[2/2] storm git commit: Added STORM-2386 to CHANGELOG.

2017-03-03 Thread sriharsha

Added STORM-2386 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/0bd70fd3
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/0bd70fd3
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/0bd70fd3

Branch: refs/heads/master
Commit: 0bd70fd3b7bc0dfa7dadec870c8811263d00bac3
Parents: 92da0c5
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri Mar 3 14:12:14 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri Mar 3 14:12:14 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/0bd70fd3/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8ef71ed..21a0874 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -205,6 +205,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.1
+* STORM-2386 Fail-back Blob deletion also fails in BlobSynchronizer.syncBlobs.
 * STORM-1363: TridentKafkaState should handle null values from 
TridentTupleToKafkaMapper.getMessageFromTuple()
 * STORM-2381: Add logging of JDBC connection string in storm-jdbc integration 
for debugging failures.

[1/2] storm git commit: STORM-2386 Fail-back Blob deletion also fails in BlobSynchronizer.syncBlobs

2017-03-03 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master f0bfe0d3d -> 0bd70fd3b


STORM-2386 Fail-back Blob deletion also fails in BlobSynchronizer.syncBlobs


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/92da0c56
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/92da0c56
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/92da0c56

Branch: refs/heads/master
Commit: 92da0c5635c8cee378aac46383847e4530e51bc4
Parents: f0bfe0d
Author: Jungtaek Lim 
Authored: Tue Feb 28 12:23:54 2017 +0900
Committer: Jungtaek Lim 
Committed: Tue Feb 28 12:29:39 2017 +0900

--
 .../src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/92da0c56/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
--
diff --git 
a/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java 
b/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
index b6a500a..18f0798 100644
--- a/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
+++ b/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
@@ -86,9 +86,7 @@ public class BlobSynchronizer {
 BlobStoreUtils.createStateInZookeeper(conf, key, 
nimbusInfo);
 }
 } catch (KeyNotFoundException e) {
-LOG.debug("Detected deletion for the key {} - deleting the 
blob instead", key);
-// race condition with a delete, delete the blob in key 
instead
-blobStore.deleteBlob(key, 
BlobStoreUtils.getNimbusSubject());
+LOG.debug("Detected deletion for the key {} while 
downloading - skipping download", key);
 }
 }
 if (zkClient !=null) {

[2/2] storm git commit: Merge branch 'apache-50022' of https://github.com/tkuwabara789/storm into STORM-2381-1.x

2017-02-26 Thread sriharsha

Merge branch 'apache-50022' of https://github.com/tkuwabara789/storm into 
STORM-2381-1.x


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/d1e8c692
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/d1e8c692
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/d1e8c692

Branch: refs/heads/1.x-branch
Commit: d1e8c692415bb2b030e2a9af273fbb63774077fb
Parents: 7662dff 5134295
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Sun Feb 26 11:47:27 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Sun Feb 26 11:47:27 2017 -0800

--
 .../storm/jdbc/common/HikariCPConnectionProvider.java | 10 ++
 1 file changed, 10 insertions(+)
--

[1/2] storm git commit: STORM-2381: Add logging of JDBC connection string in storm-jdbc integration for debugging failures

2017-02-26 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 7662dff22 -> d1e8c6924


STORM-2381: Add logging of JDBC connection string in storm-jdbc integration for 
debugging failures


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/51342959
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/51342959
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/51342959

Branch: refs/heads/1.x-branch
Commit: 513429596ebc7894a9f2d92e85e23dd6f267a052
Parents: d362052
Author: Hiro Kuwabara 
Authored: Thu Feb 23 16:55:47 2017 -0800
Committer: Hiro Kuwabara 
Committed: Fri Feb 24 11:26:10 2017 -0800

--
 .../storm/jdbc/common/HikariCPConnectionProvider.java | 10 ++
 1 file changed, 10 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/51342959/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/HikariCPConnectionProvider.java
--
diff --git 
a/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/HikariCPConnectionProvider.java
 
b/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/HikariCPConnectionProvider.java
index 1aa377f..3f3151d 100644
--- 
a/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/HikariCPConnectionProvider.java
+++ 
b/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/HikariCPConnectionProvider.java
@@ -25,7 +25,11 @@ import java.sql.SQLException;
 import java.util.Map;
 import java.util.Properties;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
 public class HikariCPConnectionProvider implements ConnectionProvider {
+private static final Logger LOG = 
LoggerFactory.getLogger(HikariCPConnectionProvider.class);
 
 private Map configMap;
 private transient HikariDataSource dataSource;
@@ -40,6 +44,12 @@ public class HikariCPConnectionProvider implements 
ConnectionProvider {
 Properties properties = new Properties();
 properties.putAll(configMap);
 HikariConfig config = new HikariConfig(properties);
+if(properties.containsKey("dataSource.url")) {
+LOG.info("DataSource Url: " + 
properties.getProperty("dataSource.url"));
+}
+else if (config.getJdbcUrl() != null) {
+LOG.info("JDBC Url: " + config.getJdbcUrl());
+}
 this.dataSource = new HikariDataSource(config);
 this.dataSource.setAutoCommit(false);
 }

[1/3] storm git commit: STORM-2381: Add logging of JDBC connection string in storm-jdbc integration for debugging failures

2017-02-26 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master b73a772e2 -> 34406ec81


STORM-2381: Add logging of JDBC connection string in storm-jdbc integration for 
debugging failures


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/0201af36
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/0201af36
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/0201af36

Branch: refs/heads/master
Commit: 0201af3648ff8bed897ad79e4fbf61e9f942c110
Parents: 71c35dd
Author: Hiro Kuwabara 
Authored: Thu Feb 23 16:55:47 2017 -0800
Committer: Hiro Kuwabara 
Committed: Fri Feb 24 11:53:05 2017 -0800

--
 .../storm/jdbc/common/HikariCPConnectionProvider.java | 10 ++
 1 file changed, 10 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/0201af36/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/HikariCPConnectionProvider.java
--
diff --git 
a/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/HikariCPConnectionProvider.java
 
b/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/HikariCPConnectionProvider.java
index 1aa377f..3f3151d 100644
--- 
a/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/HikariCPConnectionProvider.java
+++ 
b/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/HikariCPConnectionProvider.java
@@ -25,7 +25,11 @@ import java.sql.SQLException;
 import java.util.Map;
 import java.util.Properties;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
 public class HikariCPConnectionProvider implements ConnectionProvider {
+private static final Logger LOG = 
LoggerFactory.getLogger(HikariCPConnectionProvider.class);
 
 private Map configMap;
 private transient HikariDataSource dataSource;
@@ -40,6 +44,12 @@ public class HikariCPConnectionProvider implements 
ConnectionProvider {
 Properties properties = new Properties();
 properties.putAll(configMap);
 HikariConfig config = new HikariConfig(properties);
+if(properties.containsKey("dataSource.url")) {
+LOG.info("DataSource Url: " + 
properties.getProperty("dataSource.url"));
+}
+else if (config.getJdbcUrl() != null) {
+LOG.info("JDBC Url: " + config.getJdbcUrl());
+}
 this.dataSource = new HikariDataSource(config);
 this.dataSource.setAutoCommit(false);
 }

[3/3] storm git commit: Added STORM-2381 to CHANGELOG.

2017-02-26 Thread sriharsha

Added STORM-2381 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/34406ec8
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/34406ec8
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/34406ec8

Branch: refs/heads/master
Commit: 34406ec81d0257847bf8b7984edd5406d51b7f29
Parents: 8224c6b
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Sun Feb 26 11:29:10 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Sun Feb 26 11:29:10 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/34406ec8/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d2948e2..b0a27a0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -205,6 +205,7 @@
 
 ## 1.1.1
 * STORM-1363: TridentKafkaState should handle null values from 
TridentTupleToKafkaMapper.getMessageFromTuple()
+* STORM-2381: Add logging of JDBC connection string in storm-jdbc integration 
for debugging failures.
 
 ## 1.1.0
  * STORM-2340: fix AutoCommitMode issue in KafkaSpout

[2/3] storm git commit: Merge branch 'STORM-2381' of https://github.com/tkuwabara789/storm into STORM-2381

2017-02-26 Thread sriharsha

Merge branch 'STORM-2381' of https://github.com/tkuwabara789/storm into 
STORM-2381


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/8224c6b6
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/8224c6b6
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/8224c6b6

Branch: refs/heads/master
Commit: 8224c6b60957f9e0ddd208c98d4819e30e78fe51
Parents: b73a772 0201af3
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Sun Feb 26 11:27:50 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Sun Feb 26 11:27:50 2017 -0800

--
 .../storm/jdbc/common/HikariCPConnectionProvider.java | 10 ++
 1 file changed, 10 insertions(+)
--

storm git commit: Fixed storm-kafka missing imports.

2017-02-26 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 19081ef52 -> 7662dff22


Fixed storm-kafka missing imports.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/7662dff2
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/7662dff2
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/7662dff2

Branch: refs/heads/1.x-branch
Commit: 7662dff2203c1cdf2d9d5576ebd6d759791b7a88
Parents: 19081ef
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Sun Feb 26 11:10:07 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Sun Feb 26 11:10:07 2017 -0800

--
 .../storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java  | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/7662dff2/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
--
diff --git 
a/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java 
b/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
index 58d434b..75f5563 100644
--- a/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
+++ b/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
@@ -19,6 +19,8 @@ package org.apache.storm.kafka;
 
 import org.apache.kafka.common.serialization.Serializer;
 
+import java.nio.ByteBuffer;
+import java.nio.IntBuffer;
 import java.util.Map;
 
 public class IntSerializer implements Serializer {

storm git commit: Merge branch 'master' of https://github.com/pczb/storm

2017-02-25 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch d19301e94 -> 19081ef52


Merge branch 'master' of https://github.com/pczb/storm


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/19081ef5
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/19081ef5
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/19081ef5

Branch: refs/heads/1.x-branch
Commit: 19081ef52d674e93355d70c991c2f866dc42f365
Parents: d19301e
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri Feb 24 19:33:39 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri Feb 24 20:22:21 2017 -0800

--
 .../src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java   | 1 -
 .../storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java  | 2 --
 2 files changed, 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/19081ef5/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java
--
diff --git 
a/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java 
b/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java
index dcf5908..425d5c1 100644
--- 
a/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java
+++ 
b/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java
@@ -28,7 +28,6 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.apache.storm.kafka.trident.GlobalPartitionInformation;
 
-import java.io.UnsupportedEncodingException;
 import java.net.SocketTimeoutException;
 import java.util.ArrayList;
 import java.util.List;

http://git-wip-us.apache.org/repos/asf/storm/blob/19081ef5/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
--
diff --git 
a/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java 
b/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
index 75f5563..58d434b 100644
--- a/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
+++ b/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
@@ -19,8 +19,6 @@ package org.apache.storm.kafka;
 
 import org.apache.kafka.common.serialization.Serializer;
 
-import java.nio.ByteBuffer;
-import java.nio.IntBuffer;
 import java.util.Map;
 
 public class IntSerializer implements Serializer {

[1/2] storm git commit: remove unused import

2017-02-24 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 290a4271b -> b73a772e2


remove unused import


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/8ab4fd05
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/8ab4fd05
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/8ab4fd05

Branch: refs/heads/master
Commit: 8ab4fd05ed5ed95c7a5bb7356fe228c14a8d86d7
Parents: 5b0c811
Author: zhangbiao3 
Authored: Tue Jan 24 20:49:17 2017 +0800
Committer: zhangbiao3 
Committed: Tue Jan 24 21:01:02 2017 +0800

--
 .../src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java   | 1 -
 .../storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java  | 2 --
 2 files changed, 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/8ab4fd05/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java
--
diff --git 
a/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java 
b/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java
index dcf5908..425d5c1 100644
--- 
a/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java
+++ 
b/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java
@@ -28,7 +28,6 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.apache.storm.kafka.trident.GlobalPartitionInformation;
 
-import java.io.UnsupportedEncodingException;
 import java.net.SocketTimeoutException;
 import java.util.ArrayList;
 import java.util.List;

http://git-wip-us.apache.org/repos/asf/storm/blob/8ab4fd05/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
--
diff --git 
a/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java 
b/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
index 0a2937b..7cdfc87 100644
--- a/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
+++ b/external/storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java
@@ -19,8 +19,6 @@ package org.apache.storm.kafka;
 
 import org.apache.kafka.common.serialization.Serializer;
 
-import java.nio.ByteBuffer;
-import java.nio.IntBuffer;
 import java.util.Map;
 
 public class IntSerializer implements Serializer {

[2/2] storm git commit: Merge branch 'master' of https://github.com/pczb/storm

2017-02-24 Thread sriharsha

Merge branch 'master' of https://github.com/pczb/storm


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/b73a772e
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/b73a772e
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/b73a772e

Branch: refs/heads/master
Commit: b73a772e23ec8fe1101e9d8e960c282a4139352e
Parents: 290a427 8ab4fd0
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri Feb 24 19:33:39 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri Feb 24 19:33:39 2017 -0800

--
 .../src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java   | 1 -
 .../storm-kafka/src/jvm/org/apache/storm/kafka/IntSerializer.java  | 2 --
 2 files changed, 3 deletions(-)
--

[1/2] storm git commit: Update storm-kafka-client.md

2017-02-24 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 71c35ddc3 -> 290a4271b


Update storm-kafka-client.md

Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/9daddac7
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/9daddac7
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/9daddac7

Branch: refs/heads/master
Commit: 9daddac7389df3414325c0a80eac490c16bbf99a
Parents: d3250b2
Author: Tandy <982963...@qq.com>
Authored: Tue Feb 7 16:46:08 2017 +0800
Committer: GitHub 
Committed: Tue Feb 7 16:46:08 2017 +0800

--
 docs/storm-kafka-client.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/9daddac7/docs/storm-kafka-client.md
--
diff --git a/docs/storm-kafka-client.md b/docs/storm-kafka-client.md
index ec5056f..557e697 100644
--- a/docs/storm-kafka-client.md
+++ b/docs/storm-kafka-client.md
@@ -1,5 +1,5 @@
 #Storm Apache Kafka integration using the kafka-client jar
-This includes the new Apache Kafka copnsumer API.
+This includes the new Apache Kafka consumer API.
 
 ##Compatibility

storm git commit: Merge branch 'patch-2' of https://github.com/cutd/storm

2017-02-24 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 6b46ab97e -> d19301e94


Merge branch 'patch-2' of https://github.com/cutd/storm


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/d19301e9
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/d19301e9
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/d19301e9

Branch: refs/heads/1.x-branch
Commit: d19301e946707885c40466e4e81de097008e1fd9
Parents: 6b46ab9
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri Feb 24 19:27:47 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri Feb 24 19:28:19 2017 -0800

--
 docs/storm-kafka-client.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/d19301e9/docs/storm-kafka-client.md
--
diff --git a/docs/storm-kafka-client.md b/docs/storm-kafka-client.md
index 1e2498a..215f60f 100644
--- a/docs/storm-kafka-client.md
+++ b/docs/storm-kafka-client.md
@@ -1,5 +1,5 @@
 #Storm Apache Kafka integration using the kafka-client jar
-This includes the new Apache Kafka copnsumer API.
+This includes the new Apache Kafka consumer API.
 
 ##Compatibility

[2/2] storm git commit: Merge branch 'patch-2' of https://github.com/cutd/storm

2017-02-24 Thread sriharsha

Merge branch 'patch-2' of https://github.com/cutd/storm


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/290a4271
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/290a4271
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/290a4271

Branch: refs/heads/master
Commit: 290a4271bcaa73ebc8d570da63a23dcad6074df3
Parents: 71c35dd 9daddac
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri Feb 24 19:27:47 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri Feb 24 19:27:47 2017 -0800

--
 docs/storm-kafka-client.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/290a4271/docs/storm-kafka-client.md
--

[1/3] storm git commit: STORM-2321 Handle blobstore zk key deletion in KeySequenceNumber

2017-02-02 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 1445a955a -> 4e1572e29


STORM-2321 Handle blobstore zk key deletion in KeySequenceNumber

* If NoNodeException is thrown in getKeySequenceNumber, treat it as 
KeyNotFoundException
* Change callers to handle KeyNotFoundException accordingly


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/ff720438
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/ff720438
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/ff720438

Branch: refs/heads/master
Commit: ff7204385976e2de8099b5fffa858ae29db8c5a8
Parents: 4c5e34e
Author: Jungtaek Lim 
Authored: Tue Jan 31 17:00:31 2017 +0900
Committer: Jungtaek Lim 
Committed: Tue Jan 31 17:00:31 2017 +0900

--
 .../apache/storm/blobstore/BlobStoreUtils.java  |  4 ++--
 .../storm/blobstore/BlobSynchronizer.java   | 13 +---
 .../storm/blobstore/KeySequenceNumber.java  | 21 ++--
 .../storm/blobstore/LocalFsBlobStore.java   |  4 +++-
 .../org/apache/storm/daemon/nimbus/Nimbus.java  | 11 +++---
 5 files changed, 38 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/ff720438/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java
--
diff --git a/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java 
b/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java
index ba3ccdb..6486ba2 100644
--- a/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java
+++ b/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java
@@ -240,7 +240,7 @@ public class BlobStoreUtils {
 LOG.debug("StateInfo for update {}", stateInfo);
 Set nimbusInfoList = 
getNimbodesWithLatestSequenceNumberOfBlob(zkClient, key);
 
-for (NimbusInfo nimbusInfo:nimbusInfoList) {
+for (NimbusInfo nimbusInfo : nimbusInfoList) {
 if (nimbusInfo.getHost().equals(nimbusDetails.getHost())) {
 isListContainsCurrentNimbusInfo = true;
 break;
@@ -251,7 +251,7 @@ public class BlobStoreUtils {
 LOG.debug("Updating state inside zookeeper for an update");
 createStateInZookeeper(conf, key, nimbusDetails);
 }
-} catch (NoNodeException e) {
+} catch (NoNodeException | KeyNotFoundException e) {
 //race condition with a delete
 return;
 } catch (Exception exp) {

http://git-wip-us.apache.org/repos/asf/storm/blob/ff720438/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
--
diff --git 
a/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java 
b/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
index 3d26a97..b6a500a 100644
--- a/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
+++ b/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
@@ -17,6 +17,7 @@
  */
 package org.apache.storm.blobstore;
 
+import org.apache.storm.generated.KeyNotFoundException;
 import org.apache.storm.nimbus.NimbusInfo;
 import org.apache.curator.framework.CuratorFramework;
 import org.slf4j.Logger;
@@ -79,9 +80,15 @@ public class BlobSynchronizer {
 LOG.debug("Key set Blobstore-> Zookeeper-> DownloadSet {}-> {}-> 
{}", getBlobStoreKeySet(), getZookeeperKeySet(), keySetToDownload);
 
 for (String key : keySetToDownload) {
-Set nimbusInfoSet = 
BlobStoreUtils.getNimbodesWithLatestSequenceNumberOfBlob(zkClient, key);
-if(BlobStoreUtils.downloadMissingBlob(conf, blobStore, key, 
nimbusInfoSet)) {
-BlobStoreUtils.createStateInZookeeper(conf, key, 
nimbusInfo);
+try {
+Set nimbusInfoSet = 
BlobStoreUtils.getNimbodesWithLatestSequenceNumberOfBlob(zkClient, key);
+if (BlobStoreUtils.downloadMissingBlob(conf, blobStore, 
key, nimbusInfoSet)) {
+BlobStoreUtils.createStateInZookeeper(conf, key, 
nimbusInfo);
+}
+} catch (KeyNotFoundException e) {
+LOG.debug("Detected deletion for the key {} - deleting the 
blob instead", key);
+// race condition with a delete, delete the blob in key 
instead
+blobStore.deleteBlob(key, 
BlobStoreUtils.getNimbusSubject());
 }
 }
 if (zkClient !=null) {

http://git-wip-us.apache.org/repos/asf/storm/blob/ff720438/storm-core/src/jvm/org/apache/storm/blobstore/KeySequenceNumber.java

[3/3] storm git commit: Added STORM-2321 to CHANGELOG.

2017-02-02 Thread sriharsha

Added STORM-2321 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/4e1572e2
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/4e1572e2
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/4e1572e2

Branch: refs/heads/master
Commit: 4e1572e29fbc3fd983ad6163852d84f8a3c80782
Parents: 5b31d5f
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Thu Feb 2 09:19:36 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Thu Feb 2 09:19:36 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/4e1572e2/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2f25a40..32baf01 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -196,6 +196,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.0
+ * STORM-2321: Handle blobstore zk key deletion in KeySequenceNumber.
  * STORM-2336: Close Localizer and AsyncLocalizer when supervisor is shutting 
down
  * STORM-2335: Fix broken Topology visualization with empty ':transferred' in 
executor stats
  * STORM-2331: Emitting from JavaScript should work when not anchoring.

[2/3] storm git commit: Merge branch 'STORM-2321' of https://github.com/HeartSaVioR/storm into STORM-2321

2017-02-02 Thread sriharsha

Merge branch 'STORM-2321' of https://github.com/HeartSaVioR/storm into 
STORM-2321


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/5b31d5f8
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/5b31d5f8
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/5b31d5f8

Branch: refs/heads/master
Commit: 5b31d5f8a61851fab26a28e249089ada6ff26231
Parents: 1445a95 ff72043
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Thu Feb 2 09:19:05 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Thu Feb 2 09:19:05 2017 -0800

--
 .../apache/storm/blobstore/BlobStoreUtils.java  |  4 ++--
 .../storm/blobstore/BlobSynchronizer.java   | 13 +---
 .../storm/blobstore/KeySequenceNumber.java  | 21 ++--
 .../storm/blobstore/LocalFsBlobStore.java   |  4 +++-
 .../org/apache/storm/daemon/nimbus/Nimbus.java  | 11 +++---
 5 files changed, 38 insertions(+), 15 deletions(-)
--

[4/4] storm git commit: Merge branch 'STORM-2321' into 1.x-branch

2017-02-02 Thread sriharsha

Merge branch 'STORM-2321' into 1.x-branch


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/85ac6b82
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/85ac6b82
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/85ac6b82

Branch: refs/heads/1.x-branch
Commit: 85ac6b82bae03f7cef2243ff1911c9e182cf
Parents: 52ac6b6 c276396
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Thu Feb 2 09:18:08 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Thu Feb 2 09:18:08 2017 -0800

--
 .../src/clj/org/apache/storm/daemon/nimbus.clj  |  6 +-
 .../apache/storm/blobstore/BlobStoreUtils.java  |  6 +-
 .../storm/blobstore/BlobSynchronizer.java   | 13 +---
 .../storm/blobstore/KeySequenceNumber.java  | 21 ++--
 .../storm/blobstore/LocalFsBlobStore.java   |  4 +++-
 5 files changed, 38 insertions(+), 12 deletions(-)
--

[3/4] storm git commit: Added STORM-2321 to CHANGELOG.

2017-02-02 Thread sriharsha

Added STORM-2321 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/52ac6b68
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/52ac6b68
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/52ac6b68

Branch: refs/heads/1.x-branch
Commit: 52ac6b6822492a76479d58b3d02f0a75a0d4e617
Parents: 90ce07a
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Thu Feb 2 09:17:40 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Thu Feb 2 09:17:40 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/52ac6b68/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8d4f627..6bcdcf5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ## 1.1.0
+ * STORM-2321: Handle blobstore zk key deletion in KeySequenceNumber
  * STORM-2336: Close Localizer and AsyncLocalizer when supervisor is shutting 
down
  * STORM-2335: Fix broken Topology visualization with empty ':transferred' in 
executor stats
  * STORM-2331: Emitting from JavaScript should work when not anchoring.

[2/4] storm git commit: Merge branch 'STORM-2321-1.x' of https://github.com/HeartSaVioR/storm into STORM-2321

2017-02-02 Thread sriharsha

Merge branch 'STORM-2321-1.x' of https://github.com/HeartSaVioR/storm into 
STORM-2321


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/c2763960
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/c2763960
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/c2763960

Branch: refs/heads/1.x-branch
Commit: c276396046fed73070692970b6abd3c1653eea44
Parents: 90ce07a 3f1e5cf
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Thu Feb 2 09:09:02 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Thu Feb 2 09:09:02 2017 -0800

--
 .../src/clj/org/apache/storm/daemon/nimbus.clj  |  6 +-
 .../apache/storm/blobstore/BlobStoreUtils.java  |  6 +-
 .../storm/blobstore/BlobSynchronizer.java   | 13 +---
 .../storm/blobstore/KeySequenceNumber.java  | 21 ++--
 .../storm/blobstore/LocalFsBlobStore.java   |  4 +++-
 5 files changed, 38 insertions(+), 12 deletions(-)
--

[1/4] storm git commit: STORM-2321 Handle blobstore zk key deletion in KeySequenceNumber

2017-02-02 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 90ce07ac6 -> 85ac6b82b


STORM-2321 Handle blobstore zk key deletion in KeySequenceNumber

* If NoNodeException is thrown in getKeySequenceNumber, treat it as 
KeyNotFoundException
* Change callers to handle KeyNotFoundException accordingly


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/3f1e5cf9
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/3f1e5cf9
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/3f1e5cf9

Branch: refs/heads/1.x-branch
Commit: 3f1e5cf95c47ae9068d03acb52e2903dac99644c
Parents: 2a01dbc
Author: Jungtaek Lim 
Authored: Tue Jan 31 17:00:31 2017 +0900
Committer: Jungtaek Lim 
Committed: Tue Jan 31 17:23:35 2017 +0900

--
 .../src/clj/org/apache/storm/daemon/nimbus.clj  |  6 +-
 .../apache/storm/blobstore/BlobStoreUtils.java  |  6 +-
 .../storm/blobstore/BlobSynchronizer.java   | 13 +---
 .../storm/blobstore/KeySequenceNumber.java  | 21 ++--
 .../storm/blobstore/LocalFsBlobStore.java   |  4 +++-
 5 files changed, 38 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/3f1e5cf9/storm-core/src/clj/org/apache/storm/daemon/nimbus.clj
--
diff --git a/storm-core/src/clj/org/apache/storm/daemon/nimbus.clj 
b/storm-core/src/clj/org/apache/storm/daemon/nimbus.clj
index a05dabd..d9d71a1 100644
--- a/storm-core/src/clj/org/apache/storm/daemon/nimbus.clj
+++ b/storm-core/src/clj/org/apache/storm/daemon/nimbus.clj
@@ -1295,7 +1295,11 @@
   (.deleteBlob blob-store key nimbus-subject))
 (log-debug "Creating list of key entries for blobstore inside zookeeper" 
all-keys "local" locally-available-active-keys)
 (doseq [key locally-available-active-keys]
-  (.setup-blobstore! storm-cluster-state key (:nimbus-host-port-info 
nimbus) (get-version-for-key key nimbus-host-port-info conf)
+  (try
+(.setup-blobstore! storm-cluster-state key (:nimbus-host-port-info 
nimbus) (get-version-for-key key nimbus-host-port-info conf))
+(catch KeyNotFoundException _
+  ; invalid key, remove it from blobstore
+  (.deleteBlob blob-store key nimbus-subject))
 
 (defn- get-errors [storm-cluster-state storm-id component-id]
   (->> (.errors storm-cluster-state storm-id component-id)

http://git-wip-us.apache.org/repos/asf/storm/blob/3f1e5cf9/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java
--
diff --git a/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java 
b/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java
index 0cac61f..71d286c 100644
--- a/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java
+++ b/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java
@@ -30,6 +30,7 @@ import org.apache.storm.utils.Utils;
 import org.apache.storm.utils.ZookeeperAuthInfo;
 import org.apache.curator.framework.CuratorFramework;
 import org.apache.thrift.transport.TTransportException;
+import org.apache.zookeeper.KeeperException;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -237,7 +238,7 @@ public class BlobStoreUtils {
 LOG.debug("StateInfo for update {}", stateInfo);
 Set nimbusInfoList = 
getNimbodesWithLatestSequenceNumberOfBlob(zkClient, key);
 
-for (NimbusInfo nimbusInfo:nimbusInfoList) {
+for (NimbusInfo nimbusInfo : nimbusInfoList) {
 if (nimbusInfo.getHost().equals(nimbusDetails.getHost())) {
 isListContainsCurrentNimbusInfo = true;
 break;
@@ -248,6 +249,9 @@ public class BlobStoreUtils {
 LOG.debug("Updating state inside zookeeper for an update");
 createStateInZookeeper(conf, key, nimbusDetails);
 }
+} catch (KeeperException.NoNodeException | KeyNotFoundException e) {
+//race condition with a delete
+return;
 } catch (Exception exp) {
 throw new RuntimeException(exp);
 }

http://git-wip-us.apache.org/repos/asf/storm/blob/3f1e5cf9/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
--
diff --git 
a/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java 
b/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
index bd34c32..c6647ac 100644
--- a/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
+++ b/storm-core/src/jvm/org/apache/storm/blobstore/BlobSynchronizer.java
@@ -17,6 +17,7 @@
  */
 package org.apache.storm.blobstore;
 
+import

[2/2] storm git commit: Added STORM-2331 to CHANELOG.

2017-02-01 Thread sriharsha

Added STORM-2331 to CHANELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/9c2733a3
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/9c2733a3
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/9c2733a3

Branch: refs/heads/1.x-branch
Commit: 9c2733a3b9dccdf3617db3b282909f7e24d4c474
Parents: 3b3ae98
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Wed Feb 1 15:55:04 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Wed Feb 1 15:55:04 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/9c2733a3/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1bac7a7..b145656 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ## 1.1.0
+ * STORM-2331: Emitting from JavaScript should work when not anchoring.
  * STORM-2225: change spout config to be simpler.
  * STORM-2323: Precondition for Leader Nimbus should check all topology blobs 
and also corresponding dependencies
  * STORM-2330: Fix storm sql code generation for UDAF with non standard sql 
types

[1/3] storm git commit: STORM-2331: Emitting from JavaScript should work when not using anchoring or when anchoring into multiple tuples.

2017-02-01 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 1064ed107 -> 9c8d7b916


STORM-2331: Emitting from JavaScript should work when not using anchoring or 
when anchoring
into multiple tuples.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/03b8b793
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/03b8b793
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/03b8b793

Branch: refs/heads/master
Commit: 03b8b793616d8b75130746fd496c622a50ba9ad9
Parents: 1811273
Author: Antti JÃ¤rvinen 
Authored: Sat Jan 28 13:11:26 2017 +0200
Committer: Antti JÃ¤rvinen 
Committed: Sat Jan 28 13:22:26 2017 +0200

--
 .../src/main/resources/resources/storm.js   | 20 ++--
 1 file changed, 14 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/03b8b793/storm-multilang/javascript/src/main/resources/resources/storm.js
--
diff --git a/storm-multilang/javascript/src/main/resources/resources/storm.js 
b/storm-multilang/javascript/src/main/resources/resources/storm.js
index c8462ba..206119c 100755
--- a/storm-multilang/javascript/src/main/resources/resources/storm.js
+++ b/storm-multilang/javascript/src/main/resources/resources/storm.js
@@ -138,9 +138,10 @@ Storm.prototype.handleNewTaskId = function(taskIds) {
  *
  * For bolt, the json must contain the required fields:
  * - tuple - the value to emit
- * - anchorTupleId - the value of the anchor tuple (the input tuple that lead 
to this emit). Used to track the source
  * tuple and return ack when all components successfully finished to process 
it.
  * and may contain the optional fields:
+ * - anchorTupleId - the value of the anchor tuple or array of anchor tuples 
(the input tuple(s) that lead to this emit).
+ * Used to track the source tuple and return ack when all components 
successfully finished to process it.
  * - stream (if empty - emit to default stream)
  *
  * For spout, the json must contain the required fields:
@@ -175,10 +176,10 @@ Storm.prototype.emit = function(messageDetails, 
onTaskIds) {
  *
  * For bolt, the json must contain the required fields:
  * - tuple - the value to emit
- * - anchorTupleId - the value of the anchor tuple (the input tuple that lead 
to this emit). Used to track the source
- * tuple and return ack when all components successfully finished to process 
it.
  * - task - indicate the task to send the tuple to.
  * and may contain the optional fields:
+ * - anchorTupleId - the value of the anchor tuple or array of anchor tuples 
(the input tuple(s) that lead to this emit).
+ * Used to track the source tuple and return ack when all components 
successfully finished to process it.
  * - stream (if empty - emit to default stream)
  *
  * For spout, the json must contain the required fields:
@@ -246,21 +247,28 @@ BasicBolt.prototype.constructor = BasicBolt;
  * Emit message.
  * @param commandDetails json with the required fields:
  * - tuple - the value to emit
- * - anchorTupleId - the value of the anchor tuple (the input tuple that lead 
to this emit). Used to track the source
- * tuple and return ack when all components successfully finished to process 
it.
  * and the optional fields:
+ * - anchorTupleId - the value of the anchor tuple or array of anchor tuples 
(the input tuple(s) that lead to this emit).
+ * Used to track the source tuple and return ack when all components 
successfully finished to process it.
  * - stream (if empty - emit to default stream)
  * - task (pass only to emit to specific task)
  */
 BasicBolt.prototype.__emit = function(commandDetails) {
 var self = this;
 
+var anchors = [];
+if (commandDetails.anchorTupleId instanceof Array) {
+anchors = commandDetails.anchorTupleId;
+} else if (commandDetails.anchorTupleId) {
+anchors = [commandDetails.anchorTupleId];
+}
+
 var message = {
 command: "emit",
 tuple: commandDetails.tuple,
 stream: commandDetails.stream,
 task: commandDetails.task,
-anchors: [commandDetails.anchorTupleId]
+anchors: anchors
 };
 
 this.sendMsgToParent(message);

[3/3] storm git commit: Added STORM-2331 to CHANGELOG.

2017-02-01 Thread sriharsha

Added STORM-2331 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/9c8d7b91
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/9c8d7b91
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/9c8d7b91

Branch: refs/heads/master
Commit: 9c8d7b916f09e4d7cc97db1b089496e8ab356a9a
Parents: dfbf6fe
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Wed Feb 1 15:47:16 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Wed Feb 1 15:47:16 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/9c8d7b91/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 242a247..112d055 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -195,6 +195,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.0
+ * STORM-2331: Emitting from JavaScript should work when not anchoring.
  * STORM-2320:  DRPC client printer class reusable for local and remote DRPC.
  * STORM-2225: change spout config to be simpler.
  * STORM-2330: Fix storm sql code generation for UDAF with non standard sql 
types

[2/3] storm git commit: Merge branch 'master' of https://github.com/EitZei/storm into STORM-2331

2017-02-01 Thread sriharsha

Merge branch 'master' of https://github.com/EitZei/storm into STORM-2331


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/dfbf6fef
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/dfbf6fef
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/dfbf6fef

Branch: refs/heads/master
Commit: dfbf6fef81fbd8cf29e99b167299177e9907a706
Parents: 1064ed1 03b8b79
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Wed Feb 1 15:46:36 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Wed Feb 1 15:46:36 2017 -0800

--
 .../src/main/resources/resources/storm.js   | 20 ++--
 1 file changed, 14 insertions(+), 6 deletions(-)
--

[3/3] storm git commit: Added STORM-2320 to CHANGELOG.

2017-02-01 Thread sriharsha

Added STORM-2320 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/1064ed10
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/1064ed10
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/1064ed10

Branch: refs/heads/master
Commit: 1064ed10740091fb38b884b2ded29dcceed3318f
Parents: 5ccef13
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Wed Feb 1 15:25:29 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Wed Feb 1 15:25:29 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/1064ed10/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e2f7cdd..242a247 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -195,6 +195,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.0
+ * STORM-2320:  DRPC client printer class reusable for local and remote DRPC.
  * STORM-2225: change spout config to be simpler.
  * STORM-2330: Fix storm sql code generation for UDAF with non standard sql 
types
  * STORM-2298: Don't kill Nimbus when ClusterMetricsConsumer is failed to 
initialize

[1/3] storm git commit: STORM-2320: DRPC client printer class reusable for local and remote DRPC

2017-02-01 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 251cb8876 -> 1064ed107


STORM-2320: DRPC client printer class reusable for local and remote DRPC

  - Client necessary to check for DRPC results while running in distribute mode


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/c23d93c5
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/c23d93c5
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/c23d93c5

Branch: refs/heads/master
Commit: c23d93c5ebda38b66dad1177f49a6e97fb2f1957
Parents: 11db3ea
Author: Hugo Louro 
Authored: Fri Dec 23 17:09:06 2016 -0800
Committer: Hugo Louro 
Committed: Mon Jan 23 18:29:12 2017 -0800

--
 .../TridentKafkaClientWordCountNamedTopics.java |  9 ++-
 .../storm/kafka/trident/DrpcResultsPrinter.java | 85 
 .../trident/TridentKafkaConsumerTopology.java   | 13 ++-
 .../kafka/trident/TridentKafkaWordCount.java|  8 +-
 4 files changed, 109 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/c23d93c5/examples/storm-kafka-client-examples/src/main/java/org/apache/storm/kafka/trident/TridentKafkaClientWordCountNamedTopics.java
--
diff --git 
a/examples/storm-kafka-client-examples/src/main/java/org/apache/storm/kafka/trident/TridentKafkaClientWordCountNamedTopics.java
 
b/examples/storm-kafka-client-examples/src/main/java/org/apache/storm/kafka/trident/TridentKafkaClientWordCountNamedTopics.java
index 2482eae..1bdfe65 100644
--- 
a/examples/storm-kafka-client-examples/src/main/java/org/apache/storm/kafka/trident/TridentKafkaClientWordCountNamedTopics.java
+++ 
b/examples/storm-kafka-client-examples/src/main/java/org/apache/storm/kafka/trident/TridentKafkaClientWordCountNamedTopics.java
@@ -105,7 +105,7 @@ public class TridentKafkaClientWordCountNamedTopics {
 new TridentKafkaClientWordCountNamedTopics().run(args);
 }
 
-protected void run(String[] args) throws AlreadyAliveException, 
InvalidTopologyException, AuthorizationException {
+protected void run(String[] args) throws AlreadyAliveException, 
InvalidTopologyException, AuthorizationException, InterruptedException {
 if (args.length > 0 && Arrays.stream(args).anyMatch(option -> 
option.equals("-h"))) {
 System.out.printf("Usage: java %s [%s] [%s] [%s] [%s]\n", 
getClass().getName(),
 "broker_host:broker_port", "topic1", "topic2", 
"topology_name");
@@ -124,6 +124,11 @@ public class TridentKafkaClientWordCountNamedTopics {
 StormSubmitter.submitTopology(topic2 + "-producer", tpConf, 
KafkaProducerTopology.newTopology(brokerUrl, topic2));
 // Consumer
 StormSubmitter.submitTopology("topics-consumer", tpConf, 
TridentKafkaConsumerTopology.newTopology(newKafkaTridentSpoutOpaque()));
+
+// Print results to console, which also causes the print 
filter in the consumer topology to print the results in the worker log
+Thread.sleep(2000);
+DrpcResultsPrinter.remoteClient().printResults(60, 1, 
TimeUnit.SECONDS);
+
 } else { //Submit Local
 
 final LocalSubmitter localSubmitter = 
LocalSubmitter.newInstance();
@@ -140,7 +145,7 @@ public class TridentKafkaClientWordCountNamedTopics {
 localSubmitter.getDrpc(), 
newKafkaTridentSpoutOpaque()));
 
 // print
-localSubmitter.printResults(15, 1, TimeUnit.SECONDS);
+new 
DrpcResultsPrinter(localSubmitter.getDrpc()).printResults(60, 1, 
TimeUnit.SECONDS);
 } finally {
 // kill
 localSubmitter.kill(topic1Tp);

http://git-wip-us.apache.org/repos/asf/storm/blob/c23d93c5/examples/storm-kafka-examples/src/main/java/org/apache/storm/kafka/trident/DrpcResultsPrinter.java
--
diff --git 
a/examples/storm-kafka-examples/src/main/java/org/apache/storm/kafka/trident/DrpcResultsPrinter.java
 
b/examples/storm-kafka-examples/src/main/java/org/apache/storm/kafka/trident/DrpcResultsPrinter.java
new file mode 100644
index 000..f71e2df
--- /dev/null
+++ 
b/examples/storm-kafka-examples/src/main/java/org/apache/storm/kafka/trident/DrpcResultsPrinter.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ *   or more contributor license agreements.  See the NOTICE file
+ *   distributed with this work for additional information
+ *   regarding copyright ownership.  The ASF licenses this file
+ *   to you under the Apache License, Version 2.0 (the
+ *   "License"); you may not use this file except in compliance
+ *

[2/3] storm git commit: Merge branch 'Apache_STORM-master_PrintRemoteDRPC' of https://github.com/hmcl/storm-apache into STORM-2320

2017-02-01 Thread sriharsha

Merge branch 'Apache_STORM-master_PrintRemoteDRPC' of 
https://github.com/hmcl/storm-apache into STORM-2320


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/5ccef139
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/5ccef139
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/5ccef139

Branch: refs/heads/master
Commit: 5ccef1393de96ac3e87aaca8abc0b31a5c38457f
Parents: 251cb88 c23d93c
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Wed Feb 1 15:24:30 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Wed Feb 1 15:24:30 2017 -0800

--
 .../TridentKafkaClientWordCountNamedTopics.java |  9 ++-
 .../storm/kafka/trident/DrpcResultsPrinter.java | 85 
 .../trident/TridentKafkaConsumerTopology.java   | 13 ++-
 .../kafka/trident/TridentKafkaWordCount.java|  8 +-
 4 files changed, 109 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/5ccef139/examples/storm-kafka-client-examples/src/main/java/org/apache/storm/kafka/trident/TridentKafkaClientWordCountNamedTopics.java
--

[2/3] storm git commit: Merge branch 'STORM-2323-1.x' of https://github.com/HeartSaVioR/storm into 1.x-branch

2017-01-31 Thread sriharsha

Merge branch 'STORM-2323-1.x' of https://github.com/HeartSaVioR/storm into 
1.x-branch


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/0a7170b2
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/0a7170b2
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/0a7170b2

Branch: refs/heads/1.x-branch
Commit: 0a7170b27c4f9055df26a3b2a12f7a04f2bb3eba
Parents: 2a01dbc b19b0a1
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Jan 31 18:16:29 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Jan 31 18:16:29 2017 -0800

--
 .../src/clj/org/apache/storm/zookeeper.clj  |  29 +
 .../org/apache/storm/zookeeper/Zookeeper.java   | 119 ---
 2 files changed, 104 insertions(+), 44 deletions(-)
--

[3/3] storm git commit: Added STORM-2323 to CHANGELOG.

2017-01-31 Thread sriharsha

Added STORM-2323 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/83bf0ba8
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/83bf0ba8
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/83bf0ba8

Branch: refs/heads/1.x-branch
Commit: 83bf0ba8db757d4cada464e0ce397dbe44dcdd1f
Parents: 0a7170b
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Jan 31 18:17:06 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Jan 31 18:17:06 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/83bf0ba8/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b2ed9e0..9d285e2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ## 1.1.0
+ * STORM-2323: Precondition for Leader Nimbus should check all topology blobs 
and also corresponding dependencies
  * STORM-2330: Fix storm sql code generation for UDAF with non standard sql 
types
  * STORM-2298: Don't kill Nimbus when ClusterMetricsConsumer is failed to 
initialize
  * STORM-2301: [storm-cassandra] upgrade cassandra driver to 3.1.2

[1/3] storm git commit: STORM-2323 Precondition for Leader Nimbus should check all topology blobs and also corresponding dependencies

2017-01-31 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 2a01dbc75 -> 83bf0ba8d


STORM-2323 Precondition for Leader Nimbus should check all topology blobs and 
also corresponding dependencies

* change the precondition for leader Nimbus
** it should have all active topology blobs and corresponding dependencies 
locally
* change zookeeper.clj to use Zookeeper.leaderLatchListenerImpl


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/b19b0a1d
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/b19b0a1d
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/b19b0a1d

Branch: refs/heads/1.x-branch
Commit: b19b0a1da4488bf91d60aa1487d6ed6109af4162
Parents: 194a9cb
Author: Jungtaek Lim 
Authored: Wed Jan 25 13:16:44 2017 +0900
Committer: Jungtaek Lim 
Committed: Mon Jan 30 22:14:47 2017 +0900

--
 .../src/clj/org/apache/storm/zookeeper.clj  |  29 +
 .../org/apache/storm/zookeeper/Zookeeper.java   | 119 ---
 2 files changed, 104 insertions(+), 44 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/b19b0a1d/storm-core/src/clj/org/apache/storm/zookeeper.clj
--
diff --git a/storm-core/src/clj/org/apache/storm/zookeeper.clj 
b/storm-core/src/clj/org/apache/storm/zookeeper.clj
index 2b5da55..ca41093 100644
--- a/storm-core/src/clj/org/apache/storm/zookeeper.clj
+++ b/storm-core/src/clj/org/apache/storm/zookeeper.clj
@@ -32,7 +32,8 @@
   (:import [java.io File])
   (:import [java.util List Map])
   (:import [org.apache.storm.utils Utils ZookeeperAuthInfo]
-   (org.apache.storm.blobstore KeyFilter BlobStore))
+   (org.apache.storm.blobstore KeyFilter BlobStore)
+   (org.apache.storm.zookeeper Zookeeper))
   (:use [org.apache.storm util log config]))
 
 (def zk-keeper-states
@@ -249,28 +250,6 @@
 (filter [this key] (get-id-from-blob-key key)))]
 (set (.filterAndListKeys blob-store to-id
 
-(defn leader-latch-listener-impl
-  "Leader latch listener that will be invoked when we either gain or lose 
leadership"
-  [conf zk blob-store leader-latch]
-  (let [hostname (.getCanonicalHostName (InetAddress/getLocalHost))
-STORMS-ROOT (str (conf STORM-ZOOKEEPER-ROOT) "/storms")]
-(reify LeaderLatchListener
-  (^void isLeader[this]
-(log-message (str hostname " gained leadership, checking if it has all 
the topology code locally."))
-(let [active-topology-ids (set (get-children zk STORMS-ROOT false))
-  local-topology-ids (set (code-ids blob-store))
-  diff-topology (set/difference active-topology-ids 
local-topology-ids)]
-  (log-message "active-topology-ids [" (clojure.string/join "," 
active-topology-ids)
-   "] local-topology-ids [" (clojure.string/join "," 
local-topology-ids)
-   "] diff-topology [" (clojure.string/join "," 
diff-topology) "]")
-  (if (empty? diff-topology)
-(log-message "Accepting leadership, all active topology found 
localy.")
-(do
-  (log-message "code for all active topologies not available 
locally, giving up leadership.")
-  (.close leader-latch)
-  (^void notLeader[this]
-(log-message (str hostname " lost leadership."))
-
 (defn zk-leader-elector
   "Zookeeper Implementation of ILeaderElector."
   [conf blob-store]
@@ -279,7 +258,7 @@
 leader-lock-path (str (conf STORM-ZOOKEEPER-ROOT) "/leader-lock")
 id (.toHostPortString (NimbusInfo/fromConf conf))
 leader-latch (atom (LeaderLatch. zk leader-lock-path id))
-leader-latch-listener (atom (leader-latch-listener-impl conf zk 
blob-store @leader-latch))
+leader-latch-listener (atom (Zookeeper/leaderLatchListenerImpl conf zk 
blob-store @leader-latch))
 ]
 (reify ILeaderElector
   (prepare [this conf]
@@ -290,7 +269,7 @@
 (if (.equals LeaderLatch$State/CLOSED (.getState @leader-latch))
   (do
 (reset! leader-latch (LeaderLatch. zk leader-lock-path id))
-(reset! leader-latch-listener (leader-latch-listener-impl conf zk 
blob-store @leader-latch))
+(reset! leader-latch-listener (Zookeeper/leaderLatchListenerImpl 
conf zk blob-store @leader-latch))
 (log-message "LeaderLatch was in closed state. Resetted the 
leaderLatch and listeners.")
 ))
 

http://git-wip-us.apache.org/repos/asf/storm/blob/b19b0a1d/storm-core/src/jvm/org/apache/storm/zookeeper/Zookeeper.java
--
diff --git a/storm-core/src/jvm/org/apache/storm/zookeeper/Zookeeper.java

[3/3] storm git commit: Added STORM-2323 to CHANGELOG.

2017-01-31 Thread sriharsha

Added STORM-2323 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/22cecb08
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/22cecb08
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/22cecb08

Branch: refs/heads/master
Commit: 22cecb08770ffa325a2e0fc92c2077873262c434
Parents: 85fd221
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Jan 31 17:52:21 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Jan 31 17:52:21 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/22cecb08/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3d50214..1fde2c6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ## 2.0.0
+ * STORM-2323: Precondition for Leader Nimbus should check all topology blobs 
and also corresponding dependencies.
  * STORM-2305: STORM-2279 calculates task index different from grouper code
  * STORM-1292: port backtype.storm.messaging-test to java
  * STORM-2271: ClosedByInterruptException should be handled in few cases and 
removing a confusing debug statement

[1/3] storm git commit: STORM-2323 Precondition for Leader Nimbus should check all topology blobs and also corresponding dependencies

2017-01-31 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 4c5e34ee6 -> 22cecb087


STORM-2323 Precondition for Leader Nimbus should check all topology blobs and 
also corresponding dependencies

* change the precondition for leader Nimbus
** it should have all active topology blobs and corresponding dependencies 
locally


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/f7373f43
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/f7373f43
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/f7373f43

Branch: refs/heads/master
Commit: f7373f43511d1410f0f026034a4c8194e5f1a7f1
Parents: 1811273
Author: Jungtaek Lim 
Authored: Wed Jan 25 13:16:44 2017 +0900
Committer: Jungtaek Lim 
Committed: Thu Jan 26 08:58:20 2017 +0900

--
 .../org/apache/storm/zookeeper/Zookeeper.java   | 119 ---
 1 file changed, 100 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/f7373f43/storm-core/src/jvm/org/apache/storm/zookeeper/Zookeeper.java
--
diff --git a/storm-core/src/jvm/org/apache/storm/zookeeper/Zookeeper.java 
b/storm-core/src/jvm/org/apache/storm/zookeeper/Zookeeper.java
index ef35307..a2ad797 100644
--- a/storm-core/src/jvm/org/apache/storm/zookeeper/Zookeeper.java
+++ b/storm-core/src/jvm/org/apache/storm/zookeeper/Zookeeper.java
@@ -20,6 +20,7 @@ package org.apache.storm.zookeeper;
 import com.google.common.base.Joiner;
 import com.google.common.collect.Sets;
 
+import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang.StringUtils;
 import org.apache.curator.framework.CuratorFramework;
 import org.apache.curator.framework.api.CuratorEvent;
@@ -31,15 +32,17 @@ import 
org.apache.curator.framework.recipes.leader.Participant;
 import org.apache.curator.framework.state.ConnectionStateListener;
 import org.apache.storm.Config;
 import org.apache.storm.blobstore.BlobStore;
-import org.apache.storm.blobstore.KeyFilter;
+import org.apache.storm.blobstore.InputStreamWithMeta;
 import org.apache.storm.callback.DefaultWatcherCallBack;
 import org.apache.storm.callback.WatcherCallBack;
 import org.apache.storm.cluster.ClusterUtils;
-import org.apache.storm.cluster.IStateStorage;
 import org.apache.storm.cluster.VersionedData;
+import org.apache.storm.generated.AuthorizationException;
+import org.apache.storm.generated.KeyNotFoundException;
+import org.apache.storm.generated.StormTopology;
 import org.apache.storm.nimbus.ILeaderElector;
 import org.apache.storm.nimbus.NimbusInfo;
-import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.security.auth.ReqContext;
 import org.apache.storm.utils.Utils;
 import org.apache.storm.utils.ZookeeperAuthInfo;
 import org.apache.zookeeper.KeeperException;
@@ -51,6 +54,7 @@ import org.apache.zookeeper.server.ZooKeeperServer;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import javax.security.auth.Subject;
 import java.io.File;
 import java.io.IOException;
 import java.net.BindException;
@@ -336,29 +340,43 @@ public class Zookeeper {
 public static LeaderLatchListener leaderLatchListenerImpl(final Map conf, 
final CuratorFramework zk, final BlobStore blobStore, final LeaderLatch 
leaderLatch) throws UnknownHostException {
 final String hostName = 
InetAddress.getLocalHost().getCanonicalHostName();
 return new LeaderLatchListener() {
+final String STORM_JAR_SUFFIX = "-stormjar.jar";
+final String STORM_CODE_SUFFIX = "-stormcode.ser";
+final String STORM_CONF_SUFFIX = "-stormconf.ser";
+
 @Override
 public void isLeader() {
-Set activeTopologyIds = new 
HashSet<>(Zookeeper.getChildren(zk, conf.get(Config.STORM_ZOOKEEPER_ROOT) + 
ClusterUtils.STORMS_SUBTREE, false));
-Set localTopologyIds = blobStore.filterAndListKeys(new 
KeyFilter() {
-@Override
-public String filter(String key) {
-return ConfigUtils.getIdFromBlobKey(key);
-}
-});
-Sets.SetView diffTopology = 
Sets.difference(activeTopologyIds, localTopologyIds);
-LOG.info("active-topology-ids [{}] local-topology-ids [{}] 
diff-topology [{}]",
-generateJoinedString(activeTopologyIds), 
generateJoinedString(localTopologyIds),
+Set activeTopologyIds = new 
TreeSet<>(Zookeeper.getChildren(zk, conf.get(Config.STORM_ZOOKEEPER_ROOT) + 
ClusterUtils.STORMS_SUBTREE, false));
+
+Set activeTopologyBlobKeys = 
populateTopologyBlobKeys(activeTopologyIds);
+Set activeTopologyCodeKeys = 
filterTopologyCodeKeys(activeTopologyBlobKeys);
+

[2/3] storm git commit: Merge branch 'STORM-2323' of https://github.com/HeartSaVioR/storm

2017-01-31 Thread sriharsha

Merge branch 'STORM-2323' of https://github.com/HeartSaVioR/storm


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/85fd2213
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/85fd2213
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/85fd2213

Branch: refs/heads/master
Commit: 85fd22131fd8232a6744bea2fb917f382557a60c
Parents: 4c5e34e f7373f4
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Jan 31 17:51:24 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Jan 31 17:51:24 2017 -0800

--
 .../org/apache/storm/zookeeper/Zookeeper.java   | 119 ---
 1 file changed, 100 insertions(+), 19 deletions(-)
--

storm git commit: Minor: Added missing Apache License.

2017-01-18 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master b1b7d00cb -> 5b0c81154


Minor: Added missing Apache License.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/5b0c8115
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/5b0c8115
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/5b0c8115

Branch: refs/heads/master
Commit: 5b0c8115474924a24d4755b4d596a2e6feff60ce
Parents: b1b7d00
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Wed Jan 18 16:35:51 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Wed Jan 18 16:35:51 2017 -0800

--
 .../kafka/spout/TopicPartitionComparator.java | 18 ++
 1 file changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/5b0c8115/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/TopicPartitionComparator.java
--
diff --git 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/TopicPartitionComparator.java
 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/TopicPartitionComparator.java
index 05eb455..b908001 100644
--- 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/TopicPartitionComparator.java
+++ 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/TopicPartitionComparator.java
@@ -1,3 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ *   or more contributor license agreements.  See the NOTICE file
+ *   distributed with this work for additional information
+ *   regarding copyright ownership.  The ASF licenses this file
+ *   to you under the Apache License, Version 2.0 (the
+ *   "License"); you may not use this file except in compliance
+ *   with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *   Unless required by applicable law or agreed to in writing, software
+ *   distributed under the License is distributed on an "AS IS" BASIS,
+ *   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *   See the License for the specific language governing permissions and
+ *   limitations under the License.
+ */
+
 package org.apache.storm.kafka.spout;
 
 import org.apache.kafka.common.TopicPartition;

[1/3] storm git commit: STORM-2236 Add manual partition management support for kafka support.

2017-01-13 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master 00fed518d -> 0c12cc4d7


STORM-2236 Add manual partition management support for kafka support.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/00f54089
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/00f54089
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/00f54089

Branch: refs/heads/master
Commit: 00f54089caeceb6bf7293c751606163da5130a75
Parents: ff8fc2c
Author: liurenjie1024 
Authored: Mon Dec 19 18:10:30 2016 +0800
Committer: liurenjie1024 
Committed: Wed Jan 4 11:58:43 2017 +0800

--
 .../apache/storm/kafka/spout/KafkaSpout.java|  95 
 .../storm/kafka/spout/KafkaSpoutConfig.java |  34 ++
 .../kafka/spout/TopicPartitionComparator.java   |  16 +++
 .../storm/kafka/spout/internal/Timer.java   |  75 +
 .../fetcher/AutomaticKafkaRecordsFetcher.java   |  67 
 .../internal/fetcher/KafkaRecordsFetcher.java   |  25 +
 .../internal/fetcher/KafkaRecordsFetchers.java  |  54 ++
 .../fetcher/ManualKafkaRecordsFetcher.java  | 108 +++
 .../partition/KafkaPartitionReader.java |  28 +
 .../partition/KafkaPartitionReaders.java|  45 
 .../partition/NamedTopicPartitionReader.java|  47 
 .../partition/WildcardTopicPartitionReader.java |  51 +
 12 files changed, 568 insertions(+), 77 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/00f54089/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
--
diff --git 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
index d405c4d..dcb245c 100644
--- 
a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
+++ 
b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
@@ -24,7 +24,14 @@ import org.apache.kafka.clients.consumer.ConsumerRecords;
 import org.apache.kafka.clients.consumer.KafkaConsumer;
 import org.apache.kafka.clients.consumer.OffsetAndMetadata;
 import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.RetriableException;
 import org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy;
+import org.apache.storm.kafka.spout.internal.KafkaConsumerFactory;
+import org.apache.storm.kafka.spout.internal.KafkaConsumerFactoryDefault;
+import org.apache.storm.kafka.spout.internal.Timer;
+import org.apache.storm.kafka.spout.internal.fetcher.KafkaRecordsFetcher;
+import org.apache.storm.kafka.spout.internal.fetcher.KafkaRecordsFetchers;
 import org.apache.storm.spout.SpoutOutputCollector;
 import org.apache.storm.task.TopologyContext;
 import org.apache.storm.topology.OutputFieldsDeclarer;
@@ -46,16 +53,11 @@ import java.util.NavigableSet;
 import java.util.Set;
 import java.util.TreeSet;
 import java.util.concurrent.TimeUnit;
-import java.util.regex.Pattern;
 
 import static 
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.EARLIEST;
 import static 
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.LATEST;
 import static 
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST;
 import static 
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.UNCOMMITTED_LATEST;
-import org.apache.storm.kafka.spout.internal.KafkaConsumerFactory;
-import org.apache.storm.kafka.spout.internal.KafkaConsumerFactoryDefault;
-
-import org.apache.kafka.common.errors.InterruptException;
 
 public class KafkaSpout extends BaseRichSpout {
 private static final Logger LOG = 
LoggerFactory.getLogger(KafkaSpout.class);
@@ -63,6 +65,7 @@ public class KafkaSpout extends BaseRichSpout {
 
 // Storm
 protected SpoutOutputCollector collector;
+private TopologyContext topologyContext;
 
 // Kafka
 private final KafkaSpoutConfig kafkaSpoutConfig;
@@ -77,6 +80,7 @@ public class KafkaSpout extends BaseRichSpout {
 private transient KafkaSpoutRetryService retryService;  // 
Class that has the logic to handle tuple failure
 private transient Timer commitTimer;// 
timer == null for auto commit mode
 private transient boolean initialized;  // 
Flag indicating that the spout is still undergoing initialization process.
+private transient KafkaRecordsFetcher recordsFetcher; // 
Class that encapsulates the logic of

[3/3] storm git commit: Added STORM-2236 to CHANGELOG.

2017-01-13 Thread sriharsha

Added STORM-2236 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/0c12cc4d
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/0c12cc4d
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/0c12cc4d

Branch: refs/heads/master
Commit: 0c12cc4d74fe4e52a64840e30eb07794cd555e47
Parents: e2f87c3
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri Jan 13 20:46:44 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri Jan 13 20:46:44 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/0c12cc4d/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8912a46..81b1185 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -192,6 +192,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.0
+ * STORM-2236: Kafka Spout with manual partition management.
  * STORM-2280: Upgrade Calcite version to 1.11.0
  * STORM-2278: Allow max number of disruptor queue flusher threads to be 
configurable
  * STORM-2277: Add shaded jar for Druid connector

[2/3] storm git commit: Merge branch 'STORM-2236' of https://github.com/MediaV/storm into STORM-2236

2017-01-13 Thread sriharsha

Merge branch 'STORM-2236' of https://github.com/MediaV/storm into STORM-2236


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/e2f87c30
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/e2f87c30
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/e2f87c30

Branch: refs/heads/master
Commit: e2f87c304297ce95f01708e0934c44ee385fa0bb
Parents: 00fed51 00f5408
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Fri Jan 13 20:45:57 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Fri Jan 13 20:45:57 2017 -0800

--
 .../apache/storm/kafka/spout/KafkaSpout.java|  95 
 .../storm/kafka/spout/KafkaSpoutConfig.java |  34 ++
 .../kafka/spout/TopicPartitionComparator.java   |  16 +++
 .../storm/kafka/spout/internal/Timer.java   |  75 +
 .../fetcher/AutomaticKafkaRecordsFetcher.java   |  67 
 .../internal/fetcher/KafkaRecordsFetcher.java   |  25 +
 .../internal/fetcher/KafkaRecordsFetchers.java  |  54 ++
 .../fetcher/ManualKafkaRecordsFetcher.java  | 108 +++
 .../partition/KafkaPartitionReader.java |  28 +
 .../partition/KafkaPartitionReaders.java|  45 
 .../partition/NamedTopicPartitionReader.java|  47 
 .../partition/WildcardTopicPartitionReader.java |  51 +
 12 files changed, 568 insertions(+), 77 deletions(-)
--

[2/3] storm git commit: Merge branch 'STORM-2244' of https://github.com/HeartSaVioR/storm into STORM-2244

2017-01-03 Thread sriharsha

Merge branch 'STORM-2244' of https://github.com/HeartSaVioR/storm into 
STORM-2244


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/307f92b1
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/307f92b1
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/307f92b1

Branch: refs/heads/master
Commit: 307f92b176e9d4a8af35d795bc588175e1d724da
Parents: f145174 e320eac
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Jan 3 11:52:31 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Jan 3 11:52:31 2017 -0800

--
 examples/storm-elasticsearch-examples/pom.xml | 17 +
 examples/storm-hbase-examples/pom.xml | 17 +
 examples/storm-hdfs-examples/pom.xml  | 17 +
 examples/storm-hive-examples/pom.xml  | 17 +
 examples/storm-jdbc-examples/pom.xml  | 17 +
 examples/storm-kafka-client-examples/pom.xml  | 17 +
 examples/storm-kafka-examples/pom.xml | 17 +
 examples/storm-mongodb-examples/pom.xml   | 17 +
 examples/storm-mqtt-examples/pom.xml  | 17 +
 examples/storm-opentsdb-examples/pom.xml  | 17 +
 examples/storm-redis-examples/pom.xml | 17 +
 examples/storm-solr-examples/pom.xml  | 17 +
 examples/storm-starter/pom.xml| 17 +
 external/flux/flux-core/pom.xml   | 17 +
 external/flux/flux-examples/pom.xml   | 17 +
 external/storm-eventhubs/pom.xml  | 18 ++
 external/storm-kafka-monitor/pom.xml  | 20 
 external/storm-submit-tools/pom.xml   | 17 +
 storm-core/pom.xml| 15 +++
 storm-rename-hack/pom.xml | 17 +
 20 files changed, 342 insertions(+)
--

[3/3] storm git commit: Added STORM-2244 to CHANGELOG.

2017-01-03 Thread sriharsha

Added STORM-2244 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/f5c291b8
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/f5c291b8
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/f5c291b8

Branch: refs/heads/master
Commit: f5c291b871195cca9593d76ca2e3a4abc70ea57b
Parents: 307f92b
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Tue Jan 3 11:53:12 2017 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Tue Jan 3 11:53:12 2017 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/f5c291b8/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bed79b7..024901f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -188,6 +188,7 @@
  * STORM-1769: Added a test to check local nimbus with notifier plugin
 
 ## 1.1.0
+ * STORM-2244: Some shaded jars doesn't exclude dependency signature files
  * STORM-2266: Close NimbusClient instances appropriately
  * STORM-2257: Add built in support for sum function with different types
  * STORM-2082: add sql external module storm-sql-hdfs

[1/3] storm git commit: STORM-2244 Some shaded jars doesn't exclude dependency signature files

2017-01-03 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/master f14517493 -> f5c291b87


STORM-2244 Some shaded jars doesn't exclude dependency signature files

* Add filter option to all of shade plugin configurations


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/e320eac4
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/e320eac4
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/e320eac4

Branch: refs/heads/master
Commit: e320eac4673dcf90336bf088a8b1e19ff75be9c1
Parents: 2aa7791
Author: Jungtaek Lim 
Authored: Mon Dec 19 07:24:08 2016 +0900
Committer: Jungtaek Lim 
Committed: Mon Dec 19 07:24:08 2016 +0900

--
 examples/storm-elasticsearch-examples/pom.xml | 17 +
 examples/storm-hbase-examples/pom.xml | 17 +
 examples/storm-hdfs-examples/pom.xml  | 17 +
 examples/storm-hive-examples/pom.xml  | 17 +
 examples/storm-jdbc-examples/pom.xml  | 17 +
 examples/storm-kafka-client-examples/pom.xml  | 17 +
 examples/storm-kafka-examples/pom.xml | 17 +
 examples/storm-mongodb-examples/pom.xml   | 17 +
 examples/storm-mqtt-examples/pom.xml  | 17 +
 examples/storm-opentsdb-examples/pom.xml  | 17 +
 examples/storm-redis-examples/pom.xml | 17 +
 examples/storm-solr-examples/pom.xml  | 17 +
 examples/storm-starter/pom.xml| 17 +
 external/flux/flux-core/pom.xml   | 17 +
 external/flux/flux-examples/pom.xml   | 17 +
 external/storm-eventhubs/pom.xml  | 18 ++
 external/storm-kafka-monitor/pom.xml  | 20 
 external/storm-submit-tools/pom.xml   | 17 +
 storm-core/pom.xml| 15 +++
 storm-rename-hack/pom.xml | 17 +
 20 files changed, 342 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/e320eac4/examples/storm-elasticsearch-examples/pom.xml
--
diff --git a/examples/storm-elasticsearch-examples/pom.xml 
b/examples/storm-elasticsearch-examples/pom.xml
index 3bf0f2b..0c312fc 100644
--- a/examples/storm-elasticsearch-examples/pom.xml
+++ b/examples/storm-elasticsearch-examples/pom.xml
@@ -62,6 +62,23 @@
 maven-shade-plugin
 
 
true
+
+
+*:*
+
+META-INF/*.SF
+META-INF/*.sf
+META-INF/*.DSA
+META-INF/*.dsa
+META-INF/*.RSA
+META-INF/*.rsa
+META-INF/*.EC
+META-INF/*.ec
+META-INF/MSFTSIG.SF
+META-INF/MSFTSIG.RSA
+
+
+
 
 
 

http://git-wip-us.apache.org/repos/asf/storm/blob/e320eac4/examples/storm-hbase-examples/pom.xml
--
diff --git a/examples/storm-hbase-examples/pom.xml 
b/examples/storm-hbase-examples/pom.xml
index 677a66a..1b26e92 100644
--- a/examples/storm-hbase-examples/pom.xml
+++ b/examples/storm-hbase-examples/pom.xml
@@ -62,6 +62,23 @@
 maven-shade-plugin
 
 
true
+
+
+*:*
+
+META-INF/*.SF
+META-INF/*.sf
+META-INF/*.DSA
+META-INF/*.dsa
+META-INF/*.RSA
+META-INF/*.rsa
+META-INF/*.EC
+META-INF/*.ec
+META-INF/MSFTSIG.SF
+META-INF/MSFTSIG.RSA
+
+
+
 
 
 

http://git-wip-us.apache.org/repos/asf/storm/blob/e320eac4/examples/storm-hdfs-examples/pom.xml
--
diff --git a/examples/storm-hdfs-examples/pom.xml

[3/3] storm git commit: Added STORM-2257 to CHANGELOG.

2016-12-28 Thread sriharsha

Added STORM-2257 to CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/03efdb62
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/03efdb62
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/03efdb62

Branch: refs/heads/1.x-branch
Commit: 03efdb62f782a970f6707efd32759d227c0b7788
Parents: cf22540
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Wed Dec 28 12:28:16 2016 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Wed Dec 28 12:28:16 2016 -0800

--
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/03efdb62/CHANGELOG.md
--
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d5effe0..54263d1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ## 1.1.0
+ * STORM-2257: Add built in support for sum function with different types.
  * STORM-2082: add sql external module storm-sql-hdfs
  * STORM-2256: storm-pmml breaks on java 1.7
  * STORM-2223: PMML Bolt.

[2/3] storm git commit: Merge branch 'STORM-2257' of https://github.com/arunmahadevan/storm into 1.x-branch

2016-12-28 Thread sriharsha

Merge branch 'STORM-2257' of https://github.com/arunmahadevan/storm into 
1.x-branch


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/cf22540a
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/cf22540a
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/cf22540a

Branch: refs/heads/1.x-branch
Commit: cf22540ac7a74201cece8b4718f64c578f789db6
Parents: a37e2d3 10e8173
Author: Sriharsha Chintalapani <har...@hortonworks.com>
Authored: Wed Dec 28 12:27:35 2016 -0800
Committer: Sriharsha Chintalapani <har...@hortonworks.com>
Committed: Wed Dec 28 12:27:35 2016 -0800

--
 .../standalone/BuiltinAggregateFunctions.java   | 60 
 1 file changed, 60 insertions(+)
--

[1/3] storm git commit: STORM-2257: Add built in support for sum function with different types in storm-sql standalone mode

2016-12-28 Thread sriharsha

Repository: storm
Updated Branches:
  refs/heads/1.x-branch a37e2d3e9 -> 03efdb62f


STORM-2257: Add built in support for sum function with different types in 
storm-sql standalone mode


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/10e8173e
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/10e8173e
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/10e8173e

Branch: refs/heads/1.x-branch
Commit: 10e8173e9a027a28312f3cb11417e978a5c4305e
Parents: a2fe4e4
Author: Arun Mahadevan 
Authored: Mon Dec 26 16:33:42 2016 +0530
Committer: Arun Mahadevan 
Committed: Mon Dec 26 16:33:42 2016 +0530

--
 .../standalone/BuiltinAggregateFunctions.java   | 60 
 1 file changed, 60 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/storm/blob/10e8173e/external/sql/storm-sql-core/src/jvm/org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.java
--
diff --git 
a/external/sql/storm-sql-core/src/jvm/org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.java
 
b/external/sql/storm-sql-core/src/jvm/org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.java
index 780affc..9dc4ba8 100644
--- 
a/external/sql/storm-sql-core/src/jvm/org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.java
+++ 
b/external/sql/storm-sql-core/src/jvm/org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.java
@@ -51,6 +51,34 @@ public class BuiltinAggregateFunctions {
 
 static final Map TABLE = new HashMap<>();
 
+public static class ByteSum {
+public static Byte init() {
+return 0;
+}
+
+public static Byte add(Byte accumulator, Byte val) {
+return (byte) (accumulator + val);
+}
+
+public static Byte result(Byte accumulator) {
+return accumulator;
+}
+}
+
+public static class ShortSum {
+public static Short init() {
+return 0;
+}
+
+public static Short add(Short accumulator, Short val) {
+return (short) (accumulator + val);
+}
+
+public static Short result(Short accumulator) {
+return accumulator;
+}
+}
+
 public static class IntSum {
 public static Integer init() {
 return 0;
@@ -65,6 +93,34 @@ public class BuiltinAggregateFunctions {
 }
 }
 
+public static class LongSum {
+public static Long init() {
+return 0L;
+}
+
+public static Long add(Long accumulator, Long val) {
+return accumulator + val;
+}
+
+public static Long result(Long accumulator) {
+return accumulator;
+}
+}
+
+public static class FloatSum {
+public static Float init() {
+return 0.0f;
+}
+
+public static Float add(Float accumulator, Float val) {
+return accumulator + val;
+}
+
+public static Float result(Float accumulator) {
+return accumulator;
+}
+}
+
 public static class DoubleSum {
 public static Double init() {
 return 0.0;
@@ -166,7 +222,11 @@ public class BuiltinAggregateFunctions {
 
 static {
 TABLE.put("SUM", ImmutableList.of(
+TypeClass.of(float.class, FloatSum.class),
 TypeClass.of(double.class, DoubleSum.class),
+TypeClass.of(byte.class, ByteSum.class),
+TypeClass.of(short.class, ShortSum.class),
+TypeClass.of(long.class, LongSum.class),
 TypeClass.of(int.class, IntSum.class)));
 TABLE.put("AVG", ImmutableList.of(
 TypeClass.of(double.class, DoubleAvg.class),

1 2 3 4 5 6 7 8 >

1 - 100 of 791 matches

Mail list logo