[GitHub] storm issue #2880: STORM-3250: Closes Pull Requests unchanged in 2018

[danny0405]

Github user danny0405 commented on the issue:

https://github.com/apache/storm/pull/2880
  
+1


---

[GitHub] storm pull request #2881: STORM-3259: NUMA Support for Storm

[govind-menon]

Github user govind-menon commented on a diff in the pull request:

https://github.com/apache/storm/pull/2881#discussion_r227146299
  
--- Diff: storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java 
---
@@ -348,14 +350,27 @@ public void doExecutorHeartbeats() {
 if (null == executors) {
 stats = 
ClientStatsUtil.mkEmptyExecutorZkHbs(workerState.localExecutors);
 } else {
-stats = 
ClientStatsUtil.convertExecutorZkHbs(executors.stream().collect(Collectors
-   
   .toMap(IRunningExecutor::getExecutorId,
-   
  IRunningExecutor::renderStats)));
+stats = ClientStatsUtil.convertExecutorZkHbs(
+executors.stream().collect(
+Collectors.toMap(
+IRunningExecutor::getExecutorId,
+IRunningExecutor::renderStats
+)
+)
+);
 }
-Map zkHB = 
ClientStatsUtil.mkZkWorkerHb(workerState.topologyId, stats, 
workerState.uptime.upTime());
+
+Map zkHB = ClientStatsUtil.mkZkWorkerHb(
+workerState.topologyId, stats, workerState.uptime.upTime()
+);
+
 try {
+String assignmentId = workerState.assignmentId;
+if (this.numaId != null) {
+assignmentId += Constants.NUMA_ID_SEPARATOR + this.numaId;
--- End diff --

I've tested this out - the worker will launch correctly but wont' report 
back to nimbus and from the UI it will appear the workers haven't started yet.


---

[GitHub] storm issue #2882: STORM-3260: Add in support to print some state

[kishorvpatil]

Github user kishorvpatil commented on the issue:

https://github.com/apache/storm/pull/2882
  
Travis-ci build failures seems unrelated to the changes.


---

[GitHub] storm issue #2882: STORM-3260: Add in support to print some state

[kishorvpatil]

Github user kishorvpatil commented on the issue:

https://github.com/apache/storm/pull/2882
  
@revans2 , ok, looking at output, I thought it was trying to output JSON 
data. Thanks for the explanation.




---

[GitHub] storm issue #2881: STORM-3259: NUMA Support for Storm

[revans2]

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/2881
  
I have thought about it more and I am fine with NUMA support being at the 
supervisor level.  I think in the future we will need to move it so that Nimbus 
is aware of NUMA simply to be able to combat fragmentation more, but for now I 
am fine with it.


---

[GitHub] storm pull request #2888: [STORM-3263] Fix sorting by owner resources in the...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2888


---

[GitHub] storm issue #2882: STORM-3260: Add in support to print some state

[revans2]

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/2882
  
@kishorvpatil 

I didn't intend the data to be true JSON.  There are some things Thrift 
supports that JSON does not, like non-string map keys.  Storm uses those in our 
thrift data structures.

The goal is to be able to get something out that a human can look at and 
understand.

To me `not set` is less ambiguous than `"not set"`, `""`, or `{}` because 
it would never parse as JSON and it is actually not set. A `null` would 
probably be fine seeing how setting something to null in thrift is the same as 
not setting it.


---

[GitHub] storm issue #2892: Added in better docs for local mode testing.

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2892
  
+1


---

[GitHub] storm pull request #2892: Added in better docs for local mode testing.

[srdo]

Github user srdo commented on a diff in the pull request:

https://github.com/apache/storm/pull/2892#discussion_r227063418
  
--- Diff: docs/Local-mode.md ---
@@ -7,7 +7,9 @@ Local mode simulates a Storm cluster in process and is 
useful for developing and
 
 To run a topology in local mode you have two options.  The most common 
option is to run your topology with `storm local` instead of `storm jar`
 
-This will bring up a local simulated cluster and force all interactions 
with nimbus to go through the simulated cluster instead of going to a separate 
process.
+This will bring up a local simulated cluster and force all interactions 
with nimbus to go through the simulated cluster instead of going to a separate 
process. By default this will run the process for 20 seconds before tearing 
down the entire cluster.  You can override this by including a `--local-ttl` 
command line option which sets the number of seconds it should run for.
+
+### Programatic
--- End diff --

Nit: Two m's in programmatic


---

[GitHub] storm pull request #2882: STORM-3260: Add in support to print some state

[kishorvpatil]

Github user kishorvpatil commented on a diff in the pull request:

https://github.com/apache/storm/pull/2882#discussion_r227061728
  
--- Diff: storm-core/src/jvm/org/apache/storm/command/AdminCommands.java ---
@@ -104,6 +115,164 @@ public void printCliHelp(String command, PrintStream 
out) {
 }
 }
 
+/**
+ * Print value in a human readable format.
+ * @param value what to print.
+ * @return a human readable string
+ */
+public static String prettyPrint(TBase value) {
+StringBuilder builder = new StringBuilder();
+prettyPrint(value, 0, builder);
+return builder.toString();
+}
+
+private static void println(StringBuilder out, int depth, Object 
value) {
+for (int i = 0; i < depth; i++) {
+out.append("\t");
+}
+out.append(value);
+out.append("\n");
+}
+
+private static void prettyPrint(TBase value, int depth, StringBuilder 
out) {
+if (value == null) {
+println(out, depth,"null");
+return;
+}
+println(out, depth, "{");
+prettyPrintFields(value, depth + 1, out);
+println(out, depth, "}");
+}
+
+private static void prettyPrintFields(TBase value, int depth, 
StringBuilder out) {
+for (Map.Entry entry : 
FieldMetaData.getStructMetaDataMap(value.getClass()).entrySet()) {
+TFieldIdEnum key = entry.getKey();
+if (!value.isSet(key)) {
+println(out, depth, key.getFieldName() + ": not set");
+} else {
+Object o = value.getFieldValue(key);
+prettyPrintKeyValue(key.getFieldName(), o, depth, out);
+}
+}
+}
+
+private static String keyStr(String key) {
+return key == null ? "" : (key + ": ");
--- End diff --

should be probably "key" I guess


---

[GitHub] storm pull request #2882: STORM-3260: Add in support to print some state

[kishorvpatil]

Github user kishorvpatil commented on a diff in the pull request:

https://github.com/apache/storm/pull/2882#discussion_r227061218
  
--- Diff: storm-core/src/jvm/org/apache/storm/command/AdminCommands.java ---
@@ -104,6 +115,164 @@ public void printCliHelp(String command, PrintStream 
out) {
 }
 }
 
+/**
+ * Print value in a human readable format.
+ * @param value what to print.
+ * @return a human readable string
+ */
+public static String prettyPrint(TBase value) {
+StringBuilder builder = new StringBuilder();
+prettyPrint(value, 0, builder);
+return builder.toString();
+}
+
+private static void println(StringBuilder out, int depth, Object 
value) {
+for (int i = 0; i < depth; i++) {
+out.append("\t");
+}
+out.append(value);
+out.append("\n");
+}
+
+private static void prettyPrint(TBase value, int depth, StringBuilder 
out) {
+if (value == null) {
+println(out, depth,"null");
+return;
+}
+println(out, depth, "{");
+prettyPrintFields(value, depth + 1, out);
+println(out, depth, "}");
+}
+
+private static void prettyPrintFields(TBase value, int depth, 
StringBuilder out) {
+for (Map.Entry entry : 
FieldMetaData.getStructMetaDataMap(value.getClass()).entrySet()) {
+TFieldIdEnum key = entry.getKey();
+if (!value.isSet(key)) {
+println(out, depth, key.getFieldName() + ": not set");
+} else {
+Object o = value.getFieldValue(key);
+prettyPrintKeyValue(key.getFieldName(), o, depth, out);
+}
+}
+}
+
+private static String keyStr(String key) {
+return key == null ? "" : (key + ": ");
+}
+
+private static void prettyPrintKeyValue(String key, Object o, int 
depth, StringBuilder out) {
+//Special cases for storm...
+if ("json_conf".equals(key) && o instanceof String) {
+try {
+o = Utils.parseJson((String)o);
+} catch (Exception e) {
+LOG.error("Could not parse json_conf as JSON", e);
+}
+}
+if (o instanceof TBase) {
+println(out, depth, keyStr(key) + "{");
+prettyPrintFields((TBase) o, depth + 1, out);
+println(out, depth, "}");
+} else if (o instanceof Map) {
+println(out, depth, keyStr(key) + "{");
+for (Map.Entry entry : ((Map) 
o).entrySet()) {
+prettyPrintKeyValue(entry.getKey().toString(), 
entry.getValue(), depth + 1, out);
+}
+println(out, depth, "}");
+} else if (o instanceof Collection) {
+println(out, depth, keyStr(key) + "[");
+for (Object sub: (Collection)o) {
+prettyPrintKeyValue(null, sub, depth + 1, out);
+}
+println(out, depth, "]");
+} else if (o instanceof String) {
+println(out, depth, keyStr(key) + "\"" + o + "\"");
+} else {
+println(out, depth, keyStr(key) + o);
+}
+}
+
+private static class PrintTopo implements AdminCommand {
+
+@Override
+public void run(String[] args, Map conf, String 
command) throws Exception {
+for (String arg: args) {
+System.out.println(arg + ":");
--- End diff --

We should probably print quotes around `arg` to make it more compatible 
json like output


---

[GitHub] storm pull request #2882: STORM-3260: Add in support to print some state

[kishorvpatil]

Github user kishorvpatil commented on a diff in the pull request:

https://github.com/apache/storm/pull/2882#discussion_r227061504
  
--- Diff: storm-core/src/jvm/org/apache/storm/command/AdminCommands.java ---
@@ -104,6 +115,164 @@ public void printCliHelp(String command, PrintStream 
out) {
 }
 }
 
+/**
+ * Print value in a human readable format.
+ * @param value what to print.
+ * @return a human readable string
+ */
+public static String prettyPrint(TBase value) {
+StringBuilder builder = new StringBuilder();
+prettyPrint(value, 0, builder);
+return builder.toString();
+}
+
+private static void println(StringBuilder out, int depth, Object 
value) {
+for (int i = 0; i < depth; i++) {
+out.append("\t");
+}
+out.append(value);
+out.append("\n");
+}
+
+private static void prettyPrint(TBase value, int depth, StringBuilder 
out) {
+if (value == null) {
+println(out, depth,"null");
+return;
+}
+println(out, depth, "{");
+prettyPrintFields(value, depth + 1, out);
+println(out, depth, "}");
+}
+
+private static void prettyPrintFields(TBase value, int depth, 
StringBuilder out) {
+for (Map.Entry entry : 
FieldMetaData.getStructMetaDataMap(value.getClass()).entrySet()) {
+TFieldIdEnum key = entry.getKey();
+if (!value.isSet(key)) {
+println(out, depth, key.getFieldName() + ": not set");
--- End diff --

Simply make this empty "" String result or `{}`. not set is not exactly 
parsable json


---

[GitHub] storm pull request #2890: STORM-3268: Improve integration test stability

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2890


---

[GitHub] storm pull request #2887: STORM-3262 prevent falsely reporting leadership

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2887


---

[GitHub] storm pull request #2890: STORM-3268: Improve integration test stability

[d2r]

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/2890#discussion_r227036635
  
--- Diff: 
integration-test/src/main/java/org/apache/storm/st/topology/TestableTopology.java
 ---
@@ -17,14 +17,18 @@
 
 package org.apache.storm.st.topology;
 
+import java.util.concurrent.TimeUnit;
 import org.apache.storm.generated.StormTopology;
 
 public interface TestableTopology {
 String DUMMY_FIELD = "dummy";
-//Some tests rely on reading the worker log. If emits are too close 
together and too much is logged, the log might roll, breaking the test.
-int MIN_SLEEP_BETWEEN_EMITS_MS = 10;
-int MAX_SLEEP_BETWEEN_EMITS_MS = 100;
+int TIMEDATA_SLEEP_BETWEEN_EMITS_MS = 20;
+//Some tests rely on reading the worker log. If there are too many 
emits and too much is logged, the log might roll, breaking the test.
--- End diff --

OK, I assumed there was a way for the test itself to manipulate this. If 
not, then no worries.


---

[GitHub] storm pull request #2892: Added in better docs for local mode testing.

[revans2]

GitHub user revans2 opened a pull request:

https://github.com/apache/storm/pull/2892

Added in better docs for local mode testing.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/revans2/incubator-storm LOCAL_MODE_DOCS

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2892.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2892






---

[GitHub] storm pull request #1942: STORM-2358: Update storm hdfs spout to remove spec...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1942


---

[GitHub] storm pull request #2890: STORM-3268: Improve integration test stability

[srdo]

Github user srdo commented on a diff in the pull request:

https://github.com/apache/storm/pull/2890#discussion_r227033367
  
--- Diff: 
integration-test/src/main/java/org/apache/storm/st/topology/TestableTopology.java
 ---
@@ -17,14 +17,18 @@
 
 package org.apache.storm.st.topology;
 
+import java.util.concurrent.TimeUnit;
 import org.apache.storm.generated.StormTopology;
 
 public interface TestableTopology {
 String DUMMY_FIELD = "dummy";
-//Some tests rely on reading the worker log. If emits are too close 
together and too much is logged, the log might roll, breaking the test.
-int MIN_SLEEP_BETWEEN_EMITS_MS = 10;
-int MAX_SLEEP_BETWEEN_EMITS_MS = 100;
+int TIMEDATA_SLEEP_BETWEEN_EMITS_MS = 20;
+//Some tests rely on reading the worker log. If there are too many 
emits and too much is logged, the log might roll, breaking the test.
--- End diff --

Yes, we could disable log rolling by customizing the cluster's 
`log4j2.xml`. I wanted to avoid doing that because it means you can't 
(reliably) run the integration test against your own cluster without having to 
edit `log4j2.xml` as well.


---

[GitHub] storm pull request #1964: STORM-2380 worker.childopts with whitespace inside...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1964


---

[GitHub] storm pull request #1521: STORM-1901: Avro Integration for Storm-Kafka

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1521


---

[GitHub] storm pull request #1971: STORM-2384 Add log statements when spout skips cal...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1971


---

[GitHub] storm pull request #1767: STORM-2194: Report error and die, not report error...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1767


---

[GitHub] storm pull request #2108: STORM-2508:storm-solr enhancement: update solrj to...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2108


---

[GitHub] storm pull request #1575: STORM-1600: Do not report exceptions after jvm shu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1575


---

[GitHub] storm pull request #1967: STORM-2290: Upgrade zookeeper to get some critical...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1967


---

[GitHub] storm pull request #1965: STORM-2380 worker.childopts with whitespace inside...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1965


---

[GitHub] storm pull request #1918: STORM-2339: Python code format cleanup in storm.py

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1918


---

[GitHub] storm pull request #1376: STORM-1736. Change KafkaTestBroker.buildKafkaConfi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1376


---

[GitHub] storm pull request #1957: STORM-2373: HDFS Spout should support multiple ign...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1957


---

[GitHub] storm pull request #2377: [STORM-2780] MetricsConsumer record unnecessary ti...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2377


---

[GitHub] storm pull request #1584: STORM-1992 Added artifacts to make storm.js into a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1584


---

[GitHub] storm pull request #2880: STORM-3250: Closes Pull Requests unchanged in 2018

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2880


---

[GitHub] storm pull request #1874: STORM-2286 Storm Rebalance command should support ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1874


---

[GitHub] storm pull request #1550: [STORM-1957] Support Storm JDBC batch insert

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1550


---

[GitHub] storm pull request #2206: STORM-2625: reduce uncommitted count when kafka co...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2206


---

[GitHub] storm pull request #1937: STORM-2355: Storm-HDFS: inotify support

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1937


---

[GitHub] storm pull request #1515: [STORM-1929] Check when create topology

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1515


---

[GitHub] storm pull request #1525: [STORM-1931] Share mapper and selector in Storm-Ka...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1525


---

[GitHub] storm pull request #1410: STORM-1778: scheme extension framework for KafkaSo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1410


---

[GitHub] storm pull request #1427: [STORM-1701] Port PR 1370 to 1.x-branch (Git rid o...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1427


---

[GitHub] storm pull request #1591: STORM-1038: Upgrade netty to 4.x in 1.x-branch

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1591


---

[GitHub] storm pull request #1483: [STORM-1875] : Separate Jedis/JedisCluster Config

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1483


---

[GitHub] storm pull request #2142: MINOR: Fix pacemaker_state_factory.clj not compile...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2142


---

[GitHub] storm pull request #1998: Eventhub2

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1998


---

[GitHub] storm pull request #1370: [STORM-1707] Remove two minute timeout after worke...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1370


---

[GitHub] storm pull request #1277: STORM-1129: Update ui to use topology name

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1277


---

[GitHub] storm pull request #1391: (STORM-1674) Idle KafkaSpout consumes more bandwid...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1391


---

[GitHub] storm pull request #1288: STORM-971: KafkaSpout - Improve "_lostMessageCount...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1288


---

[GitHub] storm pull request #1406: [STORM-433] [WIP] Executor queue backlog metric

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1406


---

[GitHub] storm pull request #2322: Introduce config params to use latest EH client, c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2322


---

[GitHub] storm pull request #1760: Add topology stream-awareness to storm-redis

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1760


---

[GitHub] storm pull request #1799: Fixing a bug related to following exception

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1799


---

[GitHub] storm pull request #1272: [STORM-1662] Reduce map lookups in send_to_eventlo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1272


---

[GitHub] storm pull request #1468: STORM-1885. python script for squashing and mergin...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1468


---

[GitHub] storm pull request #1296: STORM-1675 - Allow submitting multiple jars from t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1296


---

[GitHub] storm pull request #1328: [STORM-979][storm-elasticsearch] BaseQueryFunction...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1328


---

[GitHub] storm pull request #1178: [STORM-1469] Remove SimpleTransportPlugin and conf...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1178


---

[GitHub] storm pull request #1342: [STORM-1712] make storage plugin for transactional...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1342


---

[GitHub] storm pull request #1544: ConcurrentModificationExceptions in WindowedBoltEx...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1544


---

[GitHub] storm pull request #1046: [STORM-1501] launch worker process exception will ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1046


---

[GitHub] storm pull request #1785: [STORM-2201] Add dynamic scheduler configuration l...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1785


---

[GitHub] storm pull request #1258: add field argument to newDRPCStream

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1258


---

[GitHub] storm pull request #1078: [STORM-1368] change heapdump file permissions so t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1078


---

[GitHub] storm pull request #692: STORM-997: Add proxy user functionality for storm h...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/692


---

[GitHub] storm pull request #1395: storm-1726: use Put#addColumn to replace the depre...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1395


---

[GitHub] storm pull request #1443: Log.warn if found a message in kafka topic larger ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1443


---

[GitHub] storm pull request #1146: [STORM-1065] kafka-partition can not find leader i...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1146


---

[GitHub] storm pull request #1051: [STORM-1506] It's better to be Integer about port ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1051


---

[GitHub] storm pull request #986: STORM-822 Implement Kafka 0.9 consumer API

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/986


---

[GitHub] storm pull request #1213: Rename README.markdown to README.md.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1213


---

[GitHub] storm pull request #1041: make the txid continuous and bug fixed

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1041


---

[GitHub] storm pull request #1164: [storm-1456]: Cannot check for ack or fail in Fixe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1164


---

[GitHub] storm pull request #730: STORM-1039: Remove commons-codec shading, commons-c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/730


---

[GitHub] storm pull request #662: STORM-904: Move bin/storm command line to java.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/662


---

[GitHub] storm pull request #728: [STORM-1038] Upgraded netty to 4.x

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/728


---

[GitHub] storm pull request #1399: update readme.md

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1399


---

[GitHub] storm pull request #1353: storm-1726: use Put#addColumn to replace the depre...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1353


---

[GitHub] storm pull request #1611: [storm-2022]fix test case

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1611


---

[GitHub] storm pull request #724: 修改jdbcClient

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/724


---

[GitHub] storm pull request #1259: modify delegateIndex to protected

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1259


---

[GitHub] storm pull request #1019: Document that persistentAggregate() does not work ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1019


---

[GitHub] storm pull request #753: [STORM-1057] Add throughput metrics to spouts/bolts...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/753


---

[GitHub] storm pull request #648: [STORM-440] completed exposing the drpcclient witho...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/648


---

[GitHub] storm pull request #705: [STORM-1015] Allow Kafka offsets to be saved using ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/705


---

[GitHub] storm pull request #1225: Hbasemapstate

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1225


---

[GitHub] storm pull request #922: STORM-1345: UpdateTopology API and implementation.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/922


---

[GitHub] storm pull request #1600: Adding F#/FsShelter reference

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1600


---

[GitHub] storm pull request #668: STORM-828 HdfsBolt takes a lot of configuration, ne...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/668


---

[GitHub] storm pull request #640: STORM-947: replace all `backtype.storm.scheduler.Ex...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/640


---

[GitHub] storm issue #2880: STORM-3250: Closes Pull Requests unchanged in 2018

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2880
  
+1, thanks for handling this.


---

[GitHub] storm pull request #854: STORM-1129: Use topology name instead of id in UI c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/854


---

[GitHub] storm pull request #1043: Storm 1226

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1043


---

[GitHub] storm pull request #1268: fallbacks to startoffset time if offset is out of ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1268


---

[GitHub] storm pull request #1040: Moved adding the hbase configuration in front of t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1040


---

[GitHub] storm pull request #2891: STORM-3269: Update version of httpclient, and fix ...

[revans2]

GitHub user revans2 opened a pull request:

https://github.com/apache/storm/pull/2891

STORM-3269: Update version of httpclient, and fix version dep issue



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/revans2/incubator-storm STORM-3269

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2891.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2891


commit 4bba9c4e8ba962dd6f35c7fd25cfa466050f5607
Author: Robert (Bobby) Evans 
Date:   2018-10-22T15:44:18Z

STORM-3269: Update version of httpclient, and fix version dep issue




---

[GitHub] storm pull request #2890: STORM-3268: Improve integration test stability

[d2r]

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/2890#discussion_r226989550
  
--- Diff: 
integration-test/src/main/java/org/apache/storm/st/topology/TestableTopology.java
 ---
@@ -17,14 +17,18 @@
 
 package org.apache.storm.st.topology;
 
+import java.util.concurrent.TimeUnit;
 import org.apache.storm.generated.StormTopology;
 
 public interface TestableTopology {
 String DUMMY_FIELD = "dummy";
-//Some tests rely on reading the worker log. If emits are too close 
together and too much is logged, the log might roll, breaking the test.
-int MIN_SLEEP_BETWEEN_EMITS_MS = 10;
-int MAX_SLEEP_BETWEEN_EMITS_MS = 100;
+int TIMEDATA_SLEEP_BETWEEN_EMITS_MS = 20;
+//Some tests rely on reading the worker log. If there are too many 
emits and too much is logged, the log might roll, breaking the test.
--- End diff --

I wonder if we could instead configure the logs to roll less often, since 
that is the real issue?


---

[GitHub] storm issue #2890: STORM-3268: Improve integration test stability

[d2r]

Github user d2r commented on the issue:

https://github.com/apache/storm/pull/2890
  
Depends on #2889 



---

Re: [VOTE] Release Apache Storm 2.0.0 (rc3)

[Bobby Evans]

Julien,

I have put up pull requests for the docs and for fixing some of the issues
with LocalCluster that you found.

https://github.com/apache/storm/pull/2891

https://github.com/apache/storm/pull/2892

The VersionInfo change is a blocker and we should fix it before releasing
(Sorry Taylor).

For the other stuff if you find more issues we can move it to a different
thread and work through them.

Thanks,

Bobby

On Mon, Oct 22, 2018 at 9:23 AM Bobby Evans  wrote:

> I'll look at upgrading that version of http client too.
>
> On Mon, Oct 22, 2018 at 9:15 AM Julien Nioche <
> lists.digitalpeb...@gmail.com> wrote:
>
>> Hi,
>>
>> I've looked into it a bit more and found that SC had a dependency on
>> storm-core and not storm-client; I've fixed this in 40612a3...
>> <
>> https://github.com/DigitalPebble/storm-crawler/commit/40612a3588d66e1d410a70b1c7e5db58d5c2ba4d
>> >
>> however
>> this doesn't affect the issues I had last week.
>>
>> *httpclient dependency conflict*
>> As seen last week, this is not shaded by Storm and the version used (4.3.3
>> <
>> https://github.com/apache/storm/blob/ce984cd31a16e7fe4b983659005f1f7648455404/pom.xml#L266
>> >)
>> is quite old. Even within Storm, the Storm-SOLR module uses a more recent
>> one (4.5
>> <
>> https://github.com/apache/storm/blob/master/external/storm-solr/pom.xml#L64
>> >).
>> StormCrawler needs at least 4.5.5
>> <
>> https://github.com/DigitalPebble/storm-crawler/blob/master/core/pom.xml#L26
>> >.
>> I expect other Storm users would use *httpclient* and have a similar
>> problem. Unless I am missing something, I can see the following solutions
>> sorted by how convenient they are to me as a user:
>>
>>1. the dependency is shaded by Storm
>>2. the dependency is upgraded to 4.5.5 by Storm
>>3. the dependency is shaded by StormCrawler
>>
>> Obviously, I'd rather not have to deal with (3) and anyone using
>> httpclient with Storm would have to do the same.
>>
>> Note: I can get my topology to work by specifying a protocol
>> implementation
>> based on OKHttp
>> *  http.protocol.implementation:
>> "com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol"*
>> *  https.protocol.implementation:
>> "com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol"*
>>
>> *LocalCluster*
>> Since removing the dependency on storm-core, I can't use LocalCluster
>> directly. I'll create a separate branch on my test repo to try to
>> reproduce
>> the issue.
>>
>> *Documentation for Local mode*
>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/Local-mode.html
>> does not mention *--local-ttl *would be good to document it and indicate
>> what the default value is otherwise users might wonder why their
>> topologies
>> run for 20 secs only.  Personally, I'd rather be able to have a default
>> behaviour where the topology runs forever or at least be able to
>> deactivate
>> the TTL completely by setting it to -1.
>>
>> *ConfigurableTopology*
>> I am getting a different behavior between the original
>> ConfigurableTopology from
>> StormCrawler
>> <
>> https://github.com/DigitalPebble/storm-crawler/blob/master/core/src/main/java/com/digitalpebble/stormcrawler/ConfigurableTopology.java
>> >
>> and when I extend the one in Storm
>> <
>> https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/topology/ConfigurableTopology.java
>> >;
>> with the latter, any configuration found in the conf files passed in args
>> to the command line are added to the default values I provide instead of
>> replacing them. I'll investigate that further and open an issue if I find
>> a
>> bug.
>>
>> *Distributed mode*
>> I managed to launch the various services and run my test topology in
>> remote
>> mode (by changing the protocol implementation as explained above)
>>
>> *Flux*
>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/flux.html tells me to run
>>
>> storm jar myTopology-0.1.0-SNAPSHOT.jar org.apache.storm.flux.Flux
>> --local my_config.yaml
>>
>>
>>
>> *apache-storm-2.0.0/bin/storm jar target/2-1.0-SNAPSHOT.jar
>> org.apache.storm.flux.Flux --local crawler.flux*
>>
>> but am getting
>>
>> *15:07:26.206 [main] ERROR o.a.s.f.Flux - To run in local mode run with
>> 'storm local' instead of 'storm jar'*
>>
>> *so *I tried both
>>
>> apache-storm-2.0.0/bin/storm local target/2-1.0-SNAPSHOT.jar
>> org.apache.storm.flux.Flux --local crawler.flux
>>
>> and
>>
>> *apache-storm-2.0.0/bin/storm local target/2-1.0-SNAPSHOT.jar
>> org.apache.storm.flux.Flux crawler.flux*
>> but in both cases I'm getting
>>
>> *15:12:06.784 [main] ERROR o.a.s.f.Flux - To run in local mode run with
>> 'storm local' instead of 'storm jar'*
>> *15:12:06.784 [main] INFO  o.a.s.LocalCluster - *
>>
>> * RUNNING LOCAL CLUSTER for 20 seconds.*
>>
>> and nothing happens, the topology just dies after 20secs without feching
>> any URLs.
>>
>> I haven't tried Flux in distributed mode yet.
>>
>> Thanks!
>>
>> Julien
>>
>> PS: my test topology is in https://github.com/DigitalPebble/storm2
>>
>>
>>
>>

Re: [VOTE] Release Apache Storm 2.0.0 (rc3)

[Bobby Evans]

Yup that is a bug.  I'll file a JIRA and work on a fix for it ASAP.

On Fri, Oct 19, 2018 at 1:33 PM Julien Nioche 
wrote:

> Hi Bobby
>
> The dependency issue happens when I have only storm-client as a dependency
> and not server.
>
> When trying to run it from Eclipse I had to add server to the pom, as
> expected but also client as I was getting
>
> 19:22:13.044 [main] ERROR o.a.s.u.VersionInfo - Could not load
> storm-core-version-info.properties
> java.io.IOException: Resource not found
> at
>
> org.apache.storm.utils.VersionInfo$VersionInfoImpl.(VersionInfo.java:53)
> [storm-client-2.0.0.jar:2.0.0]
> at org.apache.storm.utils.VersionInfo.(VersionInfo.java:41)
> [storm-client-2.0.0.jar:2.0.0]
> at org.apache.storm.daemon.nimbus.Nimbus.(Nimbus.java:281)
> [storm-server-2.0.0.jar:2.0.0]
> at org.apache.storm.LocalCluster.(LocalCluster.java:235)
> [storm-server-2.0.0.jar:2.0.0]
> at org.apache.storm.LocalCluster.(LocalCluster.java:156)
> [storm-server-2.0.0.jar:2.0.0]
> at
>
> com.digitalpebble.stormcrawler.ConfigurableTopology.submit(ConfigurableTopology.java:74)
> [classes/:?]
> at com.dipe.sc.CrawlTopology.run(CrawlTopology.java:80) [classes/:?]
> at
>
> com.digitalpebble.stormcrawler.ConfigurableTopology.start(ConfigurableTopology.java:49)
> [classes/:?]
> at com.dipe.sc.CrawlTopology.main(CrawlTopology.java:39) [classes/:?]
>
> I've put the code in https://github.com/DigitalPebble/storm2  if you want
> to have a look. You'll need to compile the branch 2.x of SC first
> https://github.com/DigitalPebble/storm-crawler/tree/2.x
>
> To reproduce the ZK issue, open the project in Eclipse and run the
> CrawlTopology class with "-local -conf crawler-conf.yaml" in arguments.
>
> For the dependency problem, mvn clean package followed by
> /data/apache-storm-2.0.0/bin/storm local target/2-1.0-SNAPSHOT.jar
> com.dipe.sc.CrawlTopology -conf crawler-conf.yaml
> should give java.lang.NoSuchMethodError:
>
> org.apache.http.impl.client.HttpClientBuilder.setConnectionManagerShared(Z)Lorg/apache/http/impl/client/HttpClientBuilder;
>
> Thanks
>
> Julien
>
> On Fri, 19 Oct 2018 at 17:26, Bobby Evans  wrote:
>
> > Sorry I should clarify a bit.
> >
> > `storm local` will run things in local mode, but the classpath will
> include
> > things that are not shaded.
> >
> > This is also true for trying to run tests from eclipse.  LocalCluster is
> a
> > part of storm-server so you will need to pull that in just for testing.
> > storm-client is what you want to depend on for the majority of your
> > topology.
> >
> > The ZK issue is new to me  We have done a lot in local mode and not seen
> > that as an issue.  If you can help me reproduce it I am happy to try and
> > debug it to see what is happening.
> >
> > Thanks,
> >
> > Bobby
> >
> > On Fri, Oct 19, 2018 at 11:21 AM Bobby Evans  wrote:
> >
> > > It is shaded in storm 2.x, but we split the classpath up, so what you
> > want
> > > to depend on is storm-client only.  I see you are pulling in storm-core
> > and
> > > a few other things that are not shaded, because they are only used by
> the
> > > daemons, not the clients.
> > >
> > > On Fri, Oct 19, 2018 at 10:55 AM Julien Nioche <
> > > lists.digitalpeb...@gmail.com> wrote:
> > >
> > >> Sorry, hit Return too quickly
> > >>
> > >> I am testing Storm 2.0.0 with StormCrawler, not very successfully. One
> > >> immediate issue is that I am getting a version conflict on httpclient
> as
> > >> the version set by Storm is older than the one I need.
> > >>
> > >> java.lang.NoSuchMethodError:
> > >>
> > >>
> >
> org.apache.http.impl.client.HttpClientBuilder.setConnectionManagerShared(Z)Lorg/apache/http/impl/client/HttpClientBuilder;
> > >> at
> > >>
> > >>
> >
> com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol.configure(HttpProtocol.java:141)
> > >> ~[2-1.0-SNAPSHOT.jar:?]
> > >> at
> > >>
> > >>
> >
> com.digitalpebble.stormcrawler.protocol.ProtocolFactory.(ProtocolFactory.java:69)
> > >> ~[2-1.0-SNAPSHOT.jar:?]
> > >> at
> > >>
> > >>
> >
> com.digitalpebble.stormcrawler.bolt.FetcherBolt.prepare(FetcherBolt.java:760)
> > >> ~[2-1.0-SNAPSHOT.jar:?]
> > >> at
> > org.apache.storm.executor.bolt.BoltExecutor.init(BoltExecutor.java:144)
> > >> ~[storm-client-2.0.0.jar:2.0.0]
> > >> at
> > org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:154)
> > >> ~[storm-client-2.0.0.jar:2.0.0]
> > >> at
> > org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:58)
> > >> ~[storm-client-2.0.0.jar:2.0.0]
> > >> at org.apache.storm.utils.Utils$1.run(Utils.java:353)
> > >> [storm-client-2.0.0.jar:2.0.0]
> > >> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> > >>
> > >> Here is the classpath when calling *storm local *
> > >>
> > >> *16:38:03.445 [main] INFO  o.a.s.s.o.a.z.ZooKeeper - Client
> > >>
> > >>
> >
> 

Re: [VOTE] Release Apache Storm 2.0.0 (rc3)

[Julien Nioche]

Hi,

I've looked into it a bit more and found that SC had a dependency on
storm-core and not storm-client; I've fixed this in 40612a3...

however
this doesn't affect the issues I had last week.

*httpclient dependency conflict*
As seen last week, this is not shaded by Storm and the version used (4.3.3
)
is quite old. Even within Storm, the Storm-SOLR module uses a more recent
one (4.5
).
StormCrawler needs at least 4.5.5
.
I expect other Storm users would use *httpclient* and have a similar
problem. Unless I am missing something, I can see the following solutions
sorted by how convenient they are to me as a user:

   1. the dependency is shaded by Storm
   2. the dependency is upgraded to 4.5.5 by Storm
   3. the dependency is shaded by StormCrawler

Obviously, I'd rather not have to deal with (3) and anyone using
httpclient with Storm would have to do the same.

Note: I can get my topology to work by specifying a protocol implementation
based on OKHttp
*  http.protocol.implementation:
"com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol"*
*  https.protocol.implementation:
"com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol"*

*LocalCluster*
Since removing the dependency on storm-core, I can't use LocalCluster
directly. I'll create a separate branch on my test repo to try to reproduce
the issue.

*Documentation for Local mode*
http://storm.apache.org/releases/2.0.0-SNAPSHOT/Local-mode.html
does not mention *--local-ttl *would be good to document it and indicate
what the default value is otherwise users might wonder why their topologies
run for 20 secs only.  Personally, I'd rather be able to have a default
behaviour where the topology runs forever or at least be able to deactivate
the TTL completely by setting it to -1.

*ConfigurableTopology*
I am getting a different behavior between the original
ConfigurableTopology from
StormCrawler

and when I extend the one in Storm
;
with the latter, any configuration found in the conf files passed in args
to the command line are added to the default values I provide instead of
replacing them. I'll investigate that further and open an issue if I find a
bug.

*Distributed mode*
I managed to launch the various services and run my test topology in remote
mode (by changing the protocol implementation as explained above)

*Flux*
http://storm.apache.org/releases/2.0.0-SNAPSHOT/flux.html tells me to run

storm jar myTopology-0.1.0-SNAPSHOT.jar org.apache.storm.flux.Flux
--local my_config.yaml



*apache-storm-2.0.0/bin/storm jar target/2-1.0-SNAPSHOT.jar
org.apache.storm.flux.Flux --local crawler.flux*

but am getting

*15:07:26.206 [main] ERROR o.a.s.f.Flux - To run in local mode run with
'storm local' instead of 'storm jar'*

*so *I tried both

apache-storm-2.0.0/bin/storm local target/2-1.0-SNAPSHOT.jar
org.apache.storm.flux.Flux --local crawler.flux

and

*apache-storm-2.0.0/bin/storm local target/2-1.0-SNAPSHOT.jar
org.apache.storm.flux.Flux crawler.flux*
but in both cases I'm getting

*15:12:06.784 [main] ERROR o.a.s.f.Flux - To run in local mode run with
'storm local' instead of 'storm jar'*
*15:12:06.784 [main] INFO  o.a.s.LocalCluster - *

* RUNNING LOCAL CLUSTER for 20 seconds.*

and nothing happens, the topology just dies after 20secs without feching
any URLs.

I haven't tried Flux in distributed mode yet.

Thanks!

Julien

PS: my test topology is in https://github.com/DigitalPebble/storm2








On Fri, 19 Oct 2018 at 19:32, Julien Nioche 
wrote:

> Hi Bobby
>
> The dependency issue happens when I have only storm-client as a dependency
> and not server.
>
> When trying to run it from Eclipse I had to add server to the pom, as
> expected but also client as I was getting
>
> 19:22:13.044 [main] ERROR o.a.s.u.VersionInfo - Could not load
> storm-core-version-info.properties
> java.io.IOException: Resource not found
> at
> org.apache.storm.utils.VersionInfo$VersionInfoImpl.(VersionInfo.java:53)
> [storm-client-2.0.0.jar:2.0.0]
> at org.apache.storm.utils.VersionInfo.(VersionInfo.java:41)
> [storm-client-2.0.0.jar:2.0.0]
> at org.apache.storm.daemon.nimbus.Nimbus.(Nimbus.java:281)
> [storm-server-2.0.0.jar:2.0.0]
> at org.apache.storm.LocalCluster.(LocalCluster.java:235)
> [storm-server-2.0.0.jar:2.0.0]
> at org.apache.storm.LocalCluster.(LocalCluster.java:156)
> [storm-server-2.0.0.jar:2.0.0]
> at
>