[GitHub] nwangtw commented on a change in pull request #2891: Refactor StatefulStorage API
nwangtw commented on a change in pull request #2891: Refactor StatefulStorage
API
URL: https://github.com/apache/incubator-heron/pull/2891#discussion_r197624374
##
File path: heron/proto/ckptmgr.proto
##
@@ -171,7 +175,23 @@ message CleanStatefulCheckpointResponse {
repeated string cleaned_checkpoint_ids = 2;
}
-// stmgr -> ckptmgr messages
+/*
+ * stmgr -> ckptmgr messages
+ */
+
+// This message encapsulates the info associated with
+// state of an instance/partition
+message InstanceStateCheckpointPartition {
Review comment:
kk. I was feeling InstanceStateCheckpointPartition is more correct, but it
can be renamed later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nwangtw commented on a change in pull request #2891: Refactor StatefulStorage API
nwangtw commented on a change in pull request #2891: Refactor StatefulStorage
API
URL: https://github.com/apache/incubator-heron/pull/2891#discussion_r197624374
##
File path: heron/proto/ckptmgr.proto
##
@@ -171,7 +175,23 @@ message CleanStatefulCheckpointResponse {
repeated string cleaned_checkpoint_ids = 2;
}
-// stmgr -> ckptmgr messages
+/*
+ * stmgr -> ckptmgr messages
+ */
+
+// This message encapsulates the info associated with
+// state of an instance/partition
+message InstanceStateCheckpointPartition {
Review comment:
kk. let me do it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kramasamy commented on issue #2891: Refactor StatefulStorage API
kramasamy commented on issue #2891: Refactor StatefulStorage API URL: https://github.com/apache/incubator-heron/pull/2891#issuecomment-399697948 @nwangtw - can you resolve conflicts if this is good to go. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kramasamy commented on a change in pull request #2929: Make TopologyBuilder less permissive
kramasamy commented on a change in pull request #2929: Make TopologyBuilder
less permissive
URL: https://github.com/apache/incubator-heron/pull/2929#discussion_r197618933
##
File path: heronpy/api/topology.py
##
@@ -194,24 +194,25 @@ def get_heron_options_from_env():
Currently supports the following options natively:
-- `cmdline.topologydefn.tmpdirectory`: the directory to which this
+- `cmdline.topologydefn.tmpdirectory`: (required) the directory to which
this
topology's defn file is written
-- `cmdline.topology.initial.state`: the initial state of the topology
-- `cmdline.topology.name`: topology name on deployment
+- `cmdline.topology.initial.state`: (default: "RUNNING") the initial state
of the topology
+- `cmdline.topology.name`: (default: class name) topology name on
deployment
Returns: map mapping from key to value
"""
-heron_options_raw = os.environ.get("HERON_OPTIONS", None)
+heron_options_raw = os.environ.get("HERON_OPTIONS")
if heron_options_raw is None:
raise RuntimeError("HERON_OPTIONS environment variable not found")
-ret = {}
-heron_opt_list = heron_options_raw.replace("", " ").split(',')
-for opt_raw in heron_opt_list:
- opt = opt_raw.split("=")
- if len(opt) == 2:
-ret[opt[0]] = opt[1]
-return ret
+options = {}
+for option_line in heron_options_raw.replace("", " ").split(','):
Review comment:
It might be worth trying and if it works - we can use json dumps and load.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-heron] branch master updated: make EvitionContext extend Serializable (#2935)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 805a401 make EvitionContext extend Serializable (#2935)
805a401 is described below
commit 805a401a14d5f211679262cb226176e0742340ea
Author: Neng Lu
AuthorDate: Sat Jun 23 11:00:30 2018 -0700
make EvitionContext extend Serializable (#2935)
* make EvitionContext extend Serializable
* fix typo
---
.../api/StatefulTumblingWindowTopology.java| 181 +
.../heron/api/windowing/EvictionContext.java | 4 +-
2 files changed, 184 insertions(+), 1 deletion(-)
diff --git
a/examples/src/java/org/apache/heron/examples/api/StatefulTumblingWindowTopology.java
b/examples/src/java/org/apache/heron/examples/api/StatefulTumblingWindowTopology.java
new file mode 100644
index 000..b39e3ff
--- /dev/null
+++
b/examples/src/java/org/apache/heron/examples/api/StatefulTumblingWindowTopology.java
@@ -0,0 +1,181 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.heron.examples.api;
+
+import java.time.Duration;
+import java.util.Map;
+import java.util.Random;
+import java.util.logging.Logger;
+
+import org.apache.heron.api.Config;
+import org.apache.heron.api.HeronSubmitter;
+import org.apache.heron.api.bolt.BaseStatefulWindowedBolt;
+import org.apache.heron.api.bolt.OutputCollector;
+import org.apache.heron.api.spout.BaseRichSpout;
+import org.apache.heron.api.spout.SpoutOutputCollector;
+import org.apache.heron.api.state.State;
+import org.apache.heron.api.topology.IStatefulComponent;
+import org.apache.heron.api.topology.OutputFieldsDeclarer;
+import org.apache.heron.api.topology.TopologyBuilder;
+import org.apache.heron.api.topology.TopologyContext;
+import org.apache.heron.api.tuple.Fields;
+import org.apache.heron.api.tuple.Tuple;
+import org.apache.heron.api.tuple.Values;
+import org.apache.heron.api.utils.Utils;
+import org.apache.heron.api.windowing.TupleWindow;
+import org.apache.heron.common.basics.ByteAmount;
+import org.apache.heron.examples.api.bolt.PrinterBolt;
+
+/**
+ * A sample topology that demonstrates the usage of {@link
org.apache.heron.api.bolt.IStatefulWindowedBolt}
+ * to calculate tumbling window sum. Topology also demonstrates how stateful
window processing is done
+ * in conjunction with effectively once guarantees
+ */
+public final class StatefulTumblingWindowTopology {
+
+ private static final Logger LOG =
Logger.getLogger(StatefulSlidingWindowTopology.class.getName());
+
+ private StatefulTumblingWindowTopology() {
+ }
+
+ private static class WindowSumBolt extends BaseStatefulWindowedBolt {
+private static final long serialVersionUID = -539382497249834244L;
+private State state;
+private long sum;
+
+private OutputCollector collector;
+
+@Override
+@SuppressWarnings("HiddenField")
+public void prepare(Map topoConf, TopologyContext context,
+OutputCollector collector) {
+ this.collector = collector;
+}
+
+@Override
+@SuppressWarnings("HiddenField")
+public void initState(State state) {
+ this.state = state;
+ sum = state.getOrDefault("sum", 0L);
+}
+
+@Override
+public void execute(TupleWindow inputWindow) {
+ for (Tuple tuple : inputWindow.get()) {
+sum += tuple.getLongByField("value");
+ }
+ collector.emit(new Values(sum));
+}
+
+@Override
+public void declareOutputFields(OutputFieldsDeclarer declarer) {
+ declarer.declare(new Fields("sum"));
+}
+
+@Override
+public void preSave(String checkpointId) {
+ state.put("sum", sum);
+}
+ }
+
+ public static class IntegerSpout extends BaseRichSpout
+ implements IStatefulComponent {
+private static final Logger LOG =
Logger.getLogger(IntegerSpout.class.getName());
+private static final long serialVersionUID = 5454291010750852782L;
+private SpoutOutputCollector collector;
+private Random rand;
+private long msgId;
+private State state;
+
+@Override
+
[GitHub] Code0x58 edited a comment on issue #2924: Py3 heronpy
Code0x58 edited a comment on issue #2924: Py3 heronpy URL: https://github.com/apache/incubator-heron/pull/2924#issuecomment-399665205 Once this does work locally, it'll need support on the schedulers, which I think is to update the executor image to include Python3.6. I will look into making `.pex` files pin to more generic interpreters (`python3` vs. `python3.6`), otherwise it will be a pain where you have to rebuild your PEXs every time the executor image's python 3 minor version increases - even if it is infrequent. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Code0x58 edited a comment on issue #2924: Py3 heronpy
Code0x58 edited a comment on issue #2924: Py3 heronpy URL: https://github.com/apache/incubator-heron/pull/2924#issuecomment-399665205 Once this does work locally, it'll need support on the schedulers, which I think is to update the executor image to include Python3.6. I will look into making `.pex` files pin to more generic interpreters (`python3` vs. `python3.6`), as otherwise that will be a pain needing you have to rebuild your PEXs every time the executor image's python 3 minor version increases - even if it is infrequent. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Code0x58 commented on issue #2924: Py3 heronpy
Code0x58 commented on issue #2924: Py3 heronpy URL: https://github.com/apache/incubator-heron/pull/2924#issuecomment-399665205 Once this does work locally, it'll need support on the schedulers, which I think is to update the executor image to include Python3. I will look into making `.pex` files pin to more generic interpreters (`python3` vs. `python3.6`), as otherwise that will be a pain causing you have to rebuild your PEXs every time the executor image's python 3 minor version increases. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Code0x58 commented on issue #2928: Clean up website gen + website python docs
Code0x58 commented on issue #2928: Clean up website gen + website python docs URL: https://github.com/apache/incubator-heron/pull/2928#issuecomment-399661032 This is failing for the same reason as master now, but should be good to merge This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yaoliclshlmch opened a new pull request #2936: Yaoli/add integration topology test
yaoliclshlmch opened a new pull request #2936: Yaoli/add integration topology test URL: https://github.com/apache/incubator-heron/pull/2936 This test aims to test the launched topology structure correctness. The previous integration tests are focusing on the data correctness by fetching the real output data and comparing it with the expected output results, which is functional black box testing. For the topology structure test, white box testing should be used which means the test platform is aware of the internal topology structures. Topology structure test here includes the following aspects: 1. Topology graph a. components: spout type, bolt type and their parallelism; b. connectivity: data stream and connection correctness; 2. Instance states (only in stateful processing): loaded states in target rollback checkpoint. For now, we focus on the implementation of topology component and connectivity, and the instance state test will be realized in the future. Test cases: 1. Topology submission 3. Spouts/bolts parallelism scale up 4. Spouts/bolts parallelism scale down 5. Spouts/bolts parallelism scale up and down 2. Topology deactivation and activation 6. container failure and recovery Running it locally: scripts/run_integration_topology_test.sh This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
