[incubator-heron] branch master updated: Refactoring Nomad documentation and add documentation on running heron via docker on Nomad (#2876)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 075e06e Refactoring Nomad documentation and add documentation on
running heron via docker on Nomad (#2876)
075e06e is described below
commit 075e06e0292e495b8a609266322a2a0b41838034
Author: Boyang Jerry Peng <jerry.boyang.p...@gmail.com>
AuthorDate: Mon May 7 21:28:54 2018 -0700
Refactoring Nomad documentation and add documentation on running heron via
docker on Nomad (#2876)
---
.../schedulers/{nomad.md => nomad-docker.md} | 116 ++--
.../schedulers/{nomad.md => nomad-raw-execs.md}| 22 ++-
.../docs/operators/deployment/schedulers/nomad.md | 205 +
3 files changed, 38 insertions(+), 305 deletions(-)
diff --git a/website/content/docs/operators/deployment/schedulers/nomad.md
b/website/content/docs/operators/deployment/schedulers/nomad-docker.md
similarity index 57%
copy from website/content/docs/operators/deployment/schedulers/nomad.md
copy to website/content/docs/operators/deployment/schedulers/nomad-docker.md
index 81ac421..063a4cc 100644
--- a/website/content/docs/operators/deployment/schedulers/nomad.md
+++ b/website/content/docs/operators/deployment/schedulers/nomad-docker.md
@@ -1,26 +1,36 @@
---
-title: Nomad
+title: Running Heron via Docker Containers on Nomad
---
-Heron supports [Hashicorp](https://hashicorp.com)'s
[Nomad](https://nomadproject.io) as a scheduler. You can use Nomad for either
small- or large-scale Heron deployments or to run Heron locally in [standalone
mode](../standalone).
+Below are instructions on how to run Heron on Nomad via docker containers. In
this mode, Heron executors will run as docker containers on host machines.
-## Nomad setup
-
-Setting up a nomad cluster will not be covered here. See the [official Nomad
docs](https://www.nomadproject.io/intro/getting-started/install.html) for
instructions.
-
-> Heron currently only supports the [raw exec
driver](https://www.nomadproject.io/docs/drivers/raw_exec.html) for Nomad.
+### Requirements
When setting up your Nomad cluster, the following are required:
* The [Heron CLI tool](../../../heron-cli) must be installed on each machine
used to deploy Heron topologies
* Python 2.7, Java 7 or 8, and [curl](https://curl.haxx.se/) must be installed
on every machine in the cluster
* A [ZooKeeper cluster](https://zookeeper.apache.org)
+* Docker installed and enabled on every machine
+* Each machine must also be able to pull the official Heron docker image from
DockerHub or have the image preloaded.
-## Configuring Heron settings
+### Configuring Heron settings
Before running Heron via Nomad, you'll need to configure some settings. Once
you've [installed Heron](../../../../getting-started), all of the
configurations you'll need to modify will be in the `~/.heron/conf/nomad`
diredctory.
-First, you'll need to use a topology uploader to deploy topology packages to
nodes in your cluster. You can use one of the following uploaders:
+First, make sure that the `heron.nomad.driver` is set to "docker" in
`~/.heron/conf/nomad/scheduler.yaml` e.g.
+
+```yaml
+heron.nomad.driver: "docker"
+```
+
+You can also adjust which docker image to use for running Heron via the
`heron.executor.docker.image` in `~/.heron/conf/nomad/scheduler.yaml` e.g.
+
+```yaml
+heron.executor.docker.image: 'heron/heron:latest'
+```
+
+You'll need to use a topology uploader to deploy topology packages to nodes in
your cluster. You can use one of the following uploaders:
* The HTTP uploader in conjunction with Heron's [API
server](../../../heron-api-server). The Heron API server acts like a file
server to which users can upload topology packages. The API server distributes
the packages, along with the Heron core package, to the relevant machines. You
can also use the API server to submit your Heron topology to Nomad (described
[below](#deploying-with-the-api-server))
* [Amazon S3](../../uploaders/s3). Please note that the S3 uploader requires
an AWS account.
@@ -40,41 +50,6 @@ Then, update the `heron.nomad.scheduler.uri` parameter in
`scheduler.yaml` to th
heron.nomad.scheduler.uri: http://127.0.0.1:4646
```
-You may also want to configure where Heron will store files on your machine if
you're running Nomad locally (in `scheduler.yaml`). Here's an example:
-
-```yaml
-heron.scheduler.local.working.directory:
${HOME}/.herondata/topologies/${CLUSTER}/${ROLE}/${TOPOLOGY_ID}
-```
-
-> Heron uses string interpolation to fill in the missing values for `CLUSTER`,
`ROLE`, etc.
-
-## Distributing Heron core
-
-The Heron core package needs to be made available for every machine in the
cluster to download. You'll need to provide a URI for the Heron core package.
Here are the cur
[incubator-heron] branch master updated: fixing test (#2894)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 056ba11 fixing test (#2894)
056ba11 is described below
commit 056ba114543a5f24c261375f2308556694c0e852
Author: Josh Fischer <j...@joshfischer.io>
AuthorDate: Tue May 8 01:09:18 2018 -0500
fixing test (#2894)
---
.../org/apache/heron/eco/parser/EcoParserTest.java | 89 +-
1 file changed, 88 insertions(+), 1 deletion(-)
diff --git a/eco/tests/java/org/apache/heron/eco/parser/EcoParserTest.java
b/eco/tests/java/org/apache/heron/eco/parser/EcoParserTest.java
index 9e665b7..30c41e1 100644
--- a/eco/tests/java/org/apache/heron/eco/parser/EcoParserTest.java
+++ b/eco/tests/java/org/apache/heron/eco/parser/EcoParserTest.java
@@ -16,7 +16,94 @@
* specific language governing permissions and limitations
* under the License.
*/
-\n"
+package org.apache.heron.eco.parser;
+import java.io.ByteArrayInputStream;
+import java.io.FileInputStream;
+import java.io.InputStream;
+import java.util.List;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.powermock.api.mockito.PowerMockito;
+
+import org.apache.heron.eco.definition.BeanDefinition;
+import org.apache.heron.eco.definition.BeanReference;
+import org.apache.heron.eco.definition.BoltDefinition;
+import org.apache.heron.eco.definition.EcoTopologyDefinition;
+import org.apache.heron.eco.definition.GroupingDefinition;
+import org.apache.heron.eco.definition.PropertyDefinition;
+import org.apache.heron.eco.definition.StreamDefinition;
+
+import static junit.framework.TestCase.assertNotNull;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.is;
+import static org.junit.Assert.*;
+
+
+/**
+ * Unit tests for {@link EcoParser}
+ */
+public class EcoParserTest {
+
+ private static final String BOLT_1 = "bolt-1";
+ private static final String BOLT_2 = "bolt-2";
+ private static final String YAML_NO_CONFIG_STR = "# topology definition\n"
+ + "# name to be used when submitting\n"
+ + "name: \"yaml-topology\"\n"
+ + "\n"
+ + "# topology configuration\n"
+ + "# this will be passed to the submitter as a map of config options\n"
+ + "#\n"
+ + "# spout definitions\n"
+ + "spouts:\n"
+ + " - id: \"spout-1\"\n"
+ + "className: \"org.apache.heron.sample.TestWordSpout\"\n"
+ + "parallelism: 1\n"
+ + "\n"
+ + "# bolt definitions\n"
+ + "bolts:\n"
+ + " - id: \"bolt-1\"\n"
+ + "className: \"org.apache.heron.sample.TestWordCounter\"\n"
+ + "parallelism: 2\n"
+ + "\n"
+ + " - id: \"bolt-2\"\n"
+ + "className: \"org.apache.heron.sample.LogInfoBolt\"\n"
+ + "parallelism: 1\n"
+ + "\n"
+ + "#stream definitions\n"
+ + "# stream definitions define connections between spouts and bolts.\n"
+ + "# note that such connections can be cyclical\n"
+ + "streams:\n"
+ + " - name: \"spout-1 --> bolt-1\" # name isn't used (placeholder for
logging, UI, etc.)\n"
+ + "id: \"connection-1\"\n"
+ + "from: \"spout-1\"\n"
+ + "to: \"bolt-1\"\n"
+ + "grouping:\n"
+ + " type: FIELDS\n"
+ + " args: [\"word\"]\n"
+ + "\n"
+ + " - name: \"bolt-1 --> bolt2\"\n"
+ + "id: \"connection-2\"\n"
+ + "from: \"bolt-1\"\n"
+ + "to: \"bolt-2\"\n"
+ + "grouping:\n"
+ + " type: SHUFFLE";
+ private static final String YAML_STR = "# Licensed to the Apache Software
Foundation"
+ + " (ASF) under one\n"
+ + "# or more contributor license agreements. See the NOTICE file\n"
+ + "# distributed with this work for additional information\n"
+ + "# regarding copyright ownership. The ASF licenses this file\n"
+ + "# to you under the Apache License, Version 2.0 (the\n"
+ + "# \"License\"); you may not use this file except in compliance\n"
+ + "# with the License. You may obtain a copy of the License at\n"
+ + "#\n"
+ + "# http://www.apache.org/licenses/LICENSE-2.0\n;
+ + "#\n"
+ + "# Unless required by applicable law or agreed to in writing,
software\n"
+ + "# distributed under the License is distributed on an \"AS IS\"
BASIS,\n"
+ + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.\n"
+ + "# See the License for the specific language governing permissions
and\n"
+ + "# limitations under the License.\n"
+ "\n"
+ "---\n"
+ "\n"
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch master updated: Nwang/update license in other files (#2895)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 5754033 Nwang/update license in other files (#2895) 5754033 is described below commit 5754033e3e4241d734df4e8e2f0ef9249d3b088f Author: Ning Wang <nw...@twitter.com> AuthorDate: Tue May 8 00:02:43 2018 -0700 Nwang/update license in other files (#2895) * Update license text in other files * whitespace * more files --- config/empty.cc| 25 +++ .../apache/heron/examples/eco/sample.properties| 29 -- .../apache/heron/examples/eco/sample.properties| 29 -- heron/proto/empty.cc | 25 +++ scripts/compile/errors.sh | 23 + scripts/resources/idea/copyright/heron.xml | 2 +- .../bazel/checkstyle/apache_coding_style.xml | 29 -- .../apache/bazel/checkstyle/heron_coding_style.xml | 29 -- 8 files changed, 106 insertions(+), 85 deletions(-) diff --git a/config/empty.cc b/config/empty.cc index 3305740..8c25c1e 100644 --- a/config/empty.cc +++ b/config/empty.cc @@ -1,17 +1,20 @@ -/* - * Copyright 2015 Twitter, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. */ // TODO(kramasamy): Can we get rid of this file? diff --git a/eco-heron-examples/src/java/org/apache/heron/examples/eco/sample.properties b/eco-heron-examples/src/java/org/apache/heron/examples/eco/sample.properties index 65a85c4..67da8b1 100644 --- a/eco-heron-examples/src/java/org/apache/heron/examples/eco/sample.properties +++ b/eco-heron-examples/src/java/org/apache/heron/examples/eco/sample.properties @@ -1,16 +1,19 @@ -# // Copyright 2018 Twitter. All rights reserved. -# // -# // Licensed under the Apache License, Version 2.0 (the "License"); -# // you may not use this file except in compliance with the License. -# // You may obtain a copy of the License at -# // -# // http://www.apache.org/licenses/LICENSE-2.0 -# // -# // Unless required by applicable law or agreed to in writing, software -# // distributed under the License is distributed on an "AS IS" BASIS, -# // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# // See the License for the specific language governing permissions and -# // limitations under the License. +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. ecoPropertyOne=thisValueWasSetFromAPropertiesFile diff --git a/eco-storm-examples/src/java/org/apache/heron/examples/eco/sample.properties b/eco-storm-examples/src/java/org/apache/heron/examples/eco/sample.properties index 65
[incubator-heron] branch master updated: add documentation about nomad consul integration (#2898)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 38f6a57 add documentation about nomad consul integration (#2898) 38f6a57 is described below commit 38f6a570d081ad1429d444f392a6679737084bcb Author: Boyang Jerry Peng <jerry.boyang.p...@gmail.com> AuthorDate: Thu May 17 07:45:57 2018 -0700 add documentation about nomad consul integration (#2898) * add documentation about nomad consul integration * add newline --- heron/config/src/yaml/conf/nomad/scheduler.yaml| 4 +-- .../config/src/yaml/conf/standalone/scheduler.yaml | 2 +- .../deployment/schedulers/nomad-docker.md | 34 ++-- .../deployment/schedulers/nomad-raw-execs.md | 36 -- 4 files changed, 68 insertions(+), 8 deletions(-) diff --git a/heron/config/src/yaml/conf/nomad/scheduler.yaml b/heron/config/src/yaml/conf/nomad/scheduler.yaml index 8e7e3a5..832ca76 100644 --- a/heron/config/src/yaml/conf/nomad/scheduler.yaml +++ b/heron/config/src/yaml/conf/nomad/scheduler.yaml @@ -58,5 +58,5 @@ heron.nomad.metrics.service.check.interval.sec: 10 heron.nomad.metrics.service.check.timeout.sec: 2 # additional tags to be attached to metrics service -# A tag of - with be automaticallu attached -heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron" \ No newline at end of file +# A tag of - with be automatically attached +heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron" diff --git a/heron/config/src/yaml/conf/standalone/scheduler.yaml b/heron/config/src/yaml/conf/standalone/scheduler.yaml index fc1cd27..a492ef2 100644 --- a/heron/config/src/yaml/conf/standalone/scheduler.yaml +++ b/heron/config/src/yaml/conf/standalone/scheduler.yaml @@ -51,5 +51,5 @@ heron.nomad.metrics.service.check.interval.sec: 10 heron.nomad.metrics.service.check.timeout.sec: 2 # additional tags to be attached to metrics service in a comma delimited list -# A tag of - with be automaticallu attached +# A tag of - with be automatically attached heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron" diff --git a/website/content/docs/operators/deployment/schedulers/nomad-docker.md b/website/content/docs/operators/deployment/schedulers/nomad-docker.md index 063a4cc..cde8320 100644 --- a/website/content/docs/operators/deployment/schedulers/nomad-docker.md +++ b/website/content/docs/operators/deployment/schedulers/nomad-docker.md @@ -4,7 +4,7 @@ title: Running Heron via Docker Containers on Nomad Below are instructions on how to run Heron on Nomad via docker containers. In this mode, Heron executors will run as docker containers on host machines. -### Requirements +## Requirements When setting up your Nomad cluster, the following are required: @@ -14,7 +14,7 @@ When setting up your Nomad cluster, the following are required: * Docker installed and enabled on every machine * Each machine must also be able to pull the official Heron docker image from DockerHub or have the image preloaded. -### Configuring Heron settings +## Configuring Heron settings Before running Heron via Nomad, you'll need to configure some settings. Once you've [installed Heron](../../../../getting-started), all of the configurations you'll need to modify will be in the `~/.heron/conf/nomad` diredctory. @@ -132,3 +132,33 @@ Heron users can upload their Heron topology packages to the Heron API server usi heron.class.uploader:org.apache.heron.uploader.http.HttpUploader heron.uploader.http.uri: http://localhost:9000/api/v1/file/upload ``` + +## Integration with Consul for metrics +Each container part of a Heron topology serves metrics out of a port randomly generated by Nomad. Thus, Consul is needed for service discovery for users to determine which port the container is serving the metrics out of. +Every Heron executor running in a docker container will automatically register itself as a service with Consul given that there is a Consul cluster running. The port Heron will be serving metrics will be registered with Consul. + +The service will be registered with the name with the following format: + +```yaml +metrics-heron-- +``` + +Each heron executor registered with Consul will be tagged with + +```yaml +- +``` + +To add additional tags, please add specify them in a comma delimited list via + +```yaml +heron.nomad.metrics.service.additional.tags +``` + +in `scheduler.yaml`. For example: + +```yaml +heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron" +``` + +Users can then configure Prometheus to scrape metrics for each container based on these tags diff --git a/website/content/docs/operators/deployment/schedulers/nomad-raw-execs.md b/website/content/docs/operators/
[incubator-heron] branch master updated: Add permissive licence to libunwind (#2914)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 13f6626 Add permissive licence to libunwind (#2914) 13f6626 is described below commit 13f662668afcf025192ec3e9e95713e49f91ce9a Author: Oliver Bristow AuthorDate: Fri Jun 8 15:25:11 2018 +0100 Add permissive licence to libunwind (#2914) --- third_party/libunwind/BUILD | 2 ++ 1 file changed, 2 insertions(+) diff --git a/third_party/libunwind/BUILD b/third_party/libunwind/BUILD index 9c73b70..9a113a3 100644 --- a/third_party/libunwind/BUILD +++ b/third_party/libunwind/BUILD @@ -1,3 +1,5 @@ +licenses(["permissive"]) + package(default_visibility = ["//visibility:public"]) exports_files([ -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] branch revert-2921-add-jdk-to-bazel_configure created (now c50506e)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a change to branch revert-2921-add-jdk-to-bazel_configure in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. at c50506e Revert "Add JDK/JAVA_HOME to bazel_configure.py (#2921)" This branch includes the following new commits: new c50506e Revert "Add JDK/JAVA_HOME to bazel_configure.py (#2921)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] 01/01: Revert "Add JDK/JAVA_HOME to bazel_configure.py (#2921)"
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch revert-2921-add-jdk-to-bazel_configure
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
commit c50506ef656f4dbac1e0347b9a39e457d73254de
Author: Karthik Ramasamy
AuthorDate: Mon Jun 18 06:17:45 2018 -0700
Revert "Add JDK/JAVA_HOME to bazel_configure.py (#2921)"
This reverts commit b3522d2c319965630b64d0c9a45a3a5b56d6fca6.
---
bazel_configure.py | 14 --
1 file changed, 14 deletions(-)
diff --git a/bazel_configure.py b/bazel_configure.py
index ed8b439..b91efe2 100755
--- a/bazel_configure.py
+++ b/bazel_configure.py
@@ -263,19 +263,6 @@ variable to specify the full path to yours.'""" %
(program, program, program, en
print('Using %s:\t%s' % (msg.ljust(20), print_value))
return VALUE
-def discover_jdk():
- try:
-jdk_path = os.environ['JAVA_HOME']
- except KeyError:
-javac_path = real_program_path('javac')
-if javac_path is None:
-fail("You need to have JDK installed to build Heron.\n"
- "You can set the JAVA_HOME environment variavle to specify the
full path to yours.")
-jdk_bin_path = os.path.dirname(javac_path)
-jdk_path = os.path.dirname(jdk_bin_path)
- print('Using %s:\t%s' % ('JDK'.ljust(20), jdk_path))
- return jdk_path
-
##
# Discover the linker directory
##
@@ -390,7 +377,6 @@ def main():
env_map['CXXCPP'] = discover_tool('cpp','C++ preprocessor', 'CXXCPP',
cpp_min)
env_map['LD'] = discover_tool('ld','linker', 'LD')
env_map['BLDFLAG'] = discover_linker(env_map)
- env_map['JAVA_HOME'] = discover_jdk()
# Discover the utilities
env_map['AUTOMAKE'] = discover_tool('automake', 'Automake', 'AUTOMAKE',
'1.9.6')
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch master updated: Correct case of acronyms (#2920)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 58078e7 Correct case of acronyms (#2920) 58078e7 is described below commit 58078e7c349af4c420713a9aa37fc13670291f17 Author: Oliver Bristow AuthorDate: Mon Jun 18 14:04:39 2018 +0100 Correct case of acronyms (#2920) * Correct dag to DAG * Correct hdfs to HDFS * Correct s3 to S3 * Correct ram to RAM * Add vagrant/.vagrant/ to .gitignore * Correct jvm to JVM * Correct apiserver to API server * Correct 'api server' to 'API server' * Correct api to API * Correct cpu to CPU * Fix error made while correcting documentation * Correct acronyms * ram, Ram → RAM * Cpu → CPU --- .gitignore | 5 ++- WORKSPACE | 6 +-- deploy/kubernetes/general/README.md| 8 ++-- deploy/kubernetes/gke/gcs-apiserver.yaml | 2 +- deploy/kubernetes/gke/medium.yaml | 2 +- deploy/kubernetes/gke/small.yaml | 2 +- deploy/kubernetes/helm/templates/NOTES.txt | 2 +- deploy/kubernetes/helm/values.yaml.template| 2 +- deploy/kubernetes/minikube/README.md | 8 ++-- .../apache/heron/examples/eco/heron_wordcount.yaml | 4 +- .../heron/examples/eco/simple_wordcount.yaml | 4 +- .../apache/heron/eco/builder/ConfigBuilder.java| 6 +-- .../examples/api/ComponentJVMOptionsTopology.java | 2 +- .../heron/examples/api/spout/TestWordSpout.java| 2 +- heron/api/src/cpp/config/config.h | 18 .../api/src/java/org/apache/heron/api/Config.java | 18 .../org/apache/heron/api/utils/TopologyUtils.java | 4 +- .../java/org/apache/heron/streamlet/Builder.java | 2 +- .../apache/heron/streamlet/impl/BuilderImpl.java | 2 +- .../org/apache/heron/streamlet/scala/Builder.scala | 4 +- .../src/cpp/config/topology-config-helper.cpp | 2 +- .../common/src/cpp/config/topology-config-helper.h | 4 +- heron/common/src/cpp/network/baseserver.h | 2 +- .../heron/common/config/SystemConfigKey.java | 2 +- .../heron/common/network/IncomingPacket.java | 4 +- .../heron/common/network/OutgoingPacket.java | 4 +- .../heron/common/utils/metrics/JVMMetrics.java | 16 +++ .../heron/common/utils/TopologyUtilsTest.java | 20 - .../src/yaml/conf/aurora/heron_internals.yaml | 2 +- heron/config/src/yaml/conf/aurora/stateful.yaml| 2 +- .../src/yaml/conf/examples/heron_internals.yaml| 2 +- .../src/yaml/conf/kubernetes/heron_internals.yaml | 2 +- .../src/yaml/conf/local/heron_internals.yaml | 2 +- .../src/yaml/conf/localzk/heron_internals.yaml | 2 +- .../src/yaml/conf/marathon/heron_internals.yaml| 2 +- .../src/yaml/conf/mesos/heron_internals.yaml | 2 +- .../src/yaml/conf/nomad/heron_internals.yaml | 2 +- heron/config/src/yaml/conf/nomad/scheduler.yaml| 4 +- .../src/yaml/conf/sandbox/heron_internals.yaml | 2 +- .../src/yaml/conf/slurm/heron_internals.yaml | 2 +- .../src/yaml/conf/standalone/heron_internals.yaml | 2 +- .../config/src/yaml/conf/standalone/scheduler.yaml | 4 +- .../standalone/templates/scheduler.template.yaml | 4 +- .../src/yaml/conf/test/test_heron_internals.yaml | 2 +- .../config/src/yaml/conf/yarn/heron_internals.yaml | 2 +- heron/executor/src/python/heron_executor.py| 4 +- .../instance/bolt/BoltOutputCollectorImpl.java | 2 +- .../instance/spout/SpoutOutputCollectorImpl.java | 2 +- .../src/python/instance/st_heron_instance.py | 4 +- .../src/python/utils/metrics/py_metrics.py | 6 +-- .../instance/src/python/utils/system_constants.py | 2 +- .../java/org/apache/heron/dlog/DLOutputStream.java | 2 +- .../sink/metricscache/MetricsCacheSink.java| 2 +- .../heron/metricsmgr/sink/tmaster/TMasterSink.java | 2 +- .../binpacking/FirstFitDecreasingPacking.java | 16 +++ .../apache/heron/packing/builder/Container.java| 10 ++--- .../heron/packing/builder/PackingPlanBuilder.java | 2 +- .../roundrobin/ResourceCompliantRRPacking.java | 16 +++ .../packing/roundrobin/RoundRobinPacking.java | 52 +++--- .../apache/heron/packing/utils/PackingUtils.java | 6 +-- .../org/apache/heron/packing/AssertPacking.java| 10 ++--- .../apache/heron/packing/CommonPackingTests.java | 10 ++--- .../binpacking/FirstFitDecreasingPackingTest.java | 24 +- .../roundrobin/ResourceCompliantRRPackingTest.java | 24 +- .../packing/roundrobin/RoundRobinPackingTest.java | 16 +++ .../heron/scheduler/aurora
[incubator-heron] branch revert-2921-add-jdk-to-bazel_configure deleted (was c50506e)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a change to branch revert-2921-add-jdk-to-bazel_configure in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. was c50506e Revert "Add JDK/JAVA_HOME to bazel_configure.py (#2921)" This change permanently discards the following revisions: discard c50506e Revert "Add JDK/JAVA_HOME to bazel_configure.py (#2921)" -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] branch master updated: Add JDK/JAVA_HOME to bazel_configure.py (#2921)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new b3522d2 Add JDK/JAVA_HOME to bazel_configure.py (#2921)
b3522d2 is described below
commit b3522d2c319965630b64d0c9a45a3a5b56d6fca6
Author: Oliver Bristow
AuthorDate: Mon Jun 18 14:10:01 2018 +0100
Add JDK/JAVA_HOME to bazel_configure.py (#2921)
---
bazel_configure.py | 14 ++
1 file changed, 14 insertions(+)
diff --git a/bazel_configure.py b/bazel_configure.py
index b91efe2..ed8b439 100755
--- a/bazel_configure.py
+++ b/bazel_configure.py
@@ -263,6 +263,19 @@ variable to specify the full path to yours.'""" %
(program, program, program, en
print('Using %s:\t%s' % (msg.ljust(20), print_value))
return VALUE
+def discover_jdk():
+ try:
+jdk_path = os.environ['JAVA_HOME']
+ except KeyError:
+javac_path = real_program_path('javac')
+if javac_path is None:
+fail("You need to have JDK installed to build Heron.\n"
+ "You can set the JAVA_HOME environment variavle to specify the
full path to yours.")
+jdk_bin_path = os.path.dirname(javac_path)
+jdk_path = os.path.dirname(jdk_bin_path)
+ print('Using %s:\t%s' % ('JDK'.ljust(20), jdk_path))
+ return jdk_path
+
##
# Discover the linker directory
##
@@ -377,6 +390,7 @@ def main():
env_map['CXXCPP'] = discover_tool('cpp','C++ preprocessor', 'CXXCPP',
cpp_min)
env_map['LD'] = discover_tool('ld','linker', 'LD')
env_map['BLDFLAG'] = discover_linker(env_map)
+ env_map['JAVA_HOME'] = discover_jdk()
# Discover the utilities
env_map['AUTOMAKE'] = discover_tool('automake', 'Automake', 'AUTOMAKE',
'1.9.6')
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch master updated: Make TopologyBuilder less permissive (#2929)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 5ee2490 Make TopologyBuilder less permissive (#2929)
5ee2490 is described below
commit 5ee2490c025c7684c097962cf6d25f54ead73cf4
Author: Oliver Bristow
AuthorDate: Tue Jun 26 16:37:58 2018 +0200
Make TopologyBuilder less permissive (#2929)
* Raise issues in TopologyBuilder when introduced
* Do not allow malformed HERON_OPTION parts
* Convert TopologyBuilder._specs to dict
---
heronpy/api/topology.py | 45 +
1 file changed, 21 insertions(+), 24 deletions(-)
diff --git a/heronpy/api/topology.py b/heronpy/api/topology.py
index 5af5873..a4c8f94 100644
--- a/heronpy/api/topology.py
+++ b/heronpy/api/topology.py
@@ -153,7 +153,7 @@ class TopologyType(type):
return
heron_options = TopologyType.get_heron_options_from_env()
initial_state = heron_options.get("cmdline.topology.initial.state",
"RUNNING")
-tmp_directory = heron_options.get("cmdline.topologydefn.tmpdirectory",
None)
+tmp_directory = heron_options.get("cmdline.topologydefn.tmpdirectory")
if tmp_directory is None:
raise RuntimeError("Topology definition temp directory not specified")
@@ -194,24 +194,25 @@ class TopologyType(type):
Currently supports the following options natively:
-- `cmdline.topologydefn.tmpdirectory`: the directory to which this
+- `cmdline.topologydefn.tmpdirectory`: (required) the directory to which
this
topology's defn file is written
-- `cmdline.topology.initial.state`: the initial state of the topology
-- `cmdline.topology.name`: topology name on deployment
+- `cmdline.topology.initial.state`: (default: "RUNNING") the initial state
of the topology
+- `cmdline.topology.name`: (default: class name) topology name on
deployment
Returns: map mapping from key to value
"""
-heron_options_raw = os.environ.get("HERON_OPTIONS", None)
+heron_options_raw = os.environ.get("HERON_OPTIONS")
if heron_options_raw is None:
raise RuntimeError("HERON_OPTIONS environment variable not found")
-ret = {}
-heron_opt_list = heron_options_raw.replace("", " ").split(',')
-for opt_raw in heron_opt_list:
- opt = opt_raw.split("=")
- if len(opt) == 2:
-ret[opt[0]] = opt[1]
-return ret
+options = {}
+for option_line in heron_options_raw.replace("", " ").split(','):
+ key, sep, value = option_line.partition("=")
+ if sep:
+options[key] = value
+ else:
+raise ValueError("Invalid HERON_OPTIONS part %r" % option_line)
+return options
@classmethod
def add_bolts_and_spouts(mcs, topology, class_dict):
@@ -336,7 +337,7 @@ class TopologyBuilder(object):
self.topology_name = name
-self._specs = []
+self._specs = {}
self._topology_config = {}
def add_spec(self, *specs):
@@ -351,7 +352,12 @@ class TopologyBuilder(object):
% str(spec))
if spec.name is None:
raise ValueError("TopologyBuilder cannot take a spec without name")
- self._specs.append(spec)
+ if spec.name == "config":
+raise ValueError("config is a reserved name")
+ if spec.name in self._specs:
+raise ValueError("Attempting to add duplicate spec name: %r %r" %
(spec.name, spec))
+
+ self._specs[spec.name] = spec
def add_spout(self, name, spout_cls, par, config=None,
optional_outputs=None):
"""Add a spout to the topology"""
@@ -378,16 +384,7 @@ class TopologyBuilder(object):
self._topology_config = config
def _construct_topo_class_dict(self):
-class_dict = {}
-
-# specs
-for spec in self._specs:
- name = spec.name
- if name in class_dict:
-raise ValueError("Duplicate spec names: %s" % name)
- class_dict[name] = spec
-
-# config
+class_dict = self._specs.copy()
class_dict["config"] = self._topology_config
return class_dict
[incubator-heron] branch master updated: Upgrade to Bazel 0.14.1 (#2932)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new d7c8c62 Upgrade to Bazel 0.14.1 (#2932)
d7c8c62 is described below
commit d7c8c62f05e7c14daf78d200129ad17f6d3fe970
Author: Oliver Bristow
AuthorDate: Sat Jun 23 01:33:47 2018 +0100
Upgrade to Bazel 0.14.1 (#2932)
---
.travis.yml | 6 +++---
docker/compile/Dockerfile.centos7 | 2 +-
docker/compile/Dockerfile.debian8 | 2 +-
docker/compile/Dockerfile.ubuntu14.04 | 2 +-
docker/compile/Dockerfile.ubuntu15.10 | 2 +-
docker/compile/Dockerfile.ubuntu16.04 | 2 +-
vagrant/init.sh | 2 +-
website/config.yaml | 2 +-
website/content/docs/developers/compiling/docker.md | 2 +-
website/content/docs/developers/compiling/linux.md | 4 ++--
10 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 51ee089..065aa74 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -28,9 +28,9 @@ env:
before_install:
# download and install bazel
- - wget -q
'https://github.com/bazelbuild/bazel/releases/download/0.9.0/bazel-0.9.0-installer-linux-x86_64.sh'
- - chmod +x bazel-0.9.0-installer-linux-x86_64.sh
- - ./bazel-0.9.0-installer-linux-x86_64.sh --user
+ - wget -q
'https://github.com/bazelbuild/bazel/releases/download/0.14.1/bazel-0.14.1-installer-linux-x86_64.sh'
+ - chmod +x bazel-0.14.1-installer-linux-x86_64.sh
+ - ./bazel-0.14.1-installer-linux-x86_64.sh --user
before_script:
# install python module for wheel files
diff --git a/docker/compile/Dockerfile.centos7
b/docker/compile/Dockerfile.centos7
index 65d7abe..f7176a0 100644
--- a/docker/compile/Dockerfile.centos7
+++ b/docker/compile/Dockerfile.centos7
@@ -2,7 +2,7 @@ FROM centos:centos7
# This is passed to the heron build command via the --config flag
ENV TARGET_PLATFORM centos
-ENV bazelVersion 0.9.0
+ENV bazelVersion 0.14.1
RUN yum -y upgrade
RUN yum -y install \
diff --git a/docker/compile/Dockerfile.debian8
b/docker/compile/Dockerfile.debian8
index 453ccaf..cfb74af 100644
--- a/docker/compile/Dockerfile.debian8
+++ b/docker/compile/Dockerfile.debian8
@@ -2,7 +2,7 @@ FROM openjdk:8-jdk-slim
# This is passed to the heron build command via the --config flag
ENV TARGET_PLATFORM debian
-ENV bazelVersion 0.9.0
+ENV bazelVersion 0.14.1
RUN apt-get update && apt-get -y install \
automake \
diff --git a/docker/compile/Dockerfile.ubuntu14.04
b/docker/compile/Dockerfile.ubuntu14.04
index 24a6bb3..c0b6aa5 100644
--- a/docker/compile/Dockerfile.ubuntu14.04
+++ b/docker/compile/Dockerfile.ubuntu14.04
@@ -2,7 +2,7 @@ FROM ubuntu:14.04
# This is passed to the heron build command via the --config flag
ENV TARGET_PLATFORM ubuntu
-ENV bazelVersion 0.9.0
+ENV bazelVersion 0.14.1
RUN apt-get update && apt-get -y install \
automake \
diff --git a/docker/compile/Dockerfile.ubuntu15.10
b/docker/compile/Dockerfile.ubuntu15.10
index 58ea71a..ee9fa2f 100644
--- a/docker/compile/Dockerfile.ubuntu15.10
+++ b/docker/compile/Dockerfile.ubuntu15.10
@@ -2,7 +2,7 @@ FROM ubuntu:15.10
# This is passed to the heron build command via the --config flag
ENV TARGET_PLATFORM ubuntu
-ENV bazelVersion 0.9.0
+ENV bazelVersion 0.14.1
RUN apt-get update && apt-get -y install \
automake \
diff --git a/docker/compile/Dockerfile.ubuntu16.04
b/docker/compile/Dockerfile.ubuntu16.04
index 97c1a24..ade285c 100644
--- a/docker/compile/Dockerfile.ubuntu16.04
+++ b/docker/compile/Dockerfile.ubuntu16.04
@@ -2,7 +2,7 @@ FROM ubuntu:16.04
# This is passed to the heron build command via the --config flag
ENV TARGET_PLATFORM ubuntu
-ENV bazelVersion 0.9.0
+ENV bazelVersion 0.14.1
RUN apt-get update && apt-get -y install \
automake \
diff --git a/vagrant/init.sh b/vagrant/init.sh
index 1906032..504ec65 100644
--- a/vagrant/init.sh
+++ b/vagrant/init.sh
@@ -55,7 +55,7 @@ install_jdk8() {
apt-get -y install oracle-java8-installer oracle-java8-set-default vim
wget screen git
}
-bazelVersion=0.2.3
+bazelVersion=0.14.1
bazel_install() {
install_jdk8
apt-get install -y g++ automake cmake gcc-4.8 g++-4.8 zlib1g-dev zip
pkg-config wget libssl-dev
diff --git a/website/config.yaml b/website/config.yaml
index b73dd7c..600874f 100755
--- a/website/config.yaml
+++ b/website/config.yaml
@@ -37,7 +37,7 @@ params:
description: A realtime, distributed, fault-tolerant stream processing
engine from Twitter
versions:
heron: 0.17.5
-bazel: 0.5.4
+bazel: 0.14.1
heronpy: 0.17.5
assets:
favicon:
diff --git a/website/content/docs/developers/compiling/docker.md
b/website/content/docs/developers/c
[incubator-heron] branch master updated: make EvitionContext extend Serializable (#2935)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 805a401 make EvitionContext extend Serializable (#2935)
805a401 is described below
commit 805a401a14d5f211679262cb226176e0742340ea
Author: Neng Lu
AuthorDate: Sat Jun 23 11:00:30 2018 -0700
make EvitionContext extend Serializable (#2935)
* make EvitionContext extend Serializable
* fix typo
---
.../api/StatefulTumblingWindowTopology.java| 181 +
.../heron/api/windowing/EvictionContext.java | 4 +-
2 files changed, 184 insertions(+), 1 deletion(-)
diff --git
a/examples/src/java/org/apache/heron/examples/api/StatefulTumblingWindowTopology.java
b/examples/src/java/org/apache/heron/examples/api/StatefulTumblingWindowTopology.java
new file mode 100644
index 000..b39e3ff
--- /dev/null
+++
b/examples/src/java/org/apache/heron/examples/api/StatefulTumblingWindowTopology.java
@@ -0,0 +1,181 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.heron.examples.api;
+
+import java.time.Duration;
+import java.util.Map;
+import java.util.Random;
+import java.util.logging.Logger;
+
+import org.apache.heron.api.Config;
+import org.apache.heron.api.HeronSubmitter;
+import org.apache.heron.api.bolt.BaseStatefulWindowedBolt;
+import org.apache.heron.api.bolt.OutputCollector;
+import org.apache.heron.api.spout.BaseRichSpout;
+import org.apache.heron.api.spout.SpoutOutputCollector;
+import org.apache.heron.api.state.State;
+import org.apache.heron.api.topology.IStatefulComponent;
+import org.apache.heron.api.topology.OutputFieldsDeclarer;
+import org.apache.heron.api.topology.TopologyBuilder;
+import org.apache.heron.api.topology.TopologyContext;
+import org.apache.heron.api.tuple.Fields;
+import org.apache.heron.api.tuple.Tuple;
+import org.apache.heron.api.tuple.Values;
+import org.apache.heron.api.utils.Utils;
+import org.apache.heron.api.windowing.TupleWindow;
+import org.apache.heron.common.basics.ByteAmount;
+import org.apache.heron.examples.api.bolt.PrinterBolt;
+
+/**
+ * A sample topology that demonstrates the usage of {@link
org.apache.heron.api.bolt.IStatefulWindowedBolt}
+ * to calculate tumbling window sum. Topology also demonstrates how stateful
window processing is done
+ * in conjunction with effectively once guarantees
+ */
+public final class StatefulTumblingWindowTopology {
+
+ private static final Logger LOG =
Logger.getLogger(StatefulSlidingWindowTopology.class.getName());
+
+ private StatefulTumblingWindowTopology() {
+ }
+
+ private static class WindowSumBolt extends BaseStatefulWindowedBolt {
+private static final long serialVersionUID = -539382497249834244L;
+private State state;
+private long sum;
+
+private OutputCollector collector;
+
+@Override
+@SuppressWarnings("HiddenField")
+public void prepare(Map topoConf, TopologyContext context,
+OutputCollector collector) {
+ this.collector = collector;
+}
+
+@Override
+@SuppressWarnings("HiddenField")
+public void initState(State state) {
+ this.state = state;
+ sum = state.getOrDefault("sum", 0L);
+}
+
+@Override
+public void execute(TupleWindow inputWindow) {
+ for (Tuple tuple : inputWindow.get()) {
+sum += tuple.getLongByField("value");
+ }
+ collector.emit(new Values(sum));
+}
+
+@Override
+public void declareOutputFields(OutputFieldsDeclarer declarer) {
+ declarer.declare(new Fields("sum"));
+}
+
+@Override
+public void preSave(String checkpointId) {
+ state.put("sum", sum);
+}
+ }
+
+ public static class IntegerSpout extends BaseRichSpout
+ implements IStatefulComponent {
+private static final Logger LOG =
Logger.getLogger(IntegerSpout.class.getName());
+private static final long serialVersionUID = 5454291010750852782L;
+private SpoutOutputCollector collector;
+private
[incubator-heron] branch master updated: Add component config in heron UI and Tracker (#2860)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 84737b0 Add component config in heron UI and Tracker (#2860)
84737b0 is described below
commit 84737b088eb224fb04727c820653679aeb297477
Author: Ning Wang
AuthorDate: Tue Jun 26 17:07:43 2018 -0700
Add component config in heron UI and Tracker (#2860)
---
heron/tools/tracker/src/python/tracker.py | 10
heron/tools/tracker/tests/python/BUILD | 2 +-
.../tools/tracker/tests/python/tracker_unittest.py | 19
heron/tools/ui/resources/static/js/config.js | 53 ++
4 files changed, 73 insertions(+), 11 deletions(-)
diff --git a/heron/tools/tracker/src/python/tracker.py
b/heron/tools/tracker/src/python/tracker.py
index c03621a..4fb6e87 100644
--- a/heron/tools/tracker/src/python/tracker.py
+++ b/heron/tools/tracker/src/python/tracker.py
@@ -430,6 +430,7 @@ class Tracker(object):
"spouts": {},
"bolts": {},
"config": {},
+"components": {}
}
if not topology.physical_plan:
@@ -447,12 +448,21 @@ class Tracker(object):
# Configs
if topology.physical_plan.topology.topology_config:
physicalPlan["config"] =
convert_pb_kvs(topology.physical_plan.topology.topology_config.kvs)
+
for spout in spouts:
spout_name = spout.comp.name
physicalPlan["spouts"][spout_name] = []
+ if spout_name not in physicalPlan["components"]:
+physicalPlan["components"][spout_name] = {
+"config": convert_pb_kvs(spout.comp.config.kvs)
+}
for bolt in bolts:
bolt_name = bolt.comp.name
physicalPlan["bolts"][bolt_name] = []
+ if bolt_name not in physicalPlan["components"]:
+physicalPlan["components"][bolt_name] = {
+"config": convert_pb_kvs(bolt.comp.config.kvs)
+}
for stmgr in stmgrs:
host = stmgr.host_name
diff --git a/heron/tools/tracker/tests/python/BUILD
b/heron/tools/tracker/tests/python/BUILD
index 2c752e1..26d1bd8 100644
--- a/heron/tools/tracker/tests/python/BUILD
+++ b/heron/tools/tracker/tests/python/BUILD
@@ -58,7 +58,7 @@ pex_pytest(
pex_pytest(
name = "tracker_unittest",
-srcs = ["tracker_unittest.py"],
+srcs = ["tracker_unittest.py", "mock_proto.py"],
deps = [
"//heron/tools/tracker/src/python:tracker-py",
"//heron/proto:proto-py",
diff --git a/heron/tools/tracker/tests/python/tracker_unittest.py
b/heron/tools/tracker/tests/python/tracker_unittest.py
index 4a4ba67..136c6b9 100644
--- a/heron/tools/tracker/tests/python/tracker_unittest.py
+++ b/heron/tools/tracker/tests/python/tracker_unittest.py
@@ -8,6 +8,7 @@ import heron.proto.execution_state_pb2 as protoEState
from heron.statemgrs.src.python import statemanagerfactory
from heron.tools.tracker.src.python.topology import Topology
from heron.tools.tracker.src.python.tracker import Tracker
+from mock_proto import MockProto
class TrackerTest(unittest.TestCase):
def setUp(self):
@@ -215,3 +216,21 @@ class TrackerTest(unittest.TestCase):
self.tracker.removeTopology('top_name4', 'mock_name2')
self.assertItemsEqual([self.topology3, self.topology5],
self.tracker.topologies)
+
+ def test_extract_physical_plan(self):
+# Create topology
+pb_pplan = MockProto().create_mock_simple_physical_plan()
+topology = Topology('topology_name', 'state_manager')
+topology.set_physical_plan(pb_pplan)
+# Extract physical plan
+pplan = self.tracker.extract_physical_plan(topology)
+# Mock topology doesn't have topology config and instances
+self.assertEqual(pplan['config'], {})
+self.assertEqual(pplan['bolts'], {'mock_bolt': []})
+self.assertEqual(pplan['spouts'], {'mock_spout': []})
+self.assertEqual(pplan['components']['mock_bolt']['config'],
+ {'topology.component.parallelism': '1'})
+self.assertEqual(pplan['components']['mock_spout']['config'],
+ {'topology.component.parallelism': '1'})
+self.assertEqual(pplan['instances'], {})
+self.assertEqual(pplan['stmgrs'], {})
diff --git a/heron/tools/ui/resources/static/js/config.js
b/heron/tools/ui/resources/static/js/config.js
index a0b19b5..87a0409 100644
--- a/heron/tools/ui/resources/static/js/config.js
+++ b/heron/tools/ui/resources/static/js/config.js
@@ -2,9 +2,12 @@
var ConfigTable = React.createClass({
getInitialState: function() {
-return {'config': {}};
+return {
+ 'config': {},
+ 'component_config': {}
+};
},
- getTopologyConfig: function() {
+ ge
[incubator-heron] branch master updated: added S3 uploader configuration in Helm chart (#2939)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new ef50533 added S3 uploader configuration in Helm chart (#2939)
ef50533 is described below
commit ef50533d4ff9021d5cc3d3cbc2f57521245a8dd9
Author: Cristóbal Carnero Liñán
AuthorDate: Sun Jul 1 16:56:41 2018 +0200
added S3 uploader configuration in Helm chart (#2939)
* added S3 uploader configuration in Helm chart
* fixed package path
* added 'packing' configuration variable (RoundRobin by default)
---
deploy/kubernetes/helm/templates/tools.yaml | 19 +++
deploy/kubernetes/helm/values.yaml.template | 12
.../deployment/schedulers/kubernetes-helm.md | 15 +++
3 files changed, 46 insertions(+)
diff --git a/deploy/kubernetes/helm/templates/tools.yaml
b/deploy/kubernetes/helm/templates/tools.yaml
index 65daefe..7ee348d 100644
--- a/deploy/kubernetes/helm/templates/tools.yaml
+++ b/deploy/kubernetes/helm/templates/tools.yaml
@@ -127,9 +127,28 @@ spec:
-D heron.kubernetes.scheduler.uri=http://localhost:8001
-D heron.kubernetes.scheduler.namespace={{ .Release.Namespace }}
-D heron.executor.docker.image={{ .Values.image }}
+ {{- if eq .Values.uploader.class "dlog" }}
-D heron.class.uploader=org.apache.heron.uploader.dlog.DLUploader
-D heron.uploader.dlog.topologies.num.replicas={{ $jobReplicas }}
-D
heron.uploader.dlog.topologies.namespace.uri=distributedlog://{{ .Release.Name
}}-zookeeper:2181/distributedlog
+ {{- else if eq .Values.uploader.class "s3" }}
+ -D heron.class.uploader=org.apache.heron.uploader.s3.S3Uploader
+ -D heron.uploader.s3.bucket={{ .Values.uploader.s3Bucket }}
+ -D heron.uploader.s3.path_prefix={{
.Values.uploader.s3PathPrefix }}
+ -D heron.uploader.s3.access_key={{ .Values.uploader.s3AccessKey
}}
+ -D heron.uploader.s3.secret_key={{ .Values.uploader.s3SecretKey
}}
+ -D heron.uploader.s3.region={{ .Values.uploader.s3Region }}
+ {{- end }}
+ {{- if eq .Values.packing "RoundRobin" }}
+ -D
heron.class.packing.algorithm=org.apache.heron.packing.roundrobin.RoundRobinPacking
+ -D
heron.class.repacking.algorithm=org.apache.heron.packing.roundrobin.RoundRobinPacking
+ {{- else if eq .Values.packing "ResourceCompliantRR" }}
+ -D
heron.class.packing.algorithm=org.apache.heron.packing.roundrobin.ResourceCompliantRRPacking
+ -D
heron.class.repacking.algorithm=org.apache.heron.packing.roundrobin.ResourceCompliantRRPacking
+ {{- else if eq .Values.packing "FirstFitDecreasing" }}
+ -D
heron.class.packing.algorithm=org.apache.heron.packing.binpacking.FirstFitDecreasingPacking
+ -D
heron.class.repacking.algorithm=org.apache.heron.packing.binpacking.FirstFitDecreasingPacking
+ {{- end }}
envFrom:
- configMapRef:
name: {{ .Release.Name }}-tools-config
diff --git a/deploy/kubernetes/helm/values.yaml.template
b/deploy/kubernetes/helm/values.yaml.template
index 114da7d..524eef7 100644
--- a/deploy/kubernetes/helm/values.yaml.template
+++ b/deploy/kubernetes/helm/values.yaml.template
@@ -22,6 +22,18 @@ jobReplicas: 1
# amount of memory to provide for API server
apiServerMemory: 512M
+# Topologies uploader
+uploader:
+ class: dlog # s3
+ s3Bucket: ""
+ s3PathPrefix: ""
+ s3AccessKey: ""
+ s3SecretKey: ""
+ s3Region: ""
+
+# Packing algorithms
+packing: RoundRobin # ResourceCompliantRR, FirstFitDecreasing
+
# Number of replicas for storage bookies, memory and storage requirements
bookieReplicas: 1
bookieCpuMin: 100m
diff --git
a/website/content/docs/operators/deployment/schedulers/kubernetes-helm.md
b/website/content/docs/operators/deployment/schedulers/kubernetes-helm.md
index b42602f..b38c14f 100644
--- a/website/content/docs/operators/deployment/schedulers/kubernetes-helm.md
+++ b/website/content/docs/operators/deployment/schedulers/kubernetes-helm.md
@@ -209,6 +209,21 @@ $ helm install heron-charts/heron \
--set platform=aws
```
+# Using S3 uploader
+
+You can make Heron to use S3 to distribute the user topologies. First you need
to set up a S3 bucket and configure an IAM user with enough permissions over
it. Get access keys for the user. Then you can deploy Heron like this:
+
+```bash
+$ helm install heron-charts/heron \
+ --set platform=aws \
+ --set uploader.class=s3 \
+ --set uploader.s3Bucket=heron \
+ --set uploader.s3PathPr
[incubator-heron] branch karthik/scribe updated (1dc95aa -> 2c69caa)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a change to branch karthik/scribe in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 1dc95aa remove thrift and scribe to prepare for apache release add 2c69caa remove scribe dependency in tests No new revisions were added by this update. Summary of changes: heron/metricscachemgr/tests/java/BUILD | 2 -- 1 file changed, 2 deletions(-) -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] branch karthik/scribe updated (2c69caa -> abcd160)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a change to branch karthik/scribe in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 2c69caa remove scribe dependency in tests add 9d621b2 Update NOTICE file (#2878) add 9957ef3 Update license text in cpp files (#2882) add 0138b8d Update license text in java files (#2881) add 1e7e35a Update license text in H files (#2885) new abcd160 Merge branch 'master' into karthik/scribe The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: NOTICE | 8 +- .../apache/heron/examples/eco/EvenAndOddBolt.java | 32 +++-- .../org/apache/heron/examples/eco/LogInfoBolt.java | 32 +++-- .../apache/heron/examples/eco/RandomString.java| 32 +++-- .../heron/examples/eco/StatefulConsumerBolt.java | 32 +++-- .../heron/examples/eco/StatefulNumberSpout.java| 32 +++-- .../heron/examples/eco/StatefulRandomIntSpout.java | 32 +++-- .../heron/examples/eco/StatefulWindowSumBolt.java | 32 +++-- .../heron/examples/eco/TestFibonacciSpout.java | 32 +++-- .../heron/examples/eco/TestIBasicPrintBolt.java| 32 +++-- .../apache/heron/examples/eco/TestNameCounter.java | 32 +++-- .../apache/heron/examples/eco/TestNameSpout.java | 32 +++-- .../apache/heron/examples/eco/TestPrintBolt.java | 32 +++-- .../heron/examples/eco/TestPropertyHolder.java | 32 +++-- .../org/apache/heron/examples/eco/TestUnits.java | 32 +++-- .../apache/heron/examples/eco/TestWindowBolt.java | 32 +++-- .../org/apache/heron/examples/eco/WordSpout.java | 32 +++-- .../apache/heron/examples/eco/EvenAndOddBolt.java | 32 +++-- .../org/apache/heron/examples/eco/LogInfoBolt.java | 32 +++-- .../heron/examples/eco/TestFibonacciSpout.java | 32 +++-- .../heron/examples/eco/TestIBasicPrintBolt.java| 32 +++-- .../apache/heron/examples/eco/TestNameCounter.java | 32 +++-- .../apache/heron/examples/eco/TestNameSpout.java | 32 +++-- .../apache/heron/examples/eco/TestPrintBolt.java | 32 +++-- .../heron/examples/eco/TestPropertyHolder.java | 32 +++-- .../org/apache/heron/examples/eco/TestUnits.java | 32 +++-- .../apache/heron/examples/eco/TestWindowBolt.java | 32 +++-- eco/src/java/org/apache/heron/eco/Eco.java | 32 +++-- .../org/apache/heron/eco/builder/BoltBuilder.java | 32 +++-- .../apache/heron/eco/builder/BuilderUtility.java | 32 +++-- .../apache/heron/eco/builder/ComponentBuilder.java | 32 +++-- .../apache/heron/eco/builder/ConfigBuilder.java| 32 +++-- .../apache/heron/eco/builder/ObjectBuilder.java| 32 +++-- .../apache/heron/eco/builder/heron/EcoBuilder.java | 32 +++-- .../heron/eco/builder/heron/SpoutBuilder.java | 32 +++-- .../heron/eco/builder/heron/StreamBuilder.java | 32 +++-- .../apache/heron/eco/builder/storm/EcoBuilder.java | 32 +++-- .../heron/eco/builder/storm/SpoutBuilder.java | 32 +++-- .../heron/eco/builder/storm/StreamBuilder.java | 32 +++-- .../heron/eco/definition/BeanDefinition.java | 32 +++-- .../heron/eco/definition/BeanListReference.java| 32 +++-- .../apache/heron/eco/definition/BeanReference.java | 32 +++-- .../heron/eco/definition/BoltDefinition.java | 32 +++-- .../heron/eco/definition/ComponentStream.java | 32 +++-- .../definition/ConfigurationMethodDefinition.java | 32 +++-- .../heron/eco/definition/EcoExecutionContext.java | 32 +++-- .../eco/definition/EcoTopologyDefinition.java | 32 +++-- .../heron/eco/definition/GroupingDefinition.java | 32 +++-- .../heron/eco/definition/ObjectDefinition.java | 32 +++-- .../heron/eco/definition/PropertyDefinition.java | 32 +++-- .../heron/eco/definition/SpoutDefinition.java | 32 +++-- .../heron/eco/definition/StreamDefinition.java | 32 +++-- .../org/apache/heron/eco/parser/EcoParser.java | 32 +++-- .../org/apache/heron/eco/submit/EcoSubmitter.java | 32 +++-- eco/tests/java/org/apache/heron/eco/EcoTest.java | 32 +++-- .../apache/heron/eco/builder/BoltBuilderTest.java | 32 +++-- .../heron/eco/builder/BuilderUtilityTest.java | 32 +++-- .../heron/eco/builder/ComponentBuilderTest.java| 32 +++-- .../heron/eco/builder/ConfigBuilderTest.java | 32 +++-- .../heron/eco/builder/ObjectBuilderTest.java | 32 +++-- .../eco/builder/heron/HeronEcoBuilderTest.java | 32 +++-- .../eco/builder/heron/HeronSpoutBuilderTest.java | 32 +++-- .../eco/builder/heron/HeronStreamBuilderTest.java | 32 +++-- .../eco/builder/storm/StormEcoBuilderTest.java | 32 +++-- .../eco/builder/storm/StormSpoutBuilderTest.java | 32 +++-- .../eco/builder/storm/StormStreamBui
[incubator-heron] 01/01: Merge branch 'master' into karthik/scribe
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch karthik/scribe in repository https://gitbox.apache.org/repos/asf/incubator-heron.git commit abcd160d23650576025ac19257f29c6db88b612a Merge: 2c69caa 1e7e35a Author: Karthik Ramasamy <kart...@streaml.io> AuthorDate: Wed May 2 23:01:32 2018 -0700 Merge branch 'master' into karthik/scribe NOTICE | 8 +- .../apache/heron/examples/eco/EvenAndOddBolt.java | 32 +++-- .../org/apache/heron/examples/eco/LogInfoBolt.java | 32 +++-- .../apache/heron/examples/eco/RandomString.java| 32 +++-- .../heron/examples/eco/StatefulConsumerBolt.java | 32 +++-- .../heron/examples/eco/StatefulNumberSpout.java| 32 +++-- .../heron/examples/eco/StatefulRandomIntSpout.java | 32 +++-- .../heron/examples/eco/StatefulWindowSumBolt.java | 32 +++-- .../heron/examples/eco/TestFibonacciSpout.java | 32 +++-- .../heron/examples/eco/TestIBasicPrintBolt.java| 32 +++-- .../apache/heron/examples/eco/TestNameCounter.java | 32 +++-- .../apache/heron/examples/eco/TestNameSpout.java | 32 +++-- .../apache/heron/examples/eco/TestPrintBolt.java | 32 +++-- .../heron/examples/eco/TestPropertyHolder.java | 32 +++-- .../org/apache/heron/examples/eco/TestUnits.java | 32 +++-- .../apache/heron/examples/eco/TestWindowBolt.java | 32 +++-- .../org/apache/heron/examples/eco/WordSpout.java | 32 +++-- .../apache/heron/examples/eco/EvenAndOddBolt.java | 32 +++-- .../org/apache/heron/examples/eco/LogInfoBolt.java | 32 +++-- .../heron/examples/eco/TestFibonacciSpout.java | 32 +++-- .../heron/examples/eco/TestIBasicPrintBolt.java| 32 +++-- .../apache/heron/examples/eco/TestNameCounter.java | 32 +++-- .../apache/heron/examples/eco/TestNameSpout.java | 32 +++-- .../apache/heron/examples/eco/TestPrintBolt.java | 32 +++-- .../heron/examples/eco/TestPropertyHolder.java | 32 +++-- .../org/apache/heron/examples/eco/TestUnits.java | 32 +++-- .../apache/heron/examples/eco/TestWindowBolt.java | 32 +++-- eco/src/java/org/apache/heron/eco/Eco.java | 32 +++-- .../org/apache/heron/eco/builder/BoltBuilder.java | 32 +++-- .../apache/heron/eco/builder/BuilderUtility.java | 32 +++-- .../apache/heron/eco/builder/ComponentBuilder.java | 32 +++-- .../apache/heron/eco/builder/ConfigBuilder.java| 32 +++-- .../apache/heron/eco/builder/ObjectBuilder.java| 32 +++-- .../apache/heron/eco/builder/heron/EcoBuilder.java | 32 +++-- .../heron/eco/builder/heron/SpoutBuilder.java | 32 +++-- .../heron/eco/builder/heron/StreamBuilder.java | 32 +++-- .../apache/heron/eco/builder/storm/EcoBuilder.java | 32 +++-- .../heron/eco/builder/storm/SpoutBuilder.java | 32 +++-- .../heron/eco/builder/storm/StreamBuilder.java | 32 +++-- .../heron/eco/definition/BeanDefinition.java | 32 +++-- .../heron/eco/definition/BeanListReference.java| 32 +++-- .../apache/heron/eco/definition/BeanReference.java | 32 +++-- .../heron/eco/definition/BoltDefinition.java | 32 +++-- .../heron/eco/definition/ComponentStream.java | 32 +++-- .../definition/ConfigurationMethodDefinition.java | 32 +++-- .../heron/eco/definition/EcoExecutionContext.java | 32 +++-- .../eco/definition/EcoTopologyDefinition.java | 32 +++-- .../heron/eco/definition/GroupingDefinition.java | 32 +++-- .../heron/eco/definition/ObjectDefinition.java | 32 +++-- .../heron/eco/definition/PropertyDefinition.java | 32 +++-- .../heron/eco/definition/SpoutDefinition.java | 32 +++-- .../heron/eco/definition/StreamDefinition.java | 32 +++-- .../org/apache/heron/eco/parser/EcoParser.java | 32 +++-- .../org/apache/heron/eco/submit/EcoSubmitter.java | 32 +++-- eco/tests/java/org/apache/heron/eco/EcoTest.java | 32 +++-- .../apache/heron/eco/builder/BoltBuilderTest.java | 32 +++-- .../heron/eco/builder/BuilderUtilityTest.java | 32 +++-- .../heron/eco/builder/ComponentBuilderTest.java| 32 +++-- .../heron/eco/builder/ConfigBuilderTest.java | 32 +++-- .../heron/eco/builder/ObjectBuilderTest.java | 32 +++-- .../eco/builder/heron/HeronEcoBuilderTest.java | 32 +++-- .../eco/builder/heron/HeronSpoutBuilderTest.java | 32 +++-- .../eco/builder/heron/HeronStreamBuilderTest.java | 32 +++-- .../eco/builder/storm/StormEcoBuilderTest.java | 32 +++-- .../eco/builder/storm/StormSpoutBuilderTest.java | 32 +++-- .../eco/builder/storm/StormStreamBuilderTest.java | 32 +++-- .../org/apache/heron/eco/parser/EcoParserTest.java | 139 +++-- .../apache/heron/eco/submit/EcoSubmitterTest.java | 32 +++-- examples/src/cpp/ackfail/ackfail-topology.cpp | 25 ++-- .../src/cpp/exclamation/exclamation-topology.cpp | 25 ++-- examples/src/cpp/spouts/test-word-spout.h | 25 ++-- .../apache/heron/exampl
[incubator-heron] branch master updated: package c++ .h and .a files (#2961)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 6be630a package c++ .h and .a files (#2961)
6be630a is described below
commit 6be630aa098800efa26a89d4c19ecbac4903035e
Author: Karthik Ramasamy
AuthorDate: Fri Jul 27 23:48:53 2018 -0700
package c++ .h and .a files (#2961)
* package c++ .h and cpp files
* remove unuwanted files
---
heron/api/src/cpp/BUILD| 45 ++
scripts/packages/BUILD | 277 +
scripts/packages/bin_common.sh | 1 -
scripts/packages/template_bin.sh | 10 +-
tools/applatix/bazel.rc| 40 --
tools/applatix/toolchain/BUILD | 59
tools/applatix/toolchain/CROSSTOOL | 130 -
tools/rules/heron_api.bzl | 12 --
tools/rules/heron_client.bzl | 93 -
tools/rules/heron_core.bzl | 93 -
tools/rules/heron_examples.bzl | 29
tools/rules/heron_tools.bzl| 31 -
12 files changed, 267 insertions(+), 553 deletions(-)
diff --git a/heron/api/src/cpp/BUILD b/heron/api/src/cpp/BUILD
index 0facdb8..6d770d3 100644
--- a/heron/api/src/cpp/BUILD
+++ b/heron/api/src/cpp/BUILD
@@ -2,6 +2,51 @@ licenses(["notice"])
package(default_visibility = ["//visibility:public"])
+filegroup(
+name = "cxx-bolt-hdrs",
+srcs = glob(["bolt/*.h"]),
+)
+
+filegroup(
+name = "cxx-config-hdrs",
+srcs = glob(["config/*.h"]),
+)
+
+filegroup(
+name = "cxx-exceptions-hdrs",
+srcs = glob(["exceptions/*.h"]),
+)
+
+filegroup(
+name = "cxx-metric-hdrs",
+srcs = glob(["metric/*.h"]),
+)
+
+filegroup(
+name = "cxx-serializer-hdrs",
+srcs = glob(["serializer/*.h"]),
+)
+
+filegroup(
+name = "cxx-spout-hdrs",
+srcs = glob(["spout/*.h"]),
+)
+
+filegroup(
+name = "cxx-topology-hdrs",
+srcs = glob(["spout/*.h"]),
+)
+
+filegroup(
+name = "cxx-tuple-hdrs",
+srcs = glob(["tuple/*.h"]),
+)
+
+filegroup(
+name = "cxx-utils-hdrs",
+srcs = glob(["utils/*.h"]),
+)
+
cc_library(
name = "cxx-api",
srcs = glob(["**/*.cpp"]),
diff --git a/scripts/packages/BUILD b/scripts/packages/BUILD
index 7557c33..13173d1 100644
--- a/scripts/packages/BUILD
+++ b/scripts/packages/BUILD
@@ -2,38 +2,6 @@ package(default_visibility = ["//visibility:public"])
load("@bazel_tools//tools/build_defs/pkg:pkg.bzl", "pkg_tar", "pkg_deb")
-load("//tools/rules:heron_client.bzl", "heron_client_bin_files")
-load("//tools/rules:heron_client.bzl", "heron_client_conf_files")
-load("//tools/rules:heron_client.bzl", "heron_client_lib_third_party_files")
-load("//tools/rules:heron_client.bzl", "heron_client_lib_scheduler_files")
-load("//tools/rules:heron_client.bzl",
"heron_client_lib_metricscachemgr_files")
-load("//tools/rules:heron_client.bzl", "heron_client_lib_packing_files")
-load("//tools/rules:heron_client.bzl", "heron_client_lib_statemgr_files")
-load("//tools/rules:heron_client.bzl", "heron_client_lib_uploader_files")
-load("//tools/rules:heron_client.bzl", "heron_client_lib_downloader_files")
-load("//tools/rules:heron_client.bzl", "heron_client_lib_healthmgr_files")
-
-load("//tools/rules:heron_core.bzl", "heron_core_files")
-load("//tools/rules:heron_core.bzl", "heron_core_bin_files")
-load("//tools/rules:heron_core.bzl", "heron_core_conf_files")
-load("//tools/rules:heron_core.bzl", "heron_core_lib_instance_files")
-load("//tools/rules:heron_core.bzl", "heron_core_lib_healthmgr_files")
-load("//tools/rules:heron_core.bzl", "heron_core_lib_metricsmgr_files")
-load("//tools/rules:heron_core.bzl", "heron_core_lib_metricscachemgr_files")
-load("//tools/rules:heron_core.bzl", "heron_core_lib_packing_files")
-load("//tools/rules:heron_core.bzl", "heron_core_lib_scheduler_files")
-load("//tools/rules:heron_core.bzl", "heron_core_lib_statemgr_files")
-load("//tools/rules:heron_core.bzl", "heron_core_lib_ckptmgr_files")
-load("//tools/rules:heron_core.bzl", "heron_core_lib_statefulstorage_files")
-load("//tools/rules:heron_core.bzl", "heron_core_lib_downloader_files&
[incubator-heron] branch master updated: Metrics added to measure the rate at which tuples are added to the queue at spout gateway (#2981)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new d669d06 Metrics added to measure the rate at which tuples are added
to the queue at spout gateway (#2981)
d669d06 is described below
commit d669d0615198ce36a8cdebadd184ba4c3bcc9e49
Author: Faria Kalim
AuthorDate: Fri Aug 17 08:37:51 2018 -0700
Metrics added to measure the rate at which tuples are added to the queue at
spout gateway (#2981)
* added metrics to measure tuples added to outgoing queue per min
* added metric to full metrics instance
* added cumulative outgoing queue size
* measuring number of tuples per tupleset
* added comment justifying use of metrics
* removed extra comment
---
.../heron/common/utils/metrics/BoltMetrics.java| 10 +-
.../common/utils/metrics/ComponentMetrics.java | 2 +-
.../common/utils/metrics/FullBoltMetrics.java | 9 -
.../common/utils/metrics/FullSpoutMetrics.java | 22 --
.../heron/common/utils/metrics/SpoutMetrics.java | 8
.../heron/instance/AbstractOutputCollector.java| 2 +-
.../heron/instance/OutgoingTupleCollection.java| 8 +++-
.../org/apache/heron/metrics/GatewayMetrics.java | 1 -
8 files changed, 54 insertions(+), 8 deletions(-)
diff --git
a/heron/common/src/java/org/apache/heron/common/utils/metrics/BoltMetrics.java
b/heron/common/src/java/org/apache/heron/common/utils/metrics/BoltMetrics.java
index 13006fb..bd40180 100644
---
a/heron/common/src/java/org/apache/heron/common/utils/metrics/BoltMetrics.java
+++
b/heron/common/src/java/org/apache/heron/common/utils/metrics/BoltMetrics.java
@@ -44,7 +44,7 @@ public class BoltMetrics implements ComponentMetrics {
private final CountMetric failCount;
private final CountMetric executeCount;
private final ReducedMetric executeLatency;
-
+ private final CountMetric tupleAddedToQueue;
// Time in nano-seconds spending in execute() at every interval
private final CountMetric emitCount;
@@ -53,6 +53,7 @@ public class BoltMetrics implements ComponentMetrics {
private final CountMetric outQueueFullCount;
+
public BoltMetrics() {
ackCount = new CountMetric();
processLatency = new ReducedMetric<>(new MeanReducer());
@@ -62,6 +63,7 @@ public class BoltMetrics implements ComponentMetrics {
executeLatency = new ReducedMetric<>(new MeanReducer());
emitCount = new CountMetric();
outQueueFullCount = new CountMetric();
+tupleAddedToQueue = new CountMetric();
}
public void registerMetrics(TopologyContextImpl topologyContext) {
@@ -78,6 +80,8 @@ public class BoltMetrics implements ComponentMetrics {
topologyContext.registerMetric("__execute-latency/default",
executeLatency, interval);
topologyContext.registerMetric("__emit-count/default", emitCount,
interval);
topologyContext.registerMetric("__out-queue-full-count",
outQueueFullCount, interval);
+
topologyContext.registerMetric("__data-tuple-added-to-outgoing-queue/default",
+tupleAddedToQueue, interval);
}
// For MultiCountMetrics, we need to set the default value for all streams.
@@ -109,6 +113,10 @@ public class BoltMetrics implements ComponentMetrics {
emitCount.incr();
}
+ public void addTupleToQueue(int size) {
+tupleAddedToQueue.incr();
+ }
+
public void updateOutQueueFullCount() {
outQueueFullCount.incr();
}
diff --git
a/heron/common/src/java/org/apache/heron/common/utils/metrics/ComponentMetrics.java
b/heron/common/src/java/org/apache/heron/common/utils/metrics/ComponentMetrics.java
index 81728f6..d9e3b23 100644
---
a/heron/common/src/java/org/apache/heron/common/utils/metrics/ComponentMetrics.java
+++
b/heron/common/src/java/org/apache/heron/common/utils/metrics/ComponentMetrics.java
@@ -28,7 +28,7 @@ import org.apache.heron.classification.InterfaceStability;
@InterfaceAudience.Private
@InterfaceStability.Evolving
public interface ComponentMetrics {
-
void serializeDataTuple(String streamId, long latency);
void emittedTuple(String streamId);
+ void addTupleToQueue(int size);
}
diff --git
a/heron/common/src/java/org/apache/heron/common/utils/metrics/FullBoltMetrics.java
b/heron/common/src/java/org/apache/heron/common/utils/metrics/FullBoltMetrics.java
index 84700d3..dfda1c5 100644
---
a/heron/common/src/java/org/apache/heron/common/utils/metrics/FullBoltMetrics.java
+++
b/heron/common/src/java/org/apache/heron/common/utils/metrics/FullBoltMetrics.java
@@ -52,6 +52,7 @@ public class FullBoltMetrics extends BoltMetrics {
// Time in nano-seconds spending in execute() at every interval
private final MultiCountMetric executeTimeNs;
private final Mult
[incubator-heron] branch master updated: Extending update command to allow number of containers to be modified at runtime (#2980)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 516eca3 Extending update command to allow number of containers to be
modified at runtime (#2980)
516eca3 is described below
commit 516eca3023a25d341094a48337a677ca64b3c0a2
Author: Faria Kalim
AuthorDate: Fri Aug 17 08:38:24 2018 -0700
Extending update command to allow number of containers to be modified at
runtime (#2980)
* added num containers to heron cli
* bug fix
* updated apiserver to accept container num
* removed bug that did allow correct parallelism to be supplied when new
containers were supplied
* corrected variable in update.py
* bug fix
* removed unused comment
* added cumulative outgoing queue size
* added num containers to heron cli
* bug fix
* updated apiserver to accept container num
* removed bug that did allow correct parallelism to be supplied when new
containers were supplied
* corrected variable in update.py
* bug fix
* removed unused comment
* added cumulative outgoing queue size
* bug fix
* remove extra code
* removed extra code
---
.../binpacking/FirstFitDecreasingPacking.java | 8 +++
.../roundrobin/ResourceCompliantRRPacking.java | 9 +++
.../packing/roundrobin/RoundRobinPacking.java | 25 +--
.../apache/heron/scheduler/RuntimeManagerMain.java | 15 +
.../heron/scheduler/RuntimeManagerRunner.java | 76 --
.../heron/scheduler/RuntimeManagerRunnerTest.java | 4 +-
.../org/apache/heron/spi/packing/IRepacking.java | 14
.../org/apache/heron/apiserver/actions/Keys.java | 2 +
.../apiserver/resources/TopologyResource.java | 16 -
heron/tools/cli/src/python/cli_helper.py | 1 -
heron/tools/cli/src/python/update.py | 43 ++--
11 files changed, 176 insertions(+), 37 deletions(-)
diff --git
a/heron/packing/src/java/org/apache/heron/packing/binpacking/FirstFitDecreasingPacking.java
b/heron/packing/src/java/org/apache/heron/packing/binpacking/FirstFitDecreasingPacking.java
index 85ad638..36bd124 100644
---
a/heron/packing/src/java/org/apache/heron/packing/binpacking/FirstFitDecreasingPacking.java
+++
b/heron/packing/src/java/org/apache/heron/packing/binpacking/FirstFitDecreasingPacking.java
@@ -201,6 +201,14 @@ public class FirstFitDecreasingPacking implements
IPacking, IRepacking {
}
@Override
+ public PackingPlan repack(PackingPlan currentPackingPlan, int containers,
+Map componentChanges)
+ throws PackingException, UnsupportedOperationException {
+throw new UnsupportedOperationException("FirstFitDecreasingPacking does
not currently support"
++ " creating a new packing plan with a new number of containers.");
+ }
+
+ @Override
public void close() {
}
diff --git
a/heron/packing/src/java/org/apache/heron/packing/roundrobin/ResourceCompliantRRPacking.java
b/heron/packing/src/java/org/apache/heron/packing/roundrobin/ResourceCompliantRRPacking.java
index 67f6644..ed7f32c 100644
---
a/heron/packing/src/java/org/apache/heron/packing/roundrobin/ResourceCompliantRRPacking.java
+++
b/heron/packing/src/java/org/apache/heron/packing/roundrobin/ResourceCompliantRRPacking.java
@@ -39,6 +39,7 @@ import org.apache.heron.spi.common.Config;
import org.apache.heron.spi.common.Context;
import org.apache.heron.spi.packing.IPacking;
import org.apache.heron.spi.packing.IRepacking;
+import org.apache.heron.spi.packing.PackingException;
import org.apache.heron.spi.packing.PackingPlan;
import org.apache.heron.spi.packing.Resource;
@@ -227,6 +228,14 @@ public class ResourceCompliantRRPacking implements
IPacking, IRepacking {
}
@Override
+ public PackingPlan repack(PackingPlan currentPackingPlan, int containers,
+Map componentChanges)
+ throws PackingException, UnsupportedOperationException {
+throw new UnsupportedOperationException("ResourceCompliantRRPacking does
not "
++ "currently support creating a new packing plan with a new number of
containers.");
+ }
+
+ @Override
public void close() {
}
diff --git
a/heron/packing/src/java/org/apache/heron/packing/roundrobin/RoundRobinPacking.java
b/heron/packing/src/java/org/apache/heron/packing/roundrobin/RoundRobinPacking.java
index 0b04e69..0a80440 100644
---
a/heron/packing/src/java/org/apache/heron/packing/roundrobin/RoundRobinPacking.java
+++
b/heron/packing/src/java/org/apache/heron/packing/roundrobin/RoundRobinPacking.java
@@ -403,15 +403,32 @@ public class RoundRobinPacking implements IPacking,
IRepacking {
double
[incubator-heron] branch karthik/c++-packaging updated: remove unuwanted files
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch karthik/c++-packaging
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/karthik/c++-packaging by this
push:
new c45b9a5 remove unuwanted files
c45b9a5 is described below
commit c45b9a5ff94c9a0758e68d70ff97ebbbc5c94a04
Author: Karthik Ramasamy
AuthorDate: Fri Jul 20 10:15:08 2018 -0700
remove unuwanted files
---
tools/applatix/bazel.rc| 40
tools/applatix/toolchain/BUILD | 59 -
tools/applatix/toolchain/CROSSTOOL | 130 -
tools/rules/heron_api.bzl | 12
tools/rules/heron_client.bzl | 93 --
tools/rules/heron_core.bzl | 93 --
tools/rules/heron_examples.bzl | 29 -
tools/rules/heron_tools.bzl| 31 -
8 files changed, 487 deletions(-)
diff --git a/tools/applatix/bazel.rc b/tools/applatix/bazel.rc
deleted file mode 100644
index 81d760a..000
--- a/tools/applatix/bazel.rc
+++ /dev/null
@@ -1,40 +0,0 @@
-# This is from Bazel's former travis setup, to avoid blowing up the RAM usage.
-startup --host_jvm_args=-Xmx2500m
-startup --host_jvm_args=-Xms2500m
-startup --batch
-test --ram_utilization_factor=10
-test --test_output=errors
-
-# This is so we understand failures better
-build --verbose_failures
-
-build --show_timestamps
-test --show_timestamps
-
-# Limits the jobs to 25 since the resources are limited
-build --jobs 25
-
-# Link with the appropriate libs
-build --linkopt -lm
-build --linkopt -lpthread
-build --linkopt -lrt
-build --experimental_action_listener=tools/cpp:compile_cpp
-build --experimental_action_listener=tools/java:compile_java
-build --experimental_action_listener=tools/python:compile_python
-build --workspace_status_command scripts/release/status.sh
-
-# This is so we use a recent enough GCC when building.
-# build --crosstool_top //tools/applatix/toolchain:CROSSTOOL
-
-# This is so we don't use sandboxed execution. Sandboxed execution
-# runs stuff in a container, and since Applatix already runs its script
-# in a container this fails to run tests.
-build --spawn_strategy=standalone --genrule_strategy=standalone
-test --test_strategy=standalone
-
-# Ignore unsupported warning for sandboxing
-build --ignore_unsupported_sandboxing
-
-# Below this line, .travis.yml will cat the default bazelrc.
-# This is needed so Bazel starts with the base workspace in its
-# package path.
diff --git a/tools/applatix/toolchain/BUILD b/tools/applatix/toolchain/BUILD
deleted file mode 100644
index 3f369b7..000
--- a/tools/applatix/toolchain/BUILD
+++ /dev/null
@@ -1,59 +0,0 @@
-licenses(["restricted"])
-
-package(default_visibility = ["//visibility:public"])
-
-filegroup(
-name = "empty",
-srcs = [],
-)
-
-# This is the entry point for --crosstool_top. Toolchains are found
-# by lopping off the name of --crosstool_top and searching for
-# "cc-compiler-${CPU}" in this BUILD file, where CPU is the target CPU
-# specified in --cpu.
-#
-# This file group should include
-# * all cc_toolchain targets supported
-# * all file groups that said cc_toolchain might refer to,
-# including the default_grte_top setting in the CROSSTOOL
-# protobuf.
-filegroup(
-name = "toolchain",
-srcs = [
-":cc-compiler-local",
-":empty",
-],
-)
-
-cc_toolchain(
-name = "cc-compiler-local",
-all_files = ":empty",
-compiler_files = ":empty",
-cpu = "local",
-dwp_files = ":empty",
-dynamic_runtime_libs = [":empty"],
-linker_files = ":empty",
-objcopy_files = ":empty",
-static_runtime_libs = [":empty"],
-strip_files = ":empty",
-supports_param_files = 0,
-)
-
-cc_toolchain(
-name = "cc-compiler-k8",
-all_files = ":empty",
-compiler_files = ":empty",
-cpu = "local",
-dwp_files = ":empty",
-dynamic_runtime_libs = [":empty"],
-linker_files = ":empty",
-objcopy_files = ":empty",
-static_runtime_libs = [":empty"],
-strip_files = ":empty",
-supports_param_files = 0,
-)
-
-filegroup(
-name = "srcs",
-srcs = glob(["**"]),
-)
diff --git a/tools/applatix/toolchain/CROSSTOOL
b/tools/applatix/toolchain/CROSSTOOL
deleted file mode 100644
index 78d7e9c..000
--- a/tools/applatix/toolchain/CROSSTOOL
+++ /dev/null
@@ -1,130 +0,0 @@
-major_version: "local"
-minor_version: ""
-default_target_cpu: "same_as_host"
-
-default_toolchain {
- cpu: "k8"
- toolchain_identifier: "local_linux"
[incubator-heron] branch master updated: fixing download issue in Nomad scheduler (#2969)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new c3968d5 fixing download issue in Nomad scheduler (#2969)
c3968d5 is described below
commit c3968d55c5e6c5db9e098f673767e54f976ad3e1
Author: Boyang Jerry Peng
AuthorDate: Fri Jul 20 12:05:59 2018 -0700
fixing download issue in Nomad scheduler (#2969)
* fixing download issue in Nomad scheduler
* fix formatting
---
.../org/apache/heron/scheduler/nomad/NomadScheduler.java | 14 +-
.../apache/heron/scheduler/nomad/NomadSchedulerTest.java | 6 +++---
2 files changed, 12 insertions(+), 8 deletions(-)
diff --git
a/heron/schedulers/src/java/org/apache/heron/scheduler/nomad/NomadScheduler.java
b/heron/schedulers/src/java/org/apache/heron/scheduler/nomad/NomadScheduler.java
index bc2da01..7f7e1c5 100644
---
a/heron/schedulers/src/java/org/apache/heron/scheduler/nomad/NomadScheduler.java
+++
b/heron/schedulers/src/java/org/apache/heron/scheduler/nomad/NomadScheduler.java
@@ -341,7 +341,8 @@ public class NomadScheduler implements IScheduler {
// get complete heron executor command
String executorCmd = executorBinary + " " + String.join(" ", executorArgs);
// get heron_downloader command for downloading topology package
-String topologyDownloadCmd = getFetchCommand(this.clusterConfig,
this.runtimeConfig);
+String topologyDownloadCmd = getFetchCommand(this.clusterConfig,
+this.clusterConfig, this.runtimeConfig);
task.setName(taskName);
// use nomad driver
@@ -382,7 +383,8 @@ public class NomadScheduler implements IScheduler {
// get complete heron executor command
String executorCmd = executorBinary + " " + String.join(" ", executorArgs);
// get heron_downloader command for downloading topology package
-String topologyDownloadCmd = getFetchCommand(this.clusterConfig,
this.runtimeConfig);
+String topologyDownloadCmd = getFetchCommand(this.localConfig,
+this.clusterConfig, this.runtimeConfig);
// read nomad heron executor start up script from file
String heronNomadScript = getHeronNomadScript(this.localConfig);
@@ -549,9 +551,11 @@ public class NomadScheduler implements IScheduler {
/**
* Get the command that will be used to retrieve the topology JAR
*/
- static String getFetchCommand(Config config, Config runtime) {
-return String.format("%s %s .", Context.downloaderBinary(config),
-Runtime.topologyPackageUri(runtime).toString());
+ static String getFetchCommand(Config localConfig, Config clusterConfig,
Config runtime) {
+return String.format("%s -u %s -f . -m local -p %s -d %s",
+Context.downloaderBinary(clusterConfig),
+Runtime.topologyPackageUri(runtime).toString(),
Context.heronConf(localConfig),
+Context.heronHome(clusterConfig));
}
static int longToInt(long val) {
diff --git
a/heron/schedulers/tests/java/org/apache/heron/scheduler/nomad/NomadSchedulerTest.java
b/heron/schedulers/tests/java/org/apache/heron/scheduler/nomad/NomadSchedulerTest.java
index c545616..276d6ff 100644
---
a/heron/schedulers/tests/java/org/apache/heron/scheduler/nomad/NomadSchedulerTest.java
+++
b/heron/schedulers/tests/java/org/apache/heron/scheduler/nomad/NomadSchedulerTest.java
@@ -324,7 +324,7 @@ public class NomadSchedulerTest {
.thenReturn(EXECUTOR_CMD_ARGS);
PowerMockito.mockStatic(NomadScheduler.class);
-PowerMockito.when(NomadScheduler.getFetchCommand(Mockito.any(),
Mockito.any()))
+PowerMockito.when(NomadScheduler.getFetchCommand(Mockito.any(),
Mockito.any(), Mockito.any()))
.thenReturn(TOPOLOGY_DOWNLOAD_CMD);
PowerMockito.when(NomadScheduler.getHeronNomadScript(this.mockConfig))
.thenReturn(HERON_NOMAD_SCRIPT);
@@ -405,7 +405,7 @@ public class NomadSchedulerTest {
.thenReturn(EXECUTOR_CMD_ARGS);
PowerMockito.mockStatic(NomadScheduler.class);
-PowerMockito.when(NomadScheduler.getFetchCommand(Mockito.any(),
Mockito.any()))
+PowerMockito.when(NomadScheduler.getFetchCommand(Mockito.any(),
Mockito.any(), Mockito.any()))
.thenReturn(TOPOLOGY_DOWNLOAD_CMD);
PowerMockito.when(NomadScheduler.getHeronNomadScript(this.mockConfig))
.thenReturn(HERON_NOMAD_SCRIPT);
@@ -485,7 +485,7 @@ public class NomadSchedulerTest {
.thenReturn(EXECUTOR_CMD_ARGS);
PowerMockito.mockStatic(NomadScheduler.class);
-PowerMockito.when(NomadScheduler.getFetchCommand(Mockito.any(),
Mockito.any()))
+PowerMockito.when(NomadScheduler.getFetchCommand(Mockito.any(),
Mockito.any(), Mockito.any()))
.thenReturn(TOPOLOGY_DOWNLOAD_CMD);
PowerMockito.when(NomadScheduler.getHeronNomadScript(this.mockConfig))
.thenReturn(HERON_NOMAD_SCRIPT);
[incubator-heron] branch master updated: Fix javadoc errors (#3017)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 4d2677e Fix javadoc errors (#3017)
4d2677e is described below
commit 4d2677ee6c117d508e66147ef3d089cf256e0298
Author: Ning Wang
AuthorDate: Tue Sep 4 10:28:59 2018 -0700
Fix javadoc errors (#3017)
---
heron/api/src/java/org/apache/heron/api/Config.java | 4 ++--
.../org/apache/heron/api/bolt/IWindowedBolt.java | 2 +-
.../apache/heron/api/windowing/WindowManager.java| 4 ++--
.../evictors/WatermarkTimeEvictionPolicy.java| 2 +-
.../GeneralReduceByKeyAndWindowStreamlet.java| 2 +-
.../streamlet/impl/streamlets/JoinStreamlet.java | 2 +-
.../streamlets/ReduceByKeyAndWindowStreamlet.java| 2 +-
.../apache/heron/scheduler/utils/SchedulerUtils.java | 2 +-
.../org/apache/heron/spi/uploader/IUploader.java | 2 ++
.../java/backtype/storm/task/TopologyContext.java| 1 +
.../org/apache/storm/topology/IWindowedBolt.java | 2 +-
.../src/java/org/apache/storm/tuple/Tuple.java | 20 ++--
12 files changed, 24 insertions(+), 21 deletions(-)
diff --git a/heron/api/src/java/org/apache/heron/api/Config.java
b/heron/api/src/java/org/apache/heron/api/Config.java
index d91c8a0..1607e39 100644
--- a/heron/api/src/java/org/apache/heron/api/Config.java
+++ b/heron/api/src/java/org/apache/heron/api/Config.java
@@ -286,13 +286,13 @@ public class Config extends HashMap {
/**
* Topology-specific environment properties to be added to an Heron instance.
* This is added to the existing environment (that of the Heron instance).
- * This variable contains Map
+ * This variable contains MapString, String
*/
public static final String TOPOLOGY_ENVIRONMENT = "topology.environment";
/**
* Timer events registered for a topology.
- * This is a Map>.
+ * This is a MapString, PairDuration, Runnable.
* Where the key is the name and the value contains the frequency of the
event
* and the task to run.
*/
diff --git a/heron/api/src/java/org/apache/heron/api/bolt/IWindowedBolt.java
b/heron/api/src/java/org/apache/heron/api/bolt/IWindowedBolt.java
index 9e4de7e..97ff7e3 100644
--- a/heron/api/src/java/org/apache/heron/api/bolt/IWindowedBolt.java
+++ b/heron/api/src/java/org/apache/heron/api/bolt/IWindowedBolt.java
@@ -27,7 +27,7 @@ import org.apache.heron.api.windowing.TimestampExtractor;
import org.apache.heron.api.windowing.TupleWindow;
/**
- * A bolt abstraction for supporting time and count based sliding & tumbling
windows.
+ * A bolt abstraction for supporting time and count based sliding
tumbling windows.
*/
public interface IWindowedBolt extends IComponent {
/**
diff --git
a/heron/api/src/java/org/apache/heron/api/windowing/WindowManager.java
b/heron/api/src/java/org/apache/heron/api/windowing/WindowManager.java
index 26c5dce..1222a50 100644
--- a/heron/api/src/java/org/apache/heron/api/windowing/WindowManager.java
+++ b/heron/api/src/java/org/apache/heron/api/windowing/WindowManager.java
@@ -75,7 +75,7 @@ public class WindowManager implements
TriggerHandler {
*
* @param lifecycleListener the {@link WindowLifecycleListener}
* @param queue a collection where the events in the window can be enqueued.
- *
+ *
* Note: This collection has to be thread safe.
*/
public WindowManager(WindowLifecycleListener lifecycleListener,
Collection> queue) {
@@ -88,7 +88,7 @@ public class WindowManager implements
TriggerHandler {
/**
* For testing purpose
- * See {@Link org.apache.heron.api.windowing.WindowManagerTest}
+ * See {@link org.apache.heron.api.windowing.WindowManagerTest}
* @param lifecycleListener
*/
public WindowManager(WindowLifecycleListener lifecycleListener) {
diff --git
a/heron/api/src/java/org/apache/heron/api/windowing/evictors/WatermarkTimeEvictionPolicy.java
b/heron/api/src/java/org/apache/heron/api/windowing/evictors/WatermarkTimeEvictionPolicy.java
index 74f4ffd..adf5e49 100644
---
a/heron/api/src/java/org/apache/heron/api/windowing/evictors/WatermarkTimeEvictionPolicy.java
+++
b/heron/api/src/java/org/apache/heron/api/windowing/evictors/WatermarkTimeEvictionPolicy.java
@@ -56,7 +56,7 @@ public class WatermarkTimeEvictionPolicy extends TimeEvi
/**
* {@inheritDoc}
- *
+ *
* Keeps events with future ts in the queue for processing in the next
* window. If the ts difference is more than the lag, stops scanning
* the queue for the current window.
diff --git
a/heron/api/src/java/org/apache/heron/streamlet/impl/streamlets/GeneralReduceByKeyAndWindowStreamlet.java
b/heron/api/src/java/org/apache/heron/streamlet/impl/streamlets/GeneralReduceByKeyAndWindowStreamlet.java
index 216ec38..2bc6bf0 100644
---
a/heron/api/src/java/org/ap
[incubator-heron] branch master updated: joshfischer/update k8s docs (#2960)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new aca05de joshfischer/update k8s docs (#2960)
aca05de is described below
commit aca05de4e977dc7ea1303ecc000decef7849b6d0
Author: Josh Fischer
AuthorDate: Thu Jul 12 11:48:09 2018 -0500
joshfischer/update k8s docs (#2960)
* change api status url
* further clean up on updated kuberntes versioned urls
* updating missed urls
* further clean up and addig disclaimers
* updating tool template with correct proxy url
* updating notes txt
---
deploy/kubernetes/general/tools.yaml | 2 +-
deploy/kubernetes/helm/templates/NOTES.txt | 8 ++--
deploy/kubernetes/helm/templates/tools.yaml| 2 +-
deploy/kubernetes/minikube/tools.yaml | 2 +-
.../deployment/schedulers/kubernetes-helm.md | 40 ---
.../operators/deployment/schedulers/kubernetes.md | 45 --
6 files changed, 67 insertions(+), 32 deletions(-)
diff --git a/deploy/kubernetes/general/tools.yaml
b/deploy/kubernetes/general/tools.yaml
index 36c903a..d4ab5ce 100644
--- a/deploy/kubernetes/general/tools.yaml
+++ b/deploy/kubernetes/general/tools.yaml
@@ -67,7 +67,7 @@ spec:
- >-
heron-ui
--port=8889
-
--base_url=/api/v1/proxy/namespaces/default/services/heron-ui:8889
+
--base_url=/api/v1/namespaces/default/services/heron-ui:8889/proxy
---
##
diff --git a/deploy/kubernetes/helm/templates/NOTES.txt
b/deploy/kubernetes/helm/templates/NOTES.txt
index c8bb950..fe61182 100644
--- a/deploy/kubernetes/helm/templates/NOTES.txt
+++ b/deploy/kubernetes/helm/templates/NOTES.txt
@@ -4,19 +4,19 @@ You can check the status of Heron by doing the following:
kubectl proxy -p 8001
2. Verify you can access the API server
- curl http://localhost:8001/api/v1/proxy/namespaces/{{ .Release.Namespace
}}/services/{{ .Release.Name }}-apiserver:9000/api/v1/version
+ curl http://localhost:8001/api/v1/namespaces/{{ .Release.Namespace
}}/services/{{ .Release.Name }}-apiserver:9000/proxy/api/v1/version
3. Verify you can access the heron ui
- open http://localhost:8001/api/v1/proxy/namespaces/{{ .Release.Namespace
}}/services/{{ .Release.Name }}-ui:8889
+ open http://localhost:8001/api/v1/namespaces/{{ .Release.Namespace
}}/services/{{ .Release.Name }}-ui:8889/proxy
4. Set your service url
- heron config {{ .Release.Name }} set service_url
http://localhost:8001/api/v1/proxy/namespaces/{{ .Release.Namespace
}}/services/{{ .Release.Name }}-apiserver:9000
+ heron config {{ .Release.Name }} set service_url
http://localhost:8001/api/v1/namespaces/{{ .Release.Namespace }}/services/{{
.Release.Name }}-apiserver:9000/proxy
5. Submit an example topology
heron submit {{ .Release.Name }} ~/.heron/examples/heron-api-examples.jar
org.apache.heron.examples.api.AckingTopology acking
6. Visit the heron ui and see if your example topology is running
- open http://localhost:8001/api/v1/proxy/namespaces/{{ .Release.Namespace
}}/services/{{ .Release.Name }}-ui:8889
+ open http://localhost:8001/api/v1/namespaces/{{ .Release.Namespace
}}/services/{{ .Release.Name }}-ui:8889/proxy
7. You can kill the example topology by
heron kill {{ .Release.Name }} acking
diff --git a/deploy/kubernetes/helm/templates/tools.yaml
b/deploy/kubernetes/helm/templates/tools.yaml
index 7ee348d..2c7b614 100644
--- a/deploy/kubernetes/helm/templates/tools.yaml
+++ b/deploy/kubernetes/helm/templates/tools.yaml
@@ -108,7 +108,7 @@ spec:
- >-
heron-ui
--port=8889
- --base_url=/api/v1/proxy/namespaces/{{ .Release.Namespace
}}/services/{{ .Release.Name }}-ui:8889
+ --base_url=/api/v1/namespaces/{{ .Release.Namespace
}}/services/{{ .Release.Name }}-ui:8889/proxy
- name: heron-apiserver
image: {{ .Values.image }}
imagePullPolicy: {{ .Values.imagePullPolicy }}
diff --git a/deploy/kubernetes/minikube/tools.yaml
b/deploy/kubernetes/minikube/tools.yaml
index 6c8e514..535d9b6 100644
--- a/deploy/kubernetes/minikube/tools.yaml
+++ b/deploy/kubernetes/minikube/tools.yaml
@@ -53,7 +53,7 @@ spec:
- >-
heron-ui
--port=8889
-
--base_url=/api/v1/proxy/namespaces/default/services/heron-ui:8889
+
--base_url=/api/v1/namespaces/default/services/heron-ui:8889/proxy
---
##
diff --git
a/website/content/docs/operators/deployment/schedulers/kubernetes-helm.md
b/website/content/docs/operators/deployment/schedulers/kubernetes-helm.md
index b38c14f..6bb97f4 100644
--- a/website/content/docs/operators/deployment/schedulers/kubernetes-helm.md
+++ b/websit
[incubator-heron] branch master updated: Universal heronpy (#2948)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new fb18dca Universal heronpy (#2948)
fb18dca is described below
commit fb18dca55445cc3ad81b77e9142363d99ffb0012
Author: Oliver Bristow
AuthorDate: Tue Jul 3 17:29:05 2018 +0100
Universal heronpy (#2948)
* Use six to make heronpy tests pass
* Build universal heronpy wheel
* make `bazel.whl` in the //scripts/packages:generate-heronpy-package
---
heronpy/api/tests/python/serializer_unittest.py | 3 ++-
heronpy/api/topology.py | 5 +++--
scripts/packages/BUILD | 6 --
scripts/packages/heronpy/requirements.txt | 1 +
scripts/packages/heronpy/setup.py.template | 3 +++
5 files changed, 13 insertions(+), 5 deletions(-)
diff --git a/heronpy/api/tests/python/serializer_unittest.py
b/heronpy/api/tests/python/serializer_unittest.py
index 0f0c417..65db24a 100644
--- a/heronpy/api/tests/python/serializer_unittest.py
+++ b/heronpy/api/tests/python/serializer_unittest.py
@@ -23,6 +23,7 @@
import unittest
+import six
from heronpy.api.serializer import PythonSerializer
prim_list = [1000, -234, 0.00023, "string",
@@ -39,6 +40,6 @@ class SerializerTest(unittest.TestCase):
# Test with a list of primitive types
for obj in prim_list:
serialized = serializer.serialize(obj)
- self.assertIsInstance(serialized, str)
+ self.assertIsInstance(serialized, six.binary_type)
deserialized = serializer.deserialize(serialized)
self.assertEqual(deserialized, obj)
diff --git a/heronpy/api/topology.py b/heronpy/api/topology.py
index a4c8f94..ae7396f 100644
--- a/heronpy/api/topology.py
+++ b/heronpy/api/topology.py
@@ -26,10 +26,11 @@ import os
import uuid
import heronpy.api.api_constants as api_constants
+import six
+from heronpy.api.component.component_spec import HeronComponentSpec
from heronpy.api.serializer import default_serializer
from heronpy.proto import topology_pb2
-from heronpy.api.component.component_spec import HeronComponentSpec
class TopologyType(type):
"""Metaclass to define a Heron topology in Python"""
@@ -256,6 +257,7 @@ class TopologyType(type):
return sanitized
+@six.add_metaclass(TopologyType)
class Topology(object):
"""Topology is an abstract class for defining a topology
@@ -286,7 +288,6 @@ class Topology(object):
inputs={word_spout:
Grouping.fields('word')},
config={"count_bolt.specific.config":
"another value"})
"""
- __metaclass__ = TopologyType
# pylint: disable=no-member
@classmethod
diff --git a/scripts/packages/BUILD b/scripts/packages/BUILD
index 01bdc2f..7557c33 100644
--- a/scripts/packages/BUILD
+++ b/scripts/packages/BUILD
@@ -486,9 +486,11 @@ genrule(
'rm setup.py.template',
'tree $$HERONPY_DIR',
'/usr/bin/env python2.7 setup.py sdist',
-'/usr/bin/env python2.7 setup.py bdist_wheel',
+'/usr/bin/env python2.7 setup.py bdist_wheel --universal',
+'cd -',
'ls -l $$HERONPY_DIR/dist',
-'cp $$HERONPY_DIR/dist/heronpy-*-py2-*.whl $$OUTPUT_DIR',
+'cp $$HERONPY_DIR/dist/heronpy-*-py2.py3-*.whl $$OUTPUT_DIR',
+'cp $$HERONPY_DIR/dist/heronpy-*-py2.py3-*.whl "$@"',
'cp $$HERONPY_DIR/dist/heronpy-*.tar.gz $$OUTPUT_DIR',
'touch $$OUTPUT_DIR/heronpy.whl',
'rm -rf $$TMP_DIR',
diff --git a/scripts/packages/heronpy/requirements.txt
b/scripts/packages/heronpy/requirements.txt
index 40e388c..6e23522 100644
--- a/scripts/packages/heronpy/requirements.txt
+++ b/scripts/packages/heronpy/requirements.txt
@@ -1 +1,2 @@
protobuf==3.4.0
+six
diff --git a/scripts/packages/heronpy/setup.py.template
b/scripts/packages/heronpy/setup.py.template
index f0de259..45e0dad 100644
--- a/scripts/packages/heronpy/setup.py.template
+++ b/scripts/packages/heronpy/setup.py.template
@@ -35,6 +35,9 @@ setup(
'Intended Audience :: Developers',
'Programming Language :: Python :: 2.7',
+'Programming Language :: Python :: 3.4',
+'Programming Language :: Python :: 3.5',
+'Programming Language :: Python :: 3.6',
],
keywords='heron topology python',
[incubator-heron] branch master updated: Joshfischer/eco stateful topology (#2851)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new ffae7af Joshfischer/eco stateful topology (#2851)
ffae7af is described below
commit ffae7af382984534b824358b4397733267d20eb5
Author: Josh Fischer <j...@joshfischer.io>
AuthorDate: Fri Apr 6 00:04:43 2018 -0500
Joshfischer/eco stateful topology (#2851)
* saving work.
* adding examples for stateful topologies with ECO
---
.../twitter/heron/examples/eco/RandomString.java | 51 +
.../heron/examples/eco/StatefulConsumerBolt.java | 66 +
.../heron/examples/eco/StatefulNumberSpout.java| 83 ++
.../heron/examples/eco/StatefulRandomIntSpout.java | 73 +++
.../heron/examples/eco/StatefulWindowSumBolt.java | 67 +
.../com/twitter/heron/examples/eco/WordSpout.java | 64 +
.../examples/eco/heron-stateful-windowing.yaml | 52 ++
.../examples/eco/heron-stateful-word-count.yaml| 38 ++
8 files changed, 494 insertions(+)
diff --git
a/eco-heron-examples/src/java/com/twitter/heron/examples/eco/RandomString.java
b/eco-heron-examples/src/java/com/twitter/heron/examples/eco/RandomString.java
new file mode 100644
index 000..6ece43c
--- /dev/null
+++
b/eco-heron-examples/src/java/com/twitter/heron/examples/eco/RandomString.java
@@ -0,0 +1,51 @@
+// Copyright 2018 Twitter. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package com.twitter.heron.examples.eco;
+
+import java.util.Random;
+
+public class RandomString {
+ private char[] symbols;
+
+ private Random random = new Random();
+
+ private char[] buf;
+
+ public RandomString(int length) {
+// Construct the symbol set
+StringBuilder tmp = new StringBuilder();
+for (char ch = '0'; ch <= '9'; ++ch) {
+ tmp.append(ch);
+}
+
+for (char ch = 'a'; ch <= 'z'; ++ch) {
+ tmp.append(ch);
+}
+
+symbols = tmp.toString().toCharArray();
+if (length < 1) {
+ throw new IllegalArgumentException("length < 1: " + length);
+}
+
+buf = new char[length];
+ }
+
+ public String nextString() {
+for (int idx = 0; idx < buf.length; ++idx) {
+ buf[idx] = symbols[random.nextInt(symbols.length)];
+}
+
+return new String(buf);
+ }
+}
diff --git
a/eco-heron-examples/src/java/com/twitter/heron/examples/eco/StatefulConsumerBolt.java
b/eco-heron-examples/src/java/com/twitter/heron/examples/eco/StatefulConsumerBolt.java
new file mode 100644
index 000..bd5cbfe
--- /dev/null
+++
b/eco-heron-examples/src/java/com/twitter/heron/examples/eco/StatefulConsumerBolt.java
@@ -0,0 +1,66 @@
+// Copyright 2018 Twitter. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package com.twitter.heron.examples.eco;
+
+import java.util.Map;
+
+import com.twitter.heron.api.bolt.BaseRichBolt;
+import com.twitter.heron.api.bolt.OutputCollector;
+import com.twitter.heron.api.state.State;
+import com.twitter.heron.api.topology.IStatefulComponent;
+import com.twitter.heron.api.topology.OutputFieldsDeclarer;
+import com.twitter.heron.api.topology.TopologyContext;
+import com.twitter.heron.api.tuple.Tuple;
+
+
+public class StatefulConsumerBolt extends BaseRichBolt
+implements IStatefulComponent<Integer, Integer> {
+ private static final long serialVersionUID = -5470591933906954522L;
+
+ private OutputCollector collector;
+ private State<Integer, Integer> myState;
+
+ @Override
+ public void initState(State<Integer, Integer> state) {
+this.myState = state;
+ }
+
+ @Override
+ public void preSave(String checkpointId) {
+// Nothing really since we o
[incubator-heron] branch master updated: expose heron metrics as a service in Nomad (#2859)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 2a4c1c1 expose heron metrics as a service in Nomad (#2859)
2a4c1c1 is described below
commit 2a4c1c1613bc2d10851503fcc35c571618a28ec1
Author: Boyang Jerry Peng <jerry.boyang.p...@gmail.com>
AuthorDate: Sat Apr 7 08:58:54 2018 -0700
expose heron metrics as a service in Nomad (#2859)
---
heron/config/src/yaml/conf/nomad/heron_nomad.sh| 3 +
.../config/src/yaml/conf/nomad/metrics_sinks.yaml | 3 +-
heron/config/src/yaml/conf/nomad/scheduler.yaml| 17
.../config/src/yaml/conf/standalone/heron_nomad.sh | 3 +
.../src/yaml/conf/standalone/metrics_sinks.yaml| 3 +-
.../config/src/yaml/conf/standalone/scheduler.yaml | 14 +++
.../standalone/templates/scheduler.template.yaml | 14 +++
.../heron/metricsmgr/sink/AbstractWebSink.java | 7 +-
heron/schedulers/src/java/BUILD| 1 +
.../heron/scheduler/nomad/NomadConstants.java | 5 +
.../heron/scheduler/nomad/NomadContext.java| 40 +++-
.../heron/scheduler/nomad/NomadScheduler.java | 95 -
.../heron/scheduler/nomad/NomadSchedulerTest.java | 113 -
13 files changed, 305 insertions(+), 13 deletions(-)
diff --git a/heron/config/src/yaml/conf/nomad/heron_nomad.sh
b/heron/config/src/yaml/conf/nomad/heron_nomad.sh
index 51ca593..826ba0d 100644
--- a/heron/config/src/yaml/conf/nomad/heron_nomad.sh
+++ b/heron/config/src/yaml/conf/nomad/heron_nomad.sh
@@ -25,6 +25,9 @@ fi
# download and extract heron topology package
${HERON_TOPOLOGY_DOWNLOAD_CMD}
+# set metrics port file
+echo ${NOMAD_PORT_metrics_port} > ${METRICS_PORT_FILE}
+
# launch heron executor
trap 'kill -TERM $PID' TERM INT
${HERON_EXECUTOR_CMD} &
diff --git a/heron/config/src/yaml/conf/nomad/metrics_sinks.yaml
b/heron/config/src/yaml/conf/nomad/metrics_sinks.yaml
index b5257e2..a70f43a 100644
--- a/heron/config/src/yaml/conf/nomad/metrics_sinks.yaml
+++ b/heron/config/src/yaml/conf/nomad/metrics_sinks.yaml
@@ -60,7 +60,8 @@ tmaster-sink:
prometheus-sink:
class: "com.twitter.heron.metricsmgr.sink.PrometheusSink"
- port: 8080 # The port on which to run (either port or port-file are
mandatory)
+# port: 8080 # The port on which to run (either port or port-file are
mandatory)
+ port-file: "port_file"
path: /metrics # The path on which to publish the metrics (mandatory)
flat-metrics: true # By default the web-sink will publish a flat "name ->
value" json map
include-topology-name: true # Include topology name in metric name (default
false)
diff --git a/heron/config/src/yaml/conf/nomad/scheduler.yaml
b/heron/config/src/yaml/conf/nomad/scheduler.yaml
index 297afc4..21d4268 100644
--- a/heron/config/src/yaml/conf/nomad/scheduler.yaml
+++ b/heron/config/src/yaml/conf/nomad/scheduler.yaml
@@ -26,3 +26,20 @@ heron.nomad.driver: "docker"
# The docker image to use for heron if the docker driver is used,
heron.executor.docker.image: 'heron/heron:latest'
+
+# Set networking mode networking when driver is docker
+heron.nomad.network.mode: "default"
+
+# whether to register metrics service endpoints for prometheus metrics sink in
consul
+# the service will be named in the format:
metrics-heron--
+heron.nomad.metrics.service.register: True
+
+# interval at which health checks should be conducted for metrics service
endpoint
+heron.nomad.metrics.service.check.interval.sec: 10
+
+# timeout of metrics service endpoint health check
+heron.nomad.metrics.service.check.timeout.sec: 2
+
+# additional tags to be attached to metrics service
+# A tag of - with be automaticallu attached
+heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
\ No newline at end of file
diff --git a/heron/config/src/yaml/conf/standalone/heron_nomad.sh
b/heron/config/src/yaml/conf/standalone/heron_nomad.sh
index 51ca593..826ba0d 100644
--- a/heron/config/src/yaml/conf/standalone/heron_nomad.sh
+++ b/heron/config/src/yaml/conf/standalone/heron_nomad.sh
@@ -25,6 +25,9 @@ fi
# download and extract heron topology package
${HERON_TOPOLOGY_DOWNLOAD_CMD}
+# set metrics port file
+echo ${NOMAD_PORT_metrics_port} > ${METRICS_PORT_FILE}
+
# launch heron executor
trap 'kill -TERM $PID' TERM INT
${HERON_EXECUTOR_CMD} &
diff --git a/heron/config/src/yaml/conf/standalone/metrics_sinks.yaml
b/heron/config/src/yaml/conf/standalone/metrics_sinks.yaml
index b5257e2..a70f43a 100644
--- a/heron/config/src/yaml/conf/standalone/metrics_sinks.yaml
+++ b/heron/config/src/yaml/conf/standalone/metrics_sinks.yaml
@@ -60,7 +60,8 @@ tmaster-sink:
prometheus-sink:
class: "com.twitter.heron.metricsmgr.sink.PrometheusSink"
- port: 8080 #
[incubator-heron] branch master updated: Eco for heron topologies (#2849)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 98aa2a2 Eco for heron topologies (#2849)
98aa2a2 is described below
commit 98aa2a22ab655a97e4a0f4a9cc87ad49981f2947
Author: Karthik Ramasamy <kart...@streaml.io>
AuthorDate: Thu Apr 5 07:35:48 2018 -0700
Eco for heron topologies (#2849)
* added cereal serialization/deserialization library
* initial restructuring for heron eco
* support for heron eco
* add the pivot
* fixing storm submitter checkstyles
* fixing eco checkstyles
* checkstyles
* added heron eco tests
* fixing check styles
* add submitter test
* fixing ecoSubmitter test
* fixing eco submitter test for storm topology
* clean up
* adding test to verify heron topology submitter
* adding test to verify heron topology submitter
* fixing checkstyles?
* fixed the check style
* added the check for topology type to be rigorous
* added heron eco examples
* modify travis and intellij scripts
* fix checkstyle issues
---
.../src/java/BUILD | 2 +-
.../twitter/heron/examples/eco/EvenAndOddBolt.java | 14 +--
.../twitter/heron/examples/eco/LogInfoBolt.java| 8 +-
.../heron/examples/eco/TestFibonacciSpout.java | 15 +--
.../heron/examples/eco/TestIBasicPrintBolt.java| 14 +--
.../heron/examples/eco/TestNameCounter.java| 12 +-
.../twitter/heron/examples/eco/TestNameSpout.java | 18 +--
.../twitter/heron/examples/eco/TestPrintBolt.java | 8 +-
.../heron/examples/eco/TestPropertyHolder.java | 0
.../com/twitter/heron/examples/eco/TestUnits.java | 0
.../twitter/heron/examples/eco/TestWindowBolt.java | 14 +--
.../heron/examples/eco/heron_fibonacci.yaml| 5 +-
.../heron/examples/eco/heron_windowing.yaml| 9 +-
.../heron/examples/eco/heron_wordcount.yaml| 5 +-
.../twitter/heron/examples/eco/sample.properties | 0
.../src/java/BUILD | 10 +-
.../twitter/heron/examples/eco/EvenAndOddBolt.java | 0
.../twitter/heron/examples/eco/LogInfoBolt.java| 0
.../heron/examples/eco/TestFibonacciSpout.java | 0
.../heron/examples/eco/TestIBasicPrintBolt.java| 0
.../heron/examples/eco/TestNameCounter.java| 0
.../twitter/heron/examples/eco/TestNameSpout.java | 0
.../twitter/heron/examples/eco/TestPrintBolt.java | 0
.../heron/examples/eco/TestPropertyHolder.java | 0
.../com/twitter/heron/examples/eco/TestUnits.java | 0
.../twitter/heron/examples/eco/TestWindowBolt.java | 0
.../com/twitter/heron/examples/eco/fibonacci.yaml | 0
.../twitter/heron/examples/eco/sample.properties | 0
.../heron/examples/eco/simple_windowing.yaml | 0
.../heron/examples/eco/simple_wordcount.yaml | 0
eco/src/java/BUILD | 53 -
eco/src/java/com/twitter/heron/eco/Eco.java| 87 +-
.../com/twitter/heron/eco/builder/BoltBuilder.java | 2 +-
.../heron/eco/builder/ComponentBuilder.java| 2 +-
.../twitter/heron/eco/builder/ConfigBuilder.java | 7 +-
.../heron/eco/builder/{ => heron}/EcoBuilder.java | 11 +-
.../eco/builder/{ => heron}/SpoutBuilder.java | 8 +-
.../eco/builder/{ => heron}/StreamBuilder.java | 20 ++--
.../heron/eco/builder/{ => storm}/EcoBuilder.java | 7 +-
.../eco/builder/{ => storm}/SpoutBuilder.java | 4 +-
.../eco/builder/{ => storm}/StreamBuilder.java | 4 +-
.../eco/definition/EcoTopologyDefinition.java | 18 ++-
.../com/twitter/heron/eco/submit/EcoSubmitter.java | 18 ++-
eco/tests/java/BUILD | 39 ++-
eco/tests/java/com/twitter/heron/eco/EcoTest.java | 48
.../HeronEcoBuilderTest.java} | 12 +-
.../HeronSpoutBuilderTest.java}| 16 +--
.../HeronStreamBuilderTest.java} | 46
.../StormEcoBuilderTest.java} | 9 +-
.../StormSpoutBuilderTest.java}| 6 +-
.../StormStreamBuilderTest.java} | 6 +-
.../twitter/heron/eco/parser/EcoParserTest.java| 2 +
.../twitter/heron/eco/submit/EcoSubmitterTest.java | 24 +++-
scripts/get_all_heron_paths.sh | 4 +-
scripts/travis/build.sh| 6 +-
third_party/cereal/BUILD | 126 +
third_party/cereal/cereal-1.2.1.tar.gz | Bin 0 -> 301689 bytes
third_party/cereal/empty.cc| 0
tools/rules/heron_examples.bzl |
[incubator-heron] branch master updated: [Streamlet Scala API] Add Scala Streamlet Integration Tests Part II (#2861)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new f444372 [Streamlet Scala API] Add Scala Streamlet Integration Tests
Part II (#2861)
f444372 is described below
commit f444372de2749c8a6953ff98a5178550ec026cd8
Author: Eren Avsarogullari <erenavsarogull...@gmail.com>
AuthorDate: Sat Apr 14 00:37:28 2018 +0100
[Streamlet Scala API] Add Scala Streamlet Integration Tests Part II (#2861)
* Scala Streamlet Integration Tests Part II
* Merge conflicts have been fixed
---
.../src/python/test_runner/resources/test.json | 5 ++
.../common/ScalaIntegrationTestBase.scala | 2 +-
.../ScalaStreamletWithFilterAndTransform.scala | 1 -
...reamletWithMapAndFlatMapAndFilterAndClone.scala | 74 ++
...tWithMapAndFlatMapAndFilterAndCloneResults.json | 1 +
5 files changed, 81 insertions(+), 2 deletions(-)
diff --git a/integration_test/src/python/test_runner/resources/test.json
b/integration_test/src/python/test_runner/resources/test.json
index 7d7cf2b..1174154 100644
--- a/integration_test/src/python/test_runner/resources/test.json
+++ b/integration_test/src/python/test_runner/resources/test.json
@@ -12,6 +12,11 @@
"topologyName" : "IntegrationTest_ScalaStreamletWithFilterAndTransform",
"classPath":
"scala_streamlet_with_filter_and_transform.ScalaStreamletWithFilterAndTransform",
"expectedResultRelativePath" :
"scala_streamlet_with_filter_and_transform/ScalaStreamletWithFilterAndTransformResults.json"
+},
+{
+ "topologyName" :
"IntegrationTest_ScalaStreamletWithMapAndFlatMapAndFilterAndClone",
+ "classPath":
"scala_streamlet_with_map_and_flatmap_and_filter_and_clone.ScalaStreamletWithMapAndFlatMapAndFilterAndClone",
+ "expectedResultRelativePath" :
"scala_streamlet_with_map_and_flatmap_and_filter_and_clone/ScalaStreamletWithMapAndFlatMapAndFilterAndCloneResults.json"
}
],
"javaTopologies": [
diff --git
a/integration_test/src/scala/org/apache/heron/integration_test/common/ScalaIntegrationTestBase.scala
b/integration_test/src/scala/org/apache/heron/integration_test/common/ScalaIntegrationTestBase.scala
index 0b36d6a..915233f 100644
---
a/integration_test/src/scala/org/apache/heron/integration_test/common/ScalaIntegrationTestBase.scala
+++
b/integration_test/src/scala/org/apache/heron/integration_test/common/ScalaIntegrationTestBase.scala
@@ -21,7 +21,7 @@ import org.apache.heron.streamlet.scala.impl.BuilderImpl
/**
* Scala Integration Test Base
*/
-trait ScalaIntegrationTestBase extends Serializable {
+trait ScalaIntegrationTestBase {
protected def build(testTopologyBuilder: TestTopologyBuilder,
streamletBuilder: Builder): TestTopologyBuilder = {
diff --git
a/integration_test/src/scala/org/apache/heron/integration_test/topology/scala_streamlet_with_filter_and_transform/ScalaStreamletWithFilterAndTransform.scala
b/integration_test/src/scala/org/apache/heron/integration_test/topology/scala_streamlet_with_filter_and_transform/ScalaStreamletWithFilterAndTransform.scala
index 023f57e..7c1d7e8 100644
---
a/integration_test/src/scala/org/apache/heron/integration_test/topology/scala_streamlet_with_filter_and_transform/ScalaStreamletWithFilterAndTransform.scala
+++
b/integration_test/src/scala/org/apache/heron/integration_test/topology/scala_streamlet_with_filter_and_transform/ScalaStreamletWithFilterAndTransform.scala
@@ -38,7 +38,6 @@ object ScalaStreamletWithFilterAndTransform {
/**
* Scala Streamlet Integration Test
*/
-@SerialVersionUID(-7280407024398984674L)
class ScalaStreamletWithFilterAndTransform(args: Array[String])
extends AbstractTestTopology(args)
with ScalaIntegrationTestBase {
diff --git
a/integration_test/src/scala/org/apache/heron/integration_test/topology/scala_streamlet_with_map_and_flatmap_and_filter_and_clone/ScalaStreamletWithMapAndFlatMapAndFilterAndClone.scala
b/integration_test/src/scala/org/apache/heron/integration_test/topology/scala_streamlet_with_map_and_flatmap_and_filter_and_clone/ScalaStreamletWithMapAndFlatMapAndFilterAndClone.scala
new file mode 100644
index 000..2742b99
--- /dev/null
+++
b/integration_test/src/scala/org/apache/heron/integration_test/topology/scala_streamlet_with_map_and_flatmap_and_filter_and_clone/ScalaStreamletWithMapAndFlatMapAndFilterAndClone.scala
@@ -0,0 +1,74 @@
+// Copyright 2018 Twitter. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://w
[incubator-heron] branch master updated: Add sha256 checksums for all http_archives (#2856)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 56bcc6d Add sha256 checksums for all http_archives (#2856) 56bcc6d is described below commit 56bcc6d4139a8d64fd6c84d40b0785b1f8c177cb Author: Andrew Jorgensen <ajorgen...@users.noreply.github.com> AuthorDate: Sat Apr 7 12:03:04 2018 -0400 Add sha256 checksums for all http_archives (#2856) I have taken the current sha256 checksum for all of the workspace archives so we can verify their signature on download. Since some of the artifacts are downloaded over http, we want to make sure there was no man in the middle attack done to change the resulting binary. This also gives assurance that the code we are downloading has not been tampered with in any way either over the wire or at the source. --- WORKSPACE | 23 --- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/WORKSPACE b/WORKSPACE index 0e87770..c0c40bb 100644 --- a/WORKSPACE +++ b/WORKSPACE @@ -792,7 +792,8 @@ new_http_archive( "data = glob(['**/*']),", "visibility = ['//visibility:public'],", ")", -]) +]), +sha256 = "02f8102c2436bb03b3ee6dede1919d1dac8a427541652e5ec95171ec8adbc93a", ) # end pex repos @@ -801,6 +802,7 @@ http_archive( name = "com_google_protobuf", urls = ["https://github.com/google/protobuf/archive/v3.4.1.tar.gz;], strip_prefix = "protobuf-3.4.1", +sha256 = "8e0236242106e680b4f9f576cc44b8cd711e948b20a9fc07769b0a20ceab9cc4", ) # end protobuf dependencies for C++ and Java @@ -809,6 +811,7 @@ http_archive( name = "com_github_gflags_gflags", urls = ["https://github.com/gflags/gflags/archive/v2.2.1.tar.gz;], strip_prefix = "gflags-2.2.1", +sha256 = "ae27cdbcd6a2f935baa78e4f21f675649271634c092b1be01469440495609d0e", ) new_http_archive( @@ -816,6 +819,7 @@ new_http_archive( urls = ["https://github.com/libevent/libevent/releases/download/release-2.1.8-stable/libevent-2.1.8-stable.tar.gz;], strip_prefix = "libevent-2.1.8-stable", build_file = "third_party/libevent/libevent.BUILD", +sha256 = "965cc5a8bb46ce4199a47e9b2c9e1cae3b137e8356ffdad6d94d3b9069b71dc2", ) new_http_archive( @@ -823,6 +827,7 @@ new_http_archive( urls = ["http://download.savannah.nongnu.org/releases/libunwind/libunwind-1.1.tar.gz;], strip_prefix = "libunwind-1.1", build_file = "third_party/libunwind/libunwind.BUILD", +sha256 = "9dfe0fcae2a866de9d3942c66995e4b460230446887dbdab302d41a8aee8d09a", ) new_http_archive( @@ -834,6 +839,7 @@ new_http_archive( ], strip_prefix = "zookeeper-3.4.10", build_file = "third_party/zookeeper/zookeeper.BUILD", +sha256 = "7f7f5414e044ac11fee2a1e0bc225469f51fb0cdf821e67df762a43098223f27", ) new_http_archive( @@ -841,6 +847,7 @@ new_http_archive( urls = ["https://github.com/gperftools/gperftools/releases/download/gperftools-2.4/gperftools-2.4.tar.gz;], strip_prefix = "gperftools-2.4", build_file = "third_party/gperftools/gperftools.BUILD", +sha256 = "982a37226eb42f40714e26b8076815d5ea677a422fb52ff8bfca3704d9c30a2d", ) new_http_archive( @@ -848,6 +855,7 @@ new_http_archive( urls = ["https://github.com/google/glog/archive/v0.3.5.tar.gz;], strip_prefix = "glog-0.3.5", build_file = "third_party/glog/glog.BUILD", +sha256 = "7580e408a2c0b5a89ca214739978ce6ff480b5e7d8d7698a2aa92fadc484d1e0", ) new_http_archive( @@ -855,6 +863,7 @@ new_http_archive( urls = ["https://github.com/google/googletest/archive/release-1.8.0.tar.gz;], strip_prefix = "googletest-release-1.8.0", build_file = "third_party/gtest/gtest.BUILD", +sha256 = "58a6f4277ca2bc8565222b3bbd58a177609e9c488e8a72649359ba51450db7d8", ) new_http_archive( @@ -862,6 +871,7 @@ new_http_archive( urls = ["https://github.com/USCiLab/cereal/archive/v1.2.2.tar.gz;], strip_prefix = "cereal-1.2.2", build_file = "third_party/cereal/cereal.BUILD", +sha256 = "1921f26d2e1daf9132da3c432e2fd02093ecaedf846e65d7679ddf868c7289c4", ) new_http_archive( @@ -869,6 +879,7 @@ new_http_archive( urls = ["https://storage.googleapis.com/heron-packages/yaml-cpp-noboost.tar.gz;], strip_prefix = "yaml-cpp-noboost", build_file = "third_party/yaml-cpp/yaml.BUILD", +sha256 = "87684ea7a640733e2fe2f753348e306a6764c88fe3f30518c3273e367ff
[incubator-heron] branch master updated: [Streamlet Scala API] Add Scala Streamlet Integration Tests Part I (#2826)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new ad29cc3 [Streamlet Scala API] Add Scala Streamlet Integration Tests
Part I (#2826)
ad29cc3 is described below
commit ad29cc36d6f5e3e55a72dd20c07e0ffad0c02a88
Author: Eren Avsarogullari <erenavsarogull...@gmail.com>
AuthorDate: Sun Apr 8 00:07:09 2018 +0100
[Streamlet Scala API] Add Scala Streamlet Integration Tests Part I (#2826)
* Scala Streamlet Integration Tests Part I
* Minor change is applied to TextTransformer
* Fix Scala Streamlet Integration Test Packaging Problem
* Fix Scala Streamlet Integration Test Result Problem
* Add Fix for Travis
* Add Fix for Travis
---
.gitignore | 3 +
.../twitter/heron/streamlet/impl/BuilderImpl.java | 6 +-
.../streamlet/scala/SerializableTransformer.scala | 2 -
.../heron/streamlet/scala/impl/BuilderImpl.scala | 3 +
integration_test/src/python/test_runner/main.py| 8 ++-
.../src/python/test_runner/resources/test.json | 7 +++
integration_test/src/scala/BUILD | 28 +
.../common/ScalaIntegrationTestBase.scala | 32 ++
.../ScalaStreamletWithFilterAndTransform.scala | 71 ++
...calaStreamletWithFilterAndTransformResults.json | 1 +
scripts/applatix/javatests.sh | 19 +-
scripts/applatix/test.sh | 19 +-
scripts/packages/BUILD | 11
scripts/run_integration_test.sh| 9 +++
scripts/travis/test.sh | 19 +-
15 files changed, 226 insertions(+), 12 deletions(-)
diff --git a/.gitignore b/.gitignore
index 927928b..12fbf25 100644
--- a/.gitignore
+++ b/.gitignore
@@ -132,3 +132,6 @@ website/public/
# Visual Studio Code
.vscode
+
+# integration_test
+results/
\ No newline at end of file
diff --git
a/heron/api/src/java/com/twitter/heron/streamlet/impl/BuilderImpl.java
b/heron/api/src/java/com/twitter/heron/streamlet/impl/BuilderImpl.java
index e00eb14..b982453 100644
--- a/heron/api/src/java/com/twitter/heron/streamlet/impl/BuilderImpl.java
+++ b/heron/api/src/java/com/twitter/heron/streamlet/impl/BuilderImpl.java
@@ -11,10 +11,8 @@
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
-
package com.twitter.heron.streamlet.impl;
-
import java.util.HashSet;
import java.util.LinkedList;
import java.util.List;
@@ -60,6 +58,10 @@ public final class BuilderImpl implements Builder {
*/
public TopologyBuilder build() {
TopologyBuilder builder = new TopologyBuilder();
+return build(builder);
+ }
+
+ public TopologyBuilder build(TopologyBuilder builder) {
Set stageNames = new HashSet<>();
for (StreamletImpl streamlet : sources) {
streamlet.build(builder, stageNames);
diff --git
a/heron/api/src/scala/com/twitter/heron/streamlet/scala/SerializableTransformer.scala
b/heron/api/src/scala/com/twitter/heron/streamlet/scala/SerializableTransformer.scala
index 7867448..7bbbdd2 100644
---
a/heron/api/src/scala/com/twitter/heron/streamlet/scala/SerializableTransformer.scala
+++
b/heron/api/src/scala/com/twitter/heron/streamlet/scala/SerializableTransformer.scala
@@ -13,8 +13,6 @@
// limitations under the License.
package com.twitter.heron.streamlet.scala
-import java.io.Serializable
-
import com.twitter.heron.streamlet.Context
/**
diff --git
a/heron/api/src/scala/com/twitter/heron/streamlet/scala/impl/BuilderImpl.scala
b/heron/api/src/scala/com/twitter/heron/streamlet/scala/impl/BuilderImpl.scala
index d6cd4c8..4458d70 100644
---
a/heron/api/src/scala/com/twitter/heron/streamlet/scala/impl/BuilderImpl.scala
+++
b/heron/api/src/scala/com/twitter/heron/streamlet/scala/impl/BuilderImpl.scala
@@ -37,4 +37,7 @@ class BuilderImpl(builder:
com.twitter.heron.streamlet.Builder)
def build(): TopologyBuilder =
builder.asInstanceOf[JavaBuilderImpl].build()
+ def build(topologyBuilder: TopologyBuilder): TopologyBuilder =
+builder.asInstanceOf[JavaBuilderImpl].build(topologyBuilder)
+
}
diff --git a/integration_test/src/python/test_runner/main.py
b/integration_test/src/python/test_runner/main.py
index 861a951..f5d4d1c 100644
--- a/integration_test/src/python/test_runner/main.py
+++ b/integration_test/src/python/test_runner/main.py
@@ -293,11 +293,15 @@ def run_tests(conf, args):
http_server_host_port = "%s:%d" % (args.http_server_hostname,
args.http_server_port)
- if args.tests_bin_path.endswith(".jar"):
+ if args.tests_bin_path.endswith("scala-integrat
[incubator-heron] branch master updated: k8s ensure response body is closed after reading an error. (#2863)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new df58705 k8s ensure response body is closed after reading an error.
(#2863)
df58705 is described below
commit df58705cb63d24cd717da2ad598c901f414b0b84
Author: cckellogg <cckell...@gmail.com>
AuthorDate: Tue Apr 17 23:21:36 2018 -0700
k8s ensure response body is closed after reading an error. (#2863)
* k8s ensure response body is closed after reading an error.
* Add another closing case.
---
.../java/org/apache/heron/scheduler/kubernetes/KubernetesUtils.java | 4
1 file changed, 4 insertions(+)
diff --git
a/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesUtils.java
b/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesUtils.java
index c4613b3..7f923f0 100644
---
a/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesUtils.java
+++
b/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesUtils.java
@@ -20,6 +20,7 @@ import java.util.logging.Logger;
import com.squareup.okhttp.Response;
import org.apache.heron.common.basics.ByteAmount;
+import org.apache.heron.common.basics.SysUtils;
import org.apache.heron.scheduler.utils.Runtime;
import org.apache.heron.spi.common.Config;
import org.apache.heron.spi.common.Context;
@@ -41,6 +42,7 @@ final class KubernetesUtils {
log.log(Level.SEVERE, "Error details:\n" + response.body().string());
} catch (IOException ioe) {
// ignore
+ SysUtils.closeIgnoringExceptions(response.body());
}
}
@@ -59,6 +61,8 @@ final class KubernetesUtils {
} catch (IOException ioe) {
// ignore
details = ioe.getMessage();
+} finally {
+ SysUtils.closeIgnoringExceptions(response.body());
}
return message + "\ndetails:\n" + details;
}
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch master updated: improving nomad launch script (#2862)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 45b953b improving nomad launch script (#2862)
45b953b is described below
commit 45b953bae3cae403abc1407f7d8fd635b3833c8b
Author: Boyang Jerry Peng <jerry.boyang.p...@gmail.com>
AuthorDate: Tue Apr 17 23:22:50 2018 -0700
improving nomad launch script (#2862)
---
heron/config/src/yaml/conf/nomad/heron_nomad.sh | 5 +
heron/config/src/yaml/conf/standalone/heron_nomad.sh | 5 +
2 files changed, 10 insertions(+)
diff --git a/heron/config/src/yaml/conf/nomad/heron_nomad.sh
b/heron/config/src/yaml/conf/nomad/heron_nomad.sh
index 826ba0d..8524702 100644
--- a/heron/config/src/yaml/conf/nomad/heron_nomad.sh
+++ b/heron/config/src/yaml/conf/nomad/heron_nomad.sh
@@ -7,6 +7,11 @@ CORE_RELEASE_FILE="heron-core.tar.gz"
CORE_RELEASE_DIR="heron-core"
TOPOLOGY_PACKAGE_FILE="topology.tar.gz"
+# remove directory if already exists
+if [ -d "$HERON_NOMAD_WORKING_DIR" ]; then
+ rm -rf $HERON_NOMAD_WORKING_DIR
+fi
+
# Create working directory if it does not exist
mkdir -p ${HERON_NOMAD_WORKING_DIR}
diff --git a/heron/config/src/yaml/conf/standalone/heron_nomad.sh
b/heron/config/src/yaml/conf/standalone/heron_nomad.sh
index 826ba0d..8524702 100644
--- a/heron/config/src/yaml/conf/standalone/heron_nomad.sh
+++ b/heron/config/src/yaml/conf/standalone/heron_nomad.sh
@@ -7,6 +7,11 @@ CORE_RELEASE_FILE="heron-core.tar.gz"
CORE_RELEASE_DIR="heron-core"
TOPOLOGY_PACKAGE_FILE="topology.tar.gz"
+# remove directory if already exists
+if [ -d "$HERON_NOMAD_WORKING_DIR" ]; then
+ rm -rf $HERON_NOMAD_WORKING_DIR
+fi
+
# Create working directory if it does not exist
mkdir -p ${HERON_NOMAD_WORKING_DIR}
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch master updated: Add extra info when failed to create new instance of class (#2867)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new b325f85 Add extra info when failed to create new instance of class
(#2867)
b325f85 is described below
commit b325f8572f7b4452a4363f46fb2aff955b5ad58e
Author: Ning Wang <nw...@twitter.com>
AuthorDate: Thu Apr 19 21:26:01 2018 -0700
Add extra info when failed to create new instance of class (#2867)
---
storm-compatibility/src/java/backtype/storm/utils/Utils.java | 2 +-
storm-compatibility/src/java/org/apache/storm/utils/Utils.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/storm-compatibility/src/java/backtype/storm/utils/Utils.java
b/storm-compatibility/src/java/backtype/storm/utils/Utils.java
index 2eb6a3c..b1d6e98 100644
--- a/storm-compatibility/src/java/backtype/storm/utils/Utils.java
+++ b/storm-compatibility/src/java/backtype/storm/utils/Utils.java
@@ -38,7 +38,7 @@ public final class Utils {
Class c = Class.forName(klass);
return c.newInstance();
} catch (ClassNotFoundException | InstantiationException |
IllegalAccessException e) {
- throw new RuntimeException(e);
+ throw new RuntimeException("Failed to create instance for class: " +
klass, e);
}
}
diff --git a/storm-compatibility/src/java/org/apache/storm/utils/Utils.java
b/storm-compatibility/src/java/org/apache/storm/utils/Utils.java
index b97c160..62b4de4 100644
--- a/storm-compatibility/src/java/org/apache/storm/utils/Utils.java
+++ b/storm-compatibility/src/java/org/apache/storm/utils/Utils.java
@@ -38,7 +38,7 @@ public final class Utils {
Class c = Class.forName(klass);
return c.newInstance();
} catch (ClassNotFoundException | InstantiationException |
IllegalAccessException e) {
- throw new RuntimeException(e);
+ throw new RuntimeException("Failed to create instance for class: " +
klass, e);
}
}
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch master updated: Scala Streamlet API Documentation is being added (#2871)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 4d417d5 Scala Streamlet API Documentation is being added (#2871)
4d417d5 is described below
commit 4d417d540d3169525e0894029299c4bc7dadf1b4
Author: Eren Avsarogullari <erenavsarogull...@gmail.com>
AuthorDate: Sun Apr 22 22:34:04 2018 +0100
Scala Streamlet API Documentation is being added (#2871)
---
.../docs/developers/scala/streamlet-api.mmark | 408 +
website/data/toc.yaml | 2 +
2 files changed, 410 insertions(+)
diff --git a/website/content/docs/developers/scala/streamlet-api.mmark
b/website/content/docs/developers/scala/streamlet-api.mmark
new file mode 100644
index 000..4476567
--- /dev/null
+++ b/website/content/docs/developers/scala/streamlet-api.mmark
@@ -0,0 +1,408 @@
+---
+title: The Heron Streamlet API for Scala
+description: Create Heron topologies in Scala using a functional programming
style
+---
+
+{{< alert "streamlet-api-beta" >}}
+
+{{content/snippets/heron-streamlet-api.md}}
+
+## Getting started
+
+In order to use the Heron Streamlet API for Scala, you'll need to install the
`heron-api` library.
+
+### Maven setup
+
+In order to use the `heron-api` library, add this to the `dependencies` block
of your `pom.xml` configuration file:
+
+```xml
+
+com.twitter.heron
+heron-api
+{{< heronVersion >}}
+
+```
+
+ Compiling a JAR with dependencies
+
+In order to run a Scala topology created using the Heron Streamlet API in a
Heron cluster, you'll need to package your topology as a "fat" JAR with
dependencies included. You can use the [Maven Assembly
Plugin](https://maven.apache.org/plugins/maven-assembly-plugin/usage.html) to
generate JARs with dependencies. To install the plugin and add a Maven goal for
a single JAR, add this to the `plugins` block in your `pom.xml`:
+
+```xml
+
+maven-assembly-plugin
+
+
+jar-with-dependencies
+
+
+
+
+
+
+
+
+
+make-assembly
+package
+
+single
+
+
+
+
+```
+
+Once your `pom.xml` is properly set up, you can compile the JAR with
dependencies using this command:
+
+```bash
+$ mvn assembly:assembly
+```
+
+By default, this will add a JAR in your project's `target` folder with the
name `PROJECT-NAME-VERSION-jar-with-dependencies.jar`. Here's an example
topology submission command using a compiled JAR:
+
+```bash
+$ mvn assembly:assembly
+$ heron submit local \
+ target/my-project-1.2.3-jar-with-dependencies.jar \
+ com.example.Main \
+ MyTopology arg1 arg2
+```
+
+## Streamlet API topology configuration
+
+Every Streamlet API topology needs to be configured using a `Config` object.
Here's an example default configuration:
+
+```scala
+import com.twitter.heron.streamlet.Config
+import com.twitter.heron.streamlet.scala.Runner
+
+val topologyConfig = Config.defaultConfig()
+
+// Apply topology configuration using the topologyConfig object
+val topologyRunner = new Runner()
+topologyRunner.run("name-for-topology", topologyConfig, topologyBuilder)
+```
+
+The table below shows the configurable parameters for Heron topologies:
+
+Parameter | Default
+:-|:---
+[Delivery semantics](#delivery-semantics) | At most once
+Serializer | [Kryo](https://github.com/EsotericSoftware/kryo)
+Number of total container topologies | 2
+Per-container CPU | 1.0
+Per-container RAM | 100 MB
+
+Here's an example non-default configuration:
+
+```scala
+val topologyConfig = Config.newBuilder()
+.setNumContainers(5)
+.setPerContainerRamInGigabytes(10)
+.setPerContainerCpu(3.5f)
+.setDeliverySemantics(Config.DeliverySemantics.EFFECTIVELY_ONCE)
+.setSerializer(Config.Serializer.JAVA)
+.setUserConfig("some-key", "some-value")
+.build()
+```
+
+### Delivery semantics
+
+You can apply [delivery semantics](../../../concepts/delivery-semantics) to a
Streamlet API topology like this:
+
+```scala
+topologyConfig
+.setDeliverySemantics(Config.DeliverySemantics.EFFECTIVELY_ONCE)
+```
+
+The other available options in the `DeliverySemantics` enum are `ATMOST_ONCE`
and `ATLEAST_ONCE`.
+
+## Streamlets
+
+In the Heron Streamlet API for Scala, processing graphs consist of
[streamlets](../../../concepts/topologies#streamlets). One or more supplier
streamlets inject data into your graph to be processed by downstream operators.
+
+## Operations
+
+Operation | Description | Example
+:-|:|:---
+[`map`](#map-operations) | Create a new streamlet by applying the supplied
mapping function to each eleme
[incubator-heron] branch master updated: add .htaccess and index.html [https://github.com/apache/incubator-heron/issues/2758] (#2869)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 2bc3a18 add .htaccess and index.html [https://github.com/apache/incubator-heron/issues/2758] (#2869) 2bc3a18 is described below commit 2bc3a188977b04f327182f6cb14448513b62e1a3 Author: SreeV <441385+sr...@users.noreply.github.com> AuthorDate: Sun Apr 22 14:33:34 2018 -0700 add .htaccess and index.html [https://github.com/apache/incubator-heron/issues/2758] (#2869) --- .htaccess | 2 ++ index.html | 3 +++ 2 files changed, 5 insertions(+) diff --git a/.htaccess b/.htaccess new file mode 100644 index 000..305761d --- /dev/null +++ b/.htaccess @@ -0,0 +1,2 @@ +Redirect /index.html https://apache.github.io/incubator-heron/ + diff --git a/index.html b/index.html new file mode 100644 index 000..6930165 --- /dev/null +++ b/index.html @@ -0,0 +1,3 @@ + + + -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] branch karthik/binaryout updated (9ce6557 -> 3d2bb16)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a change to branch karthik/binaryout in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 9ce6557 removed glog/gperftools/libevent/libunwind/zookeeper tar.gz from source instead all of them are downloaded on the fly and compiled WORKSPACE file consists of one entry for each software item and its download URL add 3d2bb16 fix errors in compilation No new revisions were added by this update. Summary of changes: WORKSPACE | 7 +++ heron/common/src/cpp/basics/BUILD | 2 +- heron/common/src/cpp/metrics/BUILD| 2 +- heron/common/src/cpp/zookeeper/BUILD | 2 +- third_party/zookeeper/zookeeper.BUILD | 4 ++-- 5 files changed, 12 insertions(+), 5 deletions(-) -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] branch master updated: Fix bug in dry-run argument for update command (#2831)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 47ae988 Fix bug in dry-run argument for update command (#2831)
47ae988 is described below
commit 47ae9883fc95c1bd502b369f4d76e4cb49985577
Author: Andrew Jorgensen <ajorgen...@users.noreply.github.com>
AuthorDate: Sat Mar 31 20:27:26 2018 -0400
Fix bug in dry-run argument for update command (#2831)
Since list_extra_args is an array the `+=` operator deconstructs a
string and add each character as an individual array element. This made
it so when the update command it would not detect if it was in dry-run
mode and always run the update command. By encapsulating the `--dry-run`
string in an list it is properly added as an argument and runs with the
expected behavior.
This also fixes an issue with the way the `dry_run_format` argument was
being updated in the extra args dictionary
---
heron/tools/cli/src/python/update.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/heron/tools/cli/src/python/update.py
b/heron/tools/cli/src/python/update.py
index ba3557f..180eaea 100644
--- a/heron/tools/cli/src/python/update.py
+++ b/heron/tools/cli/src/python/update.py
@@ -102,7 +102,7 @@ def build_extra_args_dict(cl_args):
if cl_args['dry_run']:
dict_extra_args.update({'dry_run': True})
if 'dry_run_format' in cl_args:
- dict_extra_args.update({'dry_run_format', cl_args["dry_run_format"]})
+ dict_extra_args.update({'dry_run_format': cl_args["dry_run_format"]})
return dict_extra_args
@@ -117,7 +117,7 @@ def convert_args_dict_to_list(dict_extra_args):
list_extra_args += ["--runtime_config",
','.join(dict_extra_args['runtime_config'])]
if 'dry_run' in dict_extra_args and dict_extra_args['dry_run']:
-list_extra_args += '--dry_run'
+list_extra_args += ['--dry_run']
if 'dry_run_format' in dict_extra_args:
list_extra_args += ['--dry_run_format', dict_extra_args['dry_run_format']]
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch karthik/binaryout updated: remove package_dir from libunwind srcs
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch karthik/binaryout in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/karthik/binaryout by this push: new ca7835b remove package_dir from libunwind srcs ca7835b is described below commit ca7835b249c457db2c254e5d58fdfd7c9c795193 Author: Karthik Ramasamy <kart...@streaml.io> AuthorDate: Sat Mar 31 20:19:14 2018 -0700 remove package_dir from libunwind srcs --- third_party/libunwind/libunwind.BUILD | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/third_party/libunwind/libunwind.BUILD b/third_party/libunwind/libunwind.BUILD index 6432089..11dbbef 100644 --- a/third_party/libunwind/libunwind.BUILD +++ b/third_party/libunwind/libunwind.BUILD @@ -31,9 +31,9 @@ genrule( "mkdir -p $$TMP_DIR", "cp -R $$(pwd)/external/org_savannah_libunwind/* $$TMP_DIR", "cd $$TMP_DIR", -"patch -d " + package_dir + " -p1 < $$(SOURCE_DIR)/" + lzma_patch, -"patch -d " + package_dir + " -p0 < $$(SOURCE_DIR)/" + config_patch, -"patch -d " + package_dir + " -p0 < $$(SOURCE_DIR)/" + cache_patch, +"patch -p1 < $$(SOURCE_DIR)/" + lzma_patch, +"patch -p0 < $$(SOURCE_DIR)/" + config_patch, +"patch -p0 < $$(SOURCE_DIR)/" + cache_patch, "./configure --prefix=$$INSTALL_DIR --enable-shared=no --disable-minidebuginfo", "make install", "rm -rf $$TMP_DIR", -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] tag 0.17.8 created (now 3822400)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a change to tag 0.17.8 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. at 3822400 (commit) No new revisions were added by this update. -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] branch karthik/binaryout updated: updated the download path for libunwind
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch karthik/binaryout in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/karthik/binaryout by this push: new b75d147 updated the download path for libunwind b75d147 is described below commit b75d14711c356ef3a51b62e79869767ff8af5e87 Author: Karthik Ramasamy <kart...@streaml.io> AuthorDate: Sun Apr 1 07:07:53 2018 -0700 updated the download path for libunwind --- third_party/libunwind/libunwind.BUILD | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/third_party/libunwind/libunwind.BUILD b/third_party/libunwind/libunwind.BUILD index 8eb51d2..d4b7470 100644 --- a/third_party/libunwind/libunwind.BUILD +++ b/third_party/libunwind/libunwind.BUILD @@ -41,7 +41,7 @@ genrule( "export INSTALL_DIR=$$(pwd)/$(@D)", "export TMP_DIR=$$(mktemp -d -t libunwind.X)", "mkdir -p $$TMP_DIR", -"cp -LR $$(pwd)/external/org_savannah_libunwind/* $$TMP_DIR", +"cp -LR $$(pwd)/external/org_nongnu_libunwind/* $$TMP_DIR", "cd $$TMP_DIR", "patch -p1 < $$SOURCE_DIR/$(location @org_apache_heron//third_party/libunwind:libunwind-1.1-lzma-link.patch)", "patch -p0 < $$SOURCE_DIR/$(location @org_apache_heron//third_party/libunwind:libunwind-1.1-config.patch)", -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] branch karthik/binaryout updated: truncate lines greater than 100
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch karthik/binaryout
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/karthik/binaryout by this push:
new de41ff6 truncate lines greater than 100
de41ff6 is described below
commit de41ff639d75855f93b0858c79b76af28e49b476
Author: Karthik Ramasamy <kart...@streaml.io>
AuthorDate: Sun Apr 1 10:23:59 2018 -0700
truncate lines greater than 100
---
.../heron/metricsmgr/sink/metricscache/MetricsCacheSinkTest.java | 3 ++-
.../com/twitter/heron/metricsmgr/sink/tmaster/TMasterSinkTest.java | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git
a/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/metricscache/MetricsCacheSinkTest.java
b/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/metricscache/MetricsCacheSinkTest.java
index 151df5b..6614692 100644
---
a/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/metricscache/MetricsCacheSinkTest.java
+++
b/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/metricscache/MetricsCacheSinkTest.java
@@ -85,7 +85,8 @@ public class MetricsCacheSinkTest {
String filePath =
Paths.get(runFiles,
-
"/org_apache_heron/heron/config/src/yaml/conf/test/test_heron_internals.yaml").toString();
+
"/org_apache_heron/heron/config/src/yaml/conf/test/test_heron_internals.yaml")
+ .toString();
SystemConfig.Builder sb = SystemConfig.newBuilder(true)
.putAll(filePath, true)
.put(SystemConfigKey.HERON_METRICS_EXPORT_INTERVAL, 1);
diff --git
a/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/tmaster/TMasterSinkTest.java
b/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/tmaster/TMasterSinkTest.java
index 095bdf3..93402d2 100644
---
a/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/tmaster/TMasterSinkTest.java
+++
b/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/tmaster/TMasterSinkTest.java
@@ -86,7 +86,8 @@ public class TMasterSinkTest {
String filePath =
Paths.get(runFiles,
-
"/org_apache_heron/heron/config/src/yaml/conf/test/test_heron_internals.yaml").toString();
+
"/org_apache_heron/heron/config/src/yaml/conf/test/test_heron_internals.yaml")
+ .toString();
SystemConfig.Builder sb = SystemConfig.newBuilder(true)
.putAll(filePath, true)
.put(SystemConfigKey.HERON_METRICS_EXPORT_INTERVAL, 1);
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch karthik/binaryout updated: added a default user since in a docker environment root user is not set
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch karthik/binaryout
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/karthik/binaryout by this push:
new 8413151 added a default user since in a docker environment root user
is not set
8413151 is described below
commit 841315105718d041381a3eff71e1225d956723e6
Author: Karthik Ramasamy <kart...@streaml.io>
AuthorDate: Sat Mar 31 19:48:57 2018 -0700
added a default user since in a docker environment root user is not set
---
scripts/release/status.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/release/status.sh b/scripts/release/status.sh
index 17d5c64..1418e15 100755
--- a/scripts/release/status.sh
+++ b/scripts/release/status.sh
@@ -78,7 +78,7 @@ echo "HERON_BUILD_TIMESTAMP ${build_timestamp}"
if [ -z ${HERON_BUILD_USER+x} ];
then
- build_user=${USER}
+ build_user=${USER:-root}
else
build_user=${HERON_BUILD_USER}
fi
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch karthik/binaryout updated: fix libunwind dependency
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch karthik/binaryout
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/karthik/binaryout by this push:
new df64466 fix libunwind dependency
df64466 is described below
commit df6446609c63179ee27ce79017050043de177264
Author: Karthik Ramasamy <kart...@streaml.io>
AuthorDate: Sat Mar 31 22:52:36 2018 -0700
fix libunwind dependency
---
WORKSPACE | 2 ++
third_party/glog/glog.BUILD | 4 ++--
third_party/gperftools/gperftools.BUILD | 4 +++-
third_party/libunwind/BUILD | 11 +++
third_party/libunwind/libunwind.BUILD | 18 +++---
5 files changed, 33 insertions(+), 6 deletions(-)
diff --git a/WORKSPACE b/WORKSPACE
index a619189..b040e43 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -1,3 +1,5 @@
+workspace(name = "org_apache_heron")
+
# versions shared across artifacts that should be upgraded together
aws_version = "1.11.58"
curator_version = "2.9.0"
diff --git a/third_party/glog/glog.BUILD b/third_party/glog/glog.BUILD
index f2c4b42..06baa96 100644
--- a/third_party/glog/glog.BUILD
+++ b/third_party/glog/glog.BUILD
@@ -58,11 +58,11 @@ linux_script = "\n".join(common_script + [
genrule(
name = "glog-srcs",
-outs = include_files + lib_files,
srcs = select({
":darwin": [],
-"//conditions:default": ["//third_party/libunwind:libunwind-files"]
+"//conditions:default":
["@org_apache_heron//third_party/libunwind:libunwind-files"]
}),
+outs = include_files + lib_files,
cmd = select({
":darwin": mac_script,
"//conditions:default": linux_script,
diff --git a/third_party/gperftools/gperftools.BUILD
b/third_party/gperftools/gperftools.BUILD
index 0d84397..e723f6b 100644
--- a/third_party/gperftools/gperftools.BUILD
+++ b/third_party/gperftools/gperftools.BUILD
@@ -21,6 +21,8 @@ config_setting(
common_script = [
"echo $$(pwd)",
"export UNWIND_DIR=$$(pwd)/$(GENDIR)/third_party/libunwind",
+"echo $$UNWIND_DIR",
+"ls -l $$UNWIND_DIR",
"export INSTALL_DIR=$$(pwd)/$(@D)",
"export TMP_DIR=$$(mktemp -d -t gperftools.X)",
"mkdir -p $$TMP_DIR",
@@ -44,7 +46,7 @@ genrule(
name = "gperftools-srcs",
srcs = select({
":darwin": [],
-"//conditions:default": ["//third_party/libunwind:libunwind-files"]
+"//conditions:default":
["@org_apache_heron//third_party/libunwind:libunwind-files"]
}),
outs = [
"bin/pprof",
diff --git a/third_party/libunwind/BUILD b/third_party/libunwind/BUILD
new file mode 100644
index 000..6cbb232
--- /dev/null
+++ b/third_party/libunwind/BUILD
@@ -0,0 +1,11 @@
+exports_files([
+"libunwind-1.1-cache.patch",
+"libunwind-1.1-config.patch",
+"libunwind-1.1-lzma-link.patch",
+"libunwind.BUILD",
+])
+
+filegroup(
+name = "libunwind-files",
+srcs = ["@org_nongnu_libunwind//:libunwind-files"],
+)
diff --git a/third_party/libunwind/libunwind.BUILD
b/third_party/libunwind/libunwind.BUILD
index 11dbbef..9919567 100644
--- a/third_party/libunwind/libunwind.BUILD
+++ b/third_party/libunwind/libunwind.BUILD
@@ -21,8 +21,20 @@ out_files = [
"lib/libunwind-x86_64.a",
]
+exports_files([
+"libunwind-1.1-cache.patch",
+"libunwind-1.1-config.patch",
+"libunwind-1.1-lzma-link.patch",
+"libunwind.BUILD",
+])
+
genrule(
name = "libunwind-srcs",
+srcs = [
+"@org_apache_heron//third_party/libunwind:libunwind-1.1-cache.patch",
+"@org_apache_heron//third_party/libunwind:libunwind-1.1-config.patch",
+
"@org_apache_heron//third_party/libunwind:libunwind-1.1-lzma-link.patch",
+],
outs = out_files,
cmd = "\n".join([
"export SOURCE_DIR=$$(pwd)/third_party/libunwind",
@@ -31,9 +43,9 @@ genrule(
"mkdir -p $$TMP_DIR",
"cp -R $$(pwd)/external/org_savannah_libunwind/* $$TMP_DIR",
"cd $$TMP_DIR",
-"patch -p1 < $$(SOURCE_DIR)/" + lzma_patch,
-"patch -p0 < $$(SOURCE_DIR)/" + config_patch,
-"patch -p0 < $$(SOURCE_DIR)/" + cache_patch,
+"patch -p1 < $(location
@org_apache_heron//third_party/libunwind:libunwind-1.1-lzma-link.patch)",
+"patch -p0 < $(location
@org_apache_heron//third_party/libunwind:libunwind-1.1-config.patch)",
+"patch -p0 < $(location
@org_apache_heron//third_party/libunwind:libunwind-1.1-cache.patch)",
"./configure --prefix=$$INSTALL_DIR --enable-shared=no
--disable-minidebuginfo",
"make install",
"rm -rf $$TMP_DIR",
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch master updated: update README.md (#2835)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 3822400 update README.md (#2835) 3822400 is described below commit 382240052b97e9e792d7ba2f35b9b5e4e4c15382 Author: SreeV <441385+sr...@users.noreply.github.com> AuthorDate: Sat Mar 31 23:31:03 2018 -0700 update README.md (#2835) * update README.md [https://github.com/twitter/heron/issues/2664] * update README.md further [https://github.com/twitter/heron/issues/2664] * update travis link; add meetup group; [https://github.com/apache/incubator-heron/issues/2834] --- README.md | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 2f27ed9..3e21c5e 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -[](https://travis-ci.org/twitter/heron) +[](https://travis-ci.org/apache/incubator-heron)  @@ -40,6 +40,9 @@ heron slack channel at https://heronstreaming.slack.com/ You can self-register at http://heronstreaming.herokuapp.com/ + Meetup Group +https://www.meetup.com/Apache-Heron-Bay-Area + ## For more information: -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] branch karthik/binaryout updated: fix unit errors due to workspace name change
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch karthik/binaryout
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/karthik/binaryout by this push:
new f790fb5 fix unit errors due to workspace name change
f790fb5 is described below
commit f790fb52dea149503f11969bb5c1fd69d95832aa
Author: Karthik Ramasamy <kart...@streaml.io>
AuthorDate: Sun Apr 1 00:14:18 2018 -0700
fix unit errors due to workspace name change
---
heron/instance/tests/java/com/twitter/heron/resource/Constants.java | 2 +-
.../heron/metricsmgr/sink/metricscache/MetricsCacheSinkTest.java| 2 +-
.../java/com/twitter/heron/metricsmgr/sink/tmaster/TMasterSinkTest.java | 2 +-
heron/spi/tests/java/com/twitter/heron/spi/common/ConfigLoaderTest.java | 2 +-
.../twitter/heron/uploader/localfs/LocalFileSystemConstantsTest.java| 2 +-
5 files changed, 5 insertions(+), 5 deletions(-)
diff --git
a/heron/instance/tests/java/com/twitter/heron/resource/Constants.java
b/heron/instance/tests/java/com/twitter/heron/resource/Constants.java
index dee51ec..390ab37 100644
--- a/heron/instance/tests/java/com/twitter/heron/resource/Constants.java
+++ b/heron/instance/tests/java/com/twitter/heron/resource/Constants.java
@@ -47,7 +47,7 @@ public final class Constants {
// For bazel, we use the env var to get the path of heron internals config
file
public static final String BUILD_TEST_SRCDIR = "TEST_SRCDIR";
public static final String BUILD_TEST_HERON_INTERNALS_CONFIG_PATH =
- "/__main__/heron/config/src/yaml/conf/test/test_heron_internals.yaml";
+
"/org_apache_heron/heron/config/src/yaml/conf/test/test_heron_internals.yaml";
private Constants() {
}
diff --git
a/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/metricscache/MetricsCacheSinkTest.java
b/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/metricscache/MetricsCacheSinkTest.java
index bf5a988..151df5b 100644
---
a/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/metricscache/MetricsCacheSinkTest.java
+++
b/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/metricscache/MetricsCacheSinkTest.java
@@ -85,7 +85,7 @@ public class MetricsCacheSinkTest {
String filePath =
Paths.get(runFiles,
-
"/__main__/heron/config/src/yaml/conf/test/test_heron_internals.yaml").toString();
+
"/org_apache_heron/heron/config/src/yaml/conf/test/test_heron_internals.yaml").toString();
SystemConfig.Builder sb = SystemConfig.newBuilder(true)
.putAll(filePath, true)
.put(SystemConfigKey.HERON_METRICS_EXPORT_INTERVAL, 1);
diff --git
a/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/tmaster/TMasterSinkTest.java
b/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/tmaster/TMasterSinkTest.java
index fdc3594..095bdf3 100644
---
a/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/tmaster/TMasterSinkTest.java
+++
b/heron/metricsmgr/tests/java/com/twitter/heron/metricsmgr/sink/tmaster/TMasterSinkTest.java
@@ -86,7 +86,7 @@ public class TMasterSinkTest {
String filePath =
Paths.get(runFiles,
-
"/__main__/heron/config/src/yaml/conf/test/test_heron_internals.yaml").toString();
+
"/org_apache_heron/heron/config/src/yaml/conf/test/test_heron_internals.yaml").toString();
SystemConfig.Builder sb = SystemConfig.newBuilder(true)
.putAll(filePath, true)
.put(SystemConfigKey.HERON_METRICS_EXPORT_INTERVAL, 1);
diff --git
a/heron/spi/tests/java/com/twitter/heron/spi/common/ConfigLoaderTest.java
b/heron/spi/tests/java/com/twitter/heron/spi/common/ConfigLoaderTest.java
index e815dfd..4e28a55 100644
--- a/heron/spi/tests/java/com/twitter/heron/spi/common/ConfigLoaderTest.java
+++ b/heron/spi/tests/java/com/twitter/heron/spi/common/ConfigLoaderTest.java
@@ -38,7 +38,7 @@ import static org.mockito.Mockito.times;
@PrepareForTest(ConfigLoader.class)
public class ConfigLoaderTest {
private static final String TEST_DATA_PATH =
- "/__main__/heron/spi/tests/java/com/twitter/heron/spi/common/testdata";
+
"/org_apache_heron/heron/spi/tests/java/com/twitter/heron/spi/common/testdata";
private final String heronHome =
Paths.get(System.getenv("JAVA_RUNFILES"), TEST_DATA_PATH).toString();
diff --git
a/heron/uploaders/tests/java/com/twitter/heron/uploader/localfs/LocalFileSystemConstantsTest.java
b/heron/uploaders/tests/java/com/twitter/heron/uploader/localfs/LocalFileSystemConstantsTest.java
index deb96cd..2844541 100644
---
a/heron/uploaders/tests/java/com/twitter/heron/uploader/localfs/LocalFileSystemConstantsTest.java
+++
b/heron/uploaders/tests/java/com/twitter/heron/uploader/localfs/Loca
[incubator-heron] branch master updated: removed glog/gperftools/libevent/libunwind/zookeeper tar.gz from source (#2833)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 5a7faa6 removed glog/gperftools/libevent/libunwind/zookeeper tar.gz
from source (#2833)
5a7faa6 is described below
commit 5a7faa65200d2fa215301d298b87e69156729104
Author: Karthik Ramasamy <kart...@streaml.io>
AuthorDate: Sun Apr 1 12:53:03 2018 -0700
removed glog/gperftools/libevent/libunwind/zookeeper tar.gz from source
(#2833)
* removed glog/gperftools/libevent/libunwind/zookeeper tar.gz from source
instead all of them are downloaded on the fly and compiled
WORKSPACE file consists of one entry for each software item and its
download URL
* fix errors in compilation
* added a default user since in a docker environment root user is not set
* remove package_dir from libunwind srcs
* fix libunwind dependency
* fix ubuntu compilation
* fix unit errors due to workspace name change
* updated the download path for libunwind
* truncate lines greater than 100
---
WORKSPACE | 37
heron/common/src/cpp/basics/BUILD | 6 +-
heron/common/src/cpp/metrics/BUILD | 7 +-
heron/common/src/cpp/network/BUILD | 2 +-
heron/common/src/cpp/zookeeper/BUILD | 2 +-
.../java/com/twitter/heron/resource/Constants.java | 2 +-
.../sink/metricscache/MetricsCacheSinkTest.java| 3 +-
.../metricsmgr/sink/tmaster/TMasterSinkTest.java | 3 +-
.../twitter/heron/spi/common/ConfigLoaderTest.java | 2 +-
.../localfs/LocalFileSystemConstantsTest.java | 2 +-
scripts/release/status.sh | 2 +-
third_party/cereal/cereal.BUILD| 101 -
third_party/glog/empty.cc | 2 -
third_party/glog/glog-0.3.5.tar.gz | Bin 532275 -> 0 bytes
third_party/glog/{BUILD => glog.BUILD} | 45 +
third_party/gperftools/empty.cc| 2 -
third_party/gperftools/gperftools-2.4.tar.gz | Bin 1346075 -> 0 bytes
third_party/gperftools/{BUILD => gperftools.BUILD} | 62 ++---
third_party/libevent/empty.cc | 2 -
third_party/libevent/libevent-2.1.8-stable.tar.gz | Bin 1026485 -> 0 bytes
third_party/libevent/{BUILD => libevent.BUILD} | 24 +
third_party/libunwind/BUILD| 76 ++--
third_party/libunwind/empty.cc | 2 -
third_party/libunwind/libunwind-1.1.tar.gz | Bin 1098603 -> 0 bytes
third_party/libunwind/{BUILD => libunwind.BUILD} | 42 -
third_party/zookeeper/empty.cc | 2 -
third_party/zookeeper/zookeeper-3.4.10.tar.gz | Bin 35042811 -> 0 bytes
third_party/zookeeper/{BUILD => zookeeper.BUILD} | 20 +---
28 files changed, 140 insertions(+), 308 deletions(-)
diff --git a/WORKSPACE b/WORKSPACE
index 3a59293..b040e43 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -1,3 +1,5 @@
+workspace(name = "org_apache_heron")
+
# versions shared across artifacts that should be upgraded together
aws_version = "1.11.58"
curator_version = "2.9.0"
@@ -810,6 +812,41 @@ http_archive(
)
new_http_archive(
+name = "org_libevent_libevent",
+urls =
["https://github.com/libevent/libevent/releases/download/release-2.1.8-stable/libevent-2.1.8-stable.tar.gz;],
+strip_prefix = "libevent-2.1.8-stable",
+build_file = "third_party/libevent/libevent.BUILD",
+)
+
+new_http_archive(
+name = "org_nongnu_libunwind",
+urls =
["http://download.savannah.nongnu.org/releases/libunwind/libunwind-1.1.tar.gz;],
+strip_prefix = "libunwind-1.1",
+build_file = "third_party/libunwind/libunwind.BUILD",
+)
+
+new_http_archive(
+name = "org_apache_zookeeper",
+urls =
["http://apache.cs.utah.edu/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz;],
+strip_prefix = "zookeeper-3.4.10",
+build_file = "third_party/zookeeper/zookeeper.BUILD",
+)
+
+new_http_archive(
+name = "com_github_gperftools_gperftools",
+urls =
["https://github.com/gperftools/gperftools/releases/download/gperftools-2.4/gperftools-2.4.tar.gz;],
+strip_prefix = "gperftools-2.4",
+build_file = "third_party/gperftools/gperftools.BUILD",
+)
+
+new_http_archive(
+name = "com_github_google_glog",
+urls = ["https://github.com/google/glog/archive/v0.3.5.tar.gz;],
+strip_prefix = "glog-0.3.5",
+build_file = "third_party/glog/glog.BUILD",
+)
+
+new_http_archive
[incubator-heron] branch master updated: Heron JavaDoc and Documentation Fixes (#2838)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 976f89f Heron JavaDoc and Documentation Fixes (#2838)
976f89f is described below
commit 976f89fdd096ca3f378a72fb5afa63ab4a502a98
Author: Eren Avsarogullari <erenavsarogull...@gmail.com>
AuthorDate: Mon Apr 2 17:17:12 2018 +0100
Heron JavaDoc and Documentation Fixes (#2838)
---
.../twitter/heron/common/network/HeronClient.java | 2 +-
.../twitter/heron/downloader/FileDownloader.java | 2 +-
.../com/twitter/heron/instance/HeronInstance.java | 2 +-
.../roundrobin/ResourceCompliantRRPacking.java | 4 +-
.../heron/scheduler/nomad/NomadScheduler.java | 2 +-
website/content/docs/concepts/architecture.md | 68 +++---
6 files changed, 39 insertions(+), 41 deletions(-)
diff --git
a/heron/common/src/java/com/twitter/heron/common/network/HeronClient.java
b/heron/common/src/java/com/twitter/heron/common/network/HeronClient.java
index a4760ed..adb7714 100644
--- a/heron/common/src/java/com/twitter/heron/common/network/HeronClient.java
+++ b/heron/common/src/java/com/twitter/heron/common/network/HeronClient.java
@@ -33,7 +33,7 @@ import com.twitter.heron.common.basics.NIOLooper;
/**
* Implements this class could handle some following socket related behaviors:
* 1. handleRead(SelectableChannel), which read data from a socket and convert
into incomingPacket.
- * It could handle the conditions of closedConnection, normal Reading and
partial Reading. When a
+ * It could handle the conditions of closedConnection, normal Reading and
partial Reading. When an
* incomingPacket is read, it will be pass to handlePacket(), which will
convert incomingPackets to
* messages and call onIncomingMessage(message), which should be implemented
by its child class.
*
diff --git
a/heron/downloaders/src/java/com/twitter/heron/downloader/FileDownloader.java
b/heron/downloaders/src/java/com/twitter/heron/downloader/FileDownloader.java
index 272c242..bcbd7fb 100644
---
a/heron/downloaders/src/java/com/twitter/heron/downloader/FileDownloader.java
+++
b/heron/downloaders/src/java/com/twitter/heron/downloader/FileDownloader.java
@@ -19,7 +19,7 @@ import java.nio.file.Path;
/**
* Used to download files via heron_downloader that has a URI prefix of
"file://"
- * E.g. ./heron_dowloader file:///foo/bar /path/location
+ * E.g. ./heron_downloader file:///foo/bar /path/location
*/
public class FileDownloader implements Downloader {
@Override
diff --git
a/heron/instance/src/java/com/twitter/heron/instance/HeronInstance.java
b/heron/instance/src/java/com/twitter/heron/instance/HeronInstance.java
index 15ee204..df3bf70 100644
--- a/heron/instance/src/java/com/twitter/heron/instance/HeronInstance.java
+++ b/heron/instance/src/java/com/twitter/heron/instance/HeronInstance.java
@@ -302,7 +302,7 @@ public class HeronInstance {
String logMsg = "\nStarting instance " + instanceId + " for topology " +
topologyName
+ " and topologyId " + topologyId + " for component " + componentName
+ " with taskId " + taskId + " and componentIndex " + componentIndex
-+ " and stmgrId " + streamId + " and stmgrPort " + streamPort
++ " and streamManagerId " + streamId + " and streamManagerPort " +
streamPort
+ " and metricsManagerPort " + metricsPort;
if (remoteDebuggerPort != null) {
diff --git
a/heron/packing/src/java/com/twitter/heron/packing/roundrobin/ResourceCompliantRRPacking.java
b/heron/packing/src/java/com/twitter/heron/packing/roundrobin/ResourceCompliantRRPacking.java
index 778e0cc..f07dd86 100644
---
a/heron/packing/src/java/com/twitter/heron/packing/roundrobin/ResourceCompliantRRPacking.java
+++
b/heron/packing/src/java/com/twitter/heron/packing/roundrobin/ResourceCompliantRRPacking.java
@@ -122,7 +122,7 @@ public class ResourceCompliantRRPacking implements
IPacking, IRepacking {
Context.instanceDisk(config));
resetToFirstContainer();
-LOG.info(String.format("Initalizing ResourceCompliantRRPacking. "
+LOG.info(String.format("Initializing ResourceCompliantRRPacking. "
+ "CPU default: %f, RAM default: %s, DISK default: %s.",
this.defaultInstanceResources.getCpu(),
this.defaultInstanceResources.getRam().toString(),
@@ -308,7 +308,7 @@ public class ResourceCompliantRRPacking implements
IPacking, IRepacking {
* Attempts to place the instance the current containerId.
*
* @param planBuilder packing plan builder
- * @param componentName the componet name of the instance that needs to be
placed in the container
+ * @param componentName the co
[incubator-heron] branch karthik/libunwind-makeinstall updated (2f86b71 -> e89c28c)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a change to branch karthik/libunwind-makeinstall in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 2f86b71 avoid make install add e89c28c add a missing file to be output No new revisions were added by this update. Summary of changes: third_party/libunwind/libunwind.BUILD | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] branch karthik/libunwind-makeinstall updated: Added simple easy fix
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch karthik/libunwind-makeinstall in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/karthik/libunwind-makeinstall by this push: new df5967b Added simple easy fix df5967b is described below commit df5967b55ff8382df5a5aad9eb2136dd41e70bc0 Author: Karthik Ramasamy <kart...@streaml.io> AuthorDate: Tue Apr 3 00:41:55 2018 -0700 Added simple easy fix --- third_party/libunwind/libunwind.BUILD | 7 +-- 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/third_party/libunwind/libunwind.BUILD b/third_party/libunwind/libunwind.BUILD index 395473a..bc25c71 100644 --- a/third_party/libunwind/libunwind.BUILD +++ b/third_party/libunwind/libunwind.BUILD @@ -47,12 +47,7 @@ genrule( "patch -p0 < $$SOURCE_DIR/$(location @org_apache_heron//third_party/libunwind:libunwind-1.1-config.patch)", "patch -p0 < $$SOURCE_DIR/$(location @org_apache_heron//third_party/libunwind:libunwind-1.1-cache.patch)", "./configure --prefix=$$INSTALL_DIR --enable-shared=no --disable-minidebuginfo", -"make", -"mkdir -p $$INSTALL_DIR/include $$INSTALL_DIR/lib", -"cp include/libunwind-common.h include/libunwind-coredump.h include/libunwind-dynamic.h $$INSTALL_DIR/include", -"cp include/libunwind.h include/libunwind-ptrace.h include/libunwind-x86_64.h include/unwind.h $$INSTALL_DIR/include", -"cp src/.libs/libunwind.a src/.libs/libunwind-coredump.a src/.libs/libunwind-ptrace.a $$INSTALL_DIR/lib", -"cp src/.libs/libunwind-setjmp.a src/.libs/libunwind-x86_64.a $$INSTALL_DIR/lib", +'make install SUBDIRS="src tests"', "rm -rf $$TMP_DIR", ]), ) -- To stop receiving notification emails like this one, please contact karth...@apache.org.
[incubator-heron] branch master updated: While emitting in spout, adhere to the batch size limit (#2798)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 1c47ef8 While emitting in spout, adhere to the batch size limit
(#2798)
1c47ef8 is described below
commit 1c47ef86ba57eb612cf965285211dae1a3ce3196
Author: Sanjeev Kulkarni <sanjee...@gmail.com>
AuthorDate: Tue Apr 3 11:54:44 2018 -0700
While emitting in spout, adhere to the batch size limit (#2798)
* While emitting in spout, adhere to the batch size limit
* Added accessor
* Added missing imports
* Make sure that C++ instance also adheres to batch emit size constraint
---
heron/common/src/cpp/config/heron-internals-config-reader.cpp | 5 +
heron/common/src/cpp/config/heron-internals-config-reader.h | 3 +++
heron/common/src/cpp/config/heron-internals-config-vars.cpp | 2 ++
heron/common/src/cpp/config/heron-internals-config-vars.h | 3 +++
heron/instance/src/cpp/slave/outgoing-tuple-collection.h | 2 +-
heron/instance/src/cpp/spoutimpl/spout-instance.cpp | 8 +++-
heron/instance/src/cpp/spoutimpl/spout-instance.h | 2 ++
heron/instance/src/cpp/spoutimpl/spout-output-collector-impl.h| 1 +
.../java/com/twitter/heron/instance/AbstractOutputCollector.java | 7 +++
.../src/java/com/twitter/heron/instance/spout/SpoutInstance.java | 8
10 files changed, 39 insertions(+), 2 deletions(-)
diff --git a/heron/common/src/cpp/config/heron-internals-config-reader.cpp
b/heron/common/src/cpp/config/heron-internals-config-reader.cpp
index 05c4b49..174dcb8 100644
--- a/heron/common/src/cpp/config/heron-internals-config-reader.cpp
+++ b/heron/common/src/cpp/config/heron-internals-config-reader.cpp
@@ -317,6 +317,11 @@ int
HeronInternalsConfigReader::GetHeronInstanceEmitBatchTimeMs() {
.as();
}
+int HeronInternalsConfigReader::GetHeronInstanceEmitBatchSize() {
+ return config_[HeronInternalsConfigVars::HERON_INSTANCE_EMIT_BATCH_SIZE]
+ .as();
+}
+
int HeronInternalsConfigReader::GetHeronInstanceSetDataTupleCapacity() {
return
config_[HeronInternalsConfigVars::HERON_INSTANCE_SET_DATA_TUPLE_CAPACITY]
.as();
diff --git a/heron/common/src/cpp/config/heron-internals-config-reader.h
b/heron/common/src/cpp/config/heron-internals-config-reader.h
index 696c58f..d5a34a1 100644
--- a/heron/common/src/cpp/config/heron-internals-config-reader.h
+++ b/heron/common/src/cpp/config/heron-internals-config-reader.h
@@ -216,6 +216,9 @@ class HeronInternalsConfigReader : public YamlFileReader {
// The maximum time in ms for an spout instance to emit tuples per attempt
int GetHeronInstanceEmitBatchTimeMs();
+ // The maximum number of bytes for an spout instance to emit tuples per
attempt
+ int GetHeronInstanceEmitBatchSize();
+
// The maximum # of data tuple to batch in a HeronDataTupleSet protobuf
int GetHeronInstanceSetDataTupleCapacity();
diff --git a/heron/common/src/cpp/config/heron-internals-config-vars.cpp
b/heron/common/src/cpp/config/heron-internals-config-vars.cpp
index 373ba4a..0906eaa 100644
--- a/heron/common/src/cpp/config/heron-internals-config-vars.cpp
+++ b/heron/common/src/cpp/config/heron-internals-config-vars.cpp
@@ -128,6 +128,8 @@ const sp_string
HeronInternalsConfigVars::HERON_INSTANCE_INTERNAL_SPOUT_WRITE_QU
"heron.instance.internal.spout.write.queue.capacity";
const sp_string HeronInternalsConfigVars::HERON_INSTANCE_EMIT_BATCH_TIME_MS =
"heron.instance.emit.batch.time.ms";
+const sp_string HeronInternalsConfigVars::HERON_INSTANCE_EMIT_BATCH_SIZE =
+"heron.instance.emit.batch.size.bytes";
const sp_string
HeronInternalsConfigVars::HERON_INSTANCE_SET_DATA_TUPLE_CAPACITY =
"heron.instance.set.data.tuple.capacity";
const sp_string
HeronInternalsConfigVars::HERON_INSTANCE_SET_DATA_TUPLE_SIZE_BYTES =
diff --git a/heron/common/src/cpp/config/heron-internals-config-vars.h
b/heron/common/src/cpp/config/heron-internals-config-vars.h
index bcab89a..5ac3752 100644
--- a/heron/common/src/cpp/config/heron-internals-config-vars.h
+++ b/heron/common/src/cpp/config/heron-internals-config-vars.h
@@ -206,6 +206,9 @@ class HeronInternalsConfigVars {
// The maximum time in ms for an spout instance to emit tuples per attempt
static const sp_string HERON_INSTANCE_EMIT_BATCH_TIME_MS;
+ // The maximum number of bytes for n spout instance to emit tuples per
attempt
+ static const sp_string HERON_INSTANCE_EMIT_BATCH_SIZE;
+
// The maximum # of data tuple to batch in a HeronDataTupleSet protobuf
static const sp_string HERON_INSTANCE_SET_DATA_TUPLE_CAPACITY;
diff --git a/heron/instance/src/cpp/slave/outgoing-tuple-collection.h
b/heron/instance/src/cpp/slave/outgoing-tuple-collection.h
index b9d722b..0acc76b 100644
---
[incubator-heron] branch master updated: add config for apiserver overrides (#2845)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 860a926 add config for apiserver overrides (#2845)
860a926 is described below
commit 860a926273caa37e2fa7d07ad7a27e2cac097f08
Author: Boyang Jerry Peng <jerry.boyang.p...@gmail.com>
AuthorDate: Tue Apr 3 11:55:01 2018 -0700
add config for apiserver overrides (#2845)
---
.../java/com/twitter/heron/scheduler/nomad/NomadScheduler.java | 3 ++-
heron/spi/src/java/com/twitter/heron/spi/common/Context.java | 4
heron/spi/src/java/com/twitter/heron/spi/common/Key.java | 4
.../java/com/twitter/heron/apiserver/utils/ConfigUtils.java| 10 +-
4 files changed, 19 insertions(+), 2 deletions(-)
diff --git
a/heron/schedulers/src/java/com/twitter/heron/scheduler/nomad/NomadScheduler.java
b/heron/schedulers/src/java/com/twitter/heron/scheduler/nomad/NomadScheduler.java
index 482aaa9..7531f23 100644
---
a/heron/schedulers/src/java/com/twitter/heron/scheduler/nomad/NomadScheduler.java
+++
b/heron/schedulers/src/java/com/twitter/heron/scheduler/nomad/NomadScheduler.java
@@ -70,7 +70,8 @@ public class NomadScheduler implements IScheduler {
.putAll(config)
.putAll(ConfigLoader.loadConfig(
Context.heronHome(config), Context.heronConf(config),
-null, Context.overrideFile(config)))
+null, Context.apiserverOverrideFile(config) != null
+? Context.apiserverOverrideFile(config) :
Context.overrideFile(config)))
.build());
this.runtimeConfig = runtime;
this.updateTopologyManager =
diff --git a/heron/spi/src/java/com/twitter/heron/spi/common/Context.java
b/heron/spi/src/java/com/twitter/heron/spi/common/Context.java
index 7438bee..27a5dfb 100644
--- a/heron/spi/src/java/com/twitter/heron/spi/common/Context.java
+++ b/heron/spi/src/java/com/twitter/heron/spi/common/Context.java
@@ -297,6 +297,10 @@ public class Context {
return cfg.getStringValue(Key.OVERRIDE_YAML);
}
+ public static String apiserverOverrideFile(Config cfg) {
+return cfg.getStringValue(Key.APISERVER_OVERRIDE_YAML);
+ }
+
public static String executorBinary(Config cfg) {
return cfg.getStringValue(Key.EXECUTOR_BINARY);
}
diff --git a/heron/spi/src/java/com/twitter/heron/spi/common/Key.java
b/heron/spi/src/java/com/twitter/heron/spi/common/Key.java
index 66d3313..acd74e3 100644
--- a/heron/spi/src/java/com/twitter/heron/spi/common/Key.java
+++ b/heron/spi/src/java/com/twitter/heron/spi/common/Key.java
@@ -165,6 +165,10 @@ public enum Key {
//keys for heron configuration files on the cluster
OVERRIDE_YAML("heron.config.file.override.yaml",
"${HERON_CONF}/override.yaml"),
+ // Path to the config overrides passed into the apiserver. Only applicable
to submitting
+ // topologies via apiserver
+ APISERVER_OVERRIDE_YAML("heron.apiserver.file.override.yaml", Type.STRING),
+
//keys for config provided user binaries
EXECUTOR_BINARY ("heron.binaries.executor",
"${HERON_BIN}/heron-executor"),
STMGR_BINARY ("heron.binaries.stmgr",
"${HERON_BIN}/heron-stmgr"),
diff --git
a/heron/tools/apiserver/src/java/com/twitter/heron/apiserver/utils/ConfigUtils.java
b/heron/tools/apiserver/src/java/com/twitter/heron/apiserver/utils/ConfigUtils.java
index ddf750a..05e5673 100644
---
a/heron/tools/apiserver/src/java/com/twitter/heron/apiserver/utils/ConfigUtils.java
+++
b/heron/tools/apiserver/src/java/com/twitter/heron/apiserver/utils/ConfigUtils.java
@@ -32,6 +32,7 @@ import com.twitter.heron.common.basics.SysUtils;
import com.twitter.heron.scheduler.utils.SubmitterUtils;
import com.twitter.heron.spi.common.Config;
import com.twitter.heron.spi.common.ConfigLoader;
+import com.twitter.heron.spi.common.Key;
public final class ConfigUtils {
@@ -62,10 +63,17 @@ public final class ConfigUtils {
String releaseFile,
String overrideConfigurationFile) {
// TODO add release file
-return ConfigLoader.loadConfig(heronDirectory,
+Config config = ConfigLoader.loadConfig(heronDirectory,
heronConfigDirectory,
releaseFile,
overrideConfigurationFile);
+// Put location of the override file in the config so that schedulers
invoked by
+// the apiserver can load the override configs if needed. OVERRIDE_YAML
cannot be used
+// to set this because then the location will get passed on to the heron
executors
+return Config.newBuilder()
+.putAll(config)
+.put(Key.APISERVER_OVERRIDE_YAML, overrideConfigurationFile)
+.build();
}
public static Config getTopologyConfig(String topologyPackage, String
topologyBinaryFile,
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
[incubator-heron] branch master updated: Update pidhandler.py (#3094)
This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new c2e9754 Update pidhandler.py (#3094)
c2e9754 is described below
commit c2e97547aff4ada669f82166ffc420972c02f408
Author: Windham Wong
AuthorDate: Wed Oct 31 13:58:07 2018 +0800
Update pidhandler.py (#3094)
Fixing Bash syntax error in pid checking script.
---
heron/shell/src/python/handlers/pidhandler.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/heron/shell/src/python/handlers/pidhandler.py
b/heron/shell/src/python/handlers/pidhandler.py
index b374cdc..da1c5c3 100644
--- a/heron/shell/src/python/handlers/pidhandler.py
+++ b/heron/shell/src/python/handlers/pidhandler.py
@@ -39,5 +39,5 @@ class PidHandler(tornado.web.RequestHandler):
['ps', 'aux'],
['grep', instance_id],
['grep', 'java'],
-['awk', '{print $2}']])).strip())
+['awk', '\'{print $2}\'']])).strip())
self.finish()
[incubator-heron] branch master updated: Fix 'Copyright 2015 The Heron Authors' in scripts (#3036)
This is an automated email from the ASF dual-hosted git repository. karthikz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 9d73f47 Fix 'Copyright 2015 The Heron Authors' in scripts (#3036) 9d73f47 is described below commit 9d73f474aed8f51d1184ee9672fa95dcfa39fd7a Author: Ning Wang AuthorDate: Fri Sep 28 08:07:17 2018 -0700 Fix 'Copyright 2015 The Heron Authors' in scripts (#3036) --- scripts/release/common.sh | 25 ++--- scripts/release/release.sh | 25 ++--- scripts/release/relnotes.sh | 25 ++--- scripts/release/status.sh | 5 +++-- 4 files changed, 45 insertions(+), 35 deletions(-) diff --git a/scripts/release/common.sh b/scripts/release/common.sh index cf4f969..81d583c 100755 --- a/scripts/release/common.sh +++ b/scripts/release/common.sh @@ -1,18 +1,21 @@ #!/bin/bash -eu -# Copyright 2015 The Heron Authors. All rights reserved. +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at # -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 # -#http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. # Some common method for release scripts diff --git a/scripts/release/release.sh b/scripts/release/release.sh index 63f868c..e55dfbc 100755 --- a/scripts/release/release.sh +++ b/scripts/release/release.sh @@ -1,18 +1,21 @@ #!/bin/bash -eu -# Copyright 2015 The Heron Authors. All rights reserved. +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at # -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 # -#http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. # Generate the release branches and handle the release tags. diff --git a/scripts/release/relnotes.sh b/scripts/release/relnotes.sh index e474279..2faeb03 100755 --- a/scripts/release/relnotes.sh +++ b/scripts/release/relnotes.sh @@ -1,18 +1,21 @@ #!/bin/bash -eu -# Copyright 2015 The Heron Authors. All rights reserved. +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at # -# Licensed under the Apache License, Version 2.0 (the "License&
