[incubator-heron] branch master updated: Fix use of non-default base url value (#3535)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 4308eaf Fix use of non-default base url value (#3535)
4308eaf is described below
commit 4308eaf210030056963b55c1f903138b45f22ff1
Author: Oliver Bristow
AuthorDate: Wed Jun 3 02:33:28 2020 +0100
Fix use of non-default base url value (#3535)
---
deploy/kubernetes/helm/templates/tools.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/deploy/kubernetes/helm/templates/tools.yaml
b/deploy/kubernetes/helm/templates/tools.yaml
index 2d3b8d3..dfa63e5 100644
--- a/deploy/kubernetes/helm/templates/tools.yaml
+++ b/deploy/kubernetes/helm/templates/tools.yaml
@@ -116,7 +116,7 @@ spec:
- >-
heron-ui
--port=8889
- --base_url={{ $defaultUrl }}
+ --base_url={{ .Values.heron.url | default $defaultUrl }}
- name: heron-apiserver
image: {{ .Values.image }}
imagePullPolicy: {{ .Values.imagePullPolicy }}
[incubator-heron] branch master updated: Workaround trailing slashes on website (#3531)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new e06dc8f Workaround trailing slashes on website (#3531)
e06dc8f is described below
commit e06dc8ff67bae7ff81abe8a0693ca0a25e9266d6
Author: Oliver Bristow
AuthorDate: Fri May 29 04:49:56 2020 +0100
Workaround trailing slashes on website (#3531)
This is as the website deployment 301s to add trailing slashes, and
Docusurus does not yet support that. Without one of the two yielding,
this workaround strips the trailing slash so the hrefs made still work.
Solution pinched from docusurus issue:
https://github.com/facebook/docusaurus/issues/2394
---
website2/website/siteConfig.js | 3 ++-
website2/website/static/js/fix-location.js | 3 +++
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/website2/website/siteConfig.js b/website2/website/siteConfig.js
index 9651231..e8fa9c9 100644
--- a/website2/website/siteConfig.js
+++ b/website2/website/siteConfig.js
@@ -110,7 +110,8 @@ const siteConfig = {
// Add custom scripts here that would be placed in
[incubator-heron] annotated tag 0.20.3-incubating-rc3 updated (40ae369 -> e9f14e0)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a change to annotated tag 0.20.3-incubating-rc3
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git.
*** WARNING: tag 0.20.3-incubating-rc3 was modified! ***
from 40ae369 (tag)
to e9f14e0 (tag)
tagging c713f438f25e71f95171ea0225543cb3ada8c586 (commit)
replaces 0.20.3-incubating-rc3-test
by Ning Wang
on Wed May 27 01:32:27 2020 -0700
- Log -
Apache Heron release candidate 3: 0.20.3-incubating-rc3
---
from 7bfa0c7 Fix debian9 docker file and clean up centos8 and ubuntu18.04
(#3518)
add c7a5ab4 Adding Ubuntu 20.04 Dockerfiles (#3519)
add cf08e71 Make Java optional in CLI and check JAVA_BIN (#3521)
add a295942 Pin http_file and http_archive resources in WORKSPACE (#3520)
add e065171 Fix centos7 docker build (#3523)
add e0bb08a Set Java SupportedSourceVersion to 11 (#3524)
add 07b5d8c Fix ubuntu16.04 docker file (#3525)
add d05f6e9 Fix Heron Shell PID Extractor (#3528)
add 89a1e50 Update osx documentation (3453) (#3530)
add c713f43 Fix ubunt14.04 docker file (#3526)
No new revisions were added by this update.
Summary of changes:
WORKSPACE | 7 +++
docker/compile/Dockerfile.ubuntu14.04 | 5 -
docker/compile/Dockerfile.ubuntu16.04 | 9 +++-
...ckerfile.ubuntu18.04 => Dockerfile.ubuntu20.04} | 4 +++-
docker/dist/Dockerfile.dist.centos7| 6 --
docker/dist/Dockerfile.dist.ubuntu16.04| 8 ++--
...ist.ubuntu18.04 => Dockerfile.dist.ubuntu20.04} | 4 +++-
.../classification/HeronAnnotationProcessor.java | 4 ++--
heron/shell/src/python/handlers/pidhandler.py | 6 +-
heron/tools/admin/src/python/main.py | 2 --
heron/tools/cli/src/python/execute.py | 7 ++-
heron/tools/cli/src/python/main.py | 3 ---
heron/tools/common/src/python/utils/config.py | 24 +++---
website2/docs/compiling-osx.md | 2 +-
14 files changed, 52 insertions(+), 39 deletions(-)
copy docker/compile/{Dockerfile.ubuntu18.04 => Dockerfile.ubuntu20.04} (96%)
copy docker/dist/{Dockerfile.dist.ubuntu18.04 => Dockerfile.dist.ubuntu20.04}
(97%)
[incubator-heron] annotated tag 0.20.3-incubating-rc3-test3 updated (c713f43 -> b6d2cf7)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to annotated tag 0.20.3-incubating-rc3-test3 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. *** WARNING: tag 0.20.3-incubating-rc3-test3 was modified! *** from c713f43 (commit) to b6d2cf7 (tag) tagging c713f438f25e71f95171ea0225543cb3ada8c586 (commit) replaces 0.20.3-incubating-rc3 by Ning Wang on Tue May 26 11:10:00 2020 -0700 - Log - Test tag for 0.20.3-incubating-rc3 --- No new revisions were added by this update. Summary of changes:
[incubator-heron] branch master updated: Fix ubunt14.04 docker file (#3526)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new c713f43 Fix ubunt14.04 docker file (#3526) c713f43 is described below commit c713f438f25e71f95171ea0225543cb3ada8c586 Author: Ning Wang AuthorDate: Tue May 26 11:09:12 2020 -0700 Fix ubunt14.04 docker file (#3526) --- docker/compile/Dockerfile.ubuntu14.04 | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docker/compile/Dockerfile.ubuntu14.04 b/docker/compile/Dockerfile.ubuntu14.04 index 922cd89..521cce9 100644 --- a/docker/compile/Dockerfile.ubuntu14.04 +++ b/docker/compile/Dockerfile.ubuntu14.04 @@ -34,12 +34,15 @@ RUN apt-get update && apt-get -y install \ git \ libtool \ python-dev \ + pkg-config \ libcppunit-dev \ zip \ unzip \ wget \ cmake \ - tree \ + tree + +RUN apt-get update && apt-get -y install \ openjdk-11-jdk-headless ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64
[incubator-heron] 02/02: Fix ubuntu docker file
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to tag 0.20.3-incubating-rc3-test2 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git commit 6681dc33d759a89861cc456296d3fec96040eeb5 Author: Ning Wang AuthorDate: Sat May 16 23:09:05 2020 -0700 Fix ubuntu docker file --- docker/compile/apiVersion: v1 | 104 docker/dist/Dockerfile.dist.ubuntu16.04 | 8 ++- 2 files changed, 110 insertions(+), 2 deletions(-) diff --git a/docker/compile/apiVersion: v1 b/docker/compile/apiVersion: v1 new file mode 100644 index 000..676f4fb --- /dev/null +++ b/docker/compile/apiVersion: v1 @@ -0,0 +1,104 @@ +apiVersion: v1 +kind: PersistentVolume +metadata: + name: drupal-mysql-pv + labels: +type: local +spec: + storageClassName: mysql-pv + capacity: +storage: 5Gi + accessModes: +- ReadWriteOnce + hostPath: +path: "/drupal-mysql-data" + + + +apiVersion: v1 +kind: PersistentVolume +metadata: + name: drupal-pv + labels: +type: local +spec: + storageClassName: pv + capacity: +storage: 5Gi + accessModes: +- ReadWriteOnce + hostPath: +path: "/drupal-data" + + + +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: drupal-mysql-pvc +spec: + storageClassName: mysql-pv + accessModes: +- ReadWriteOnce + resources: +requests: + storage: 5Gi + + +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: drupal-pvc +spec: + storageClassName: pv + accessModes: +- ReadWriteOnce + resources: +requests: + storage: 5Gi + + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: drupal-mysql +spec: + replicas: 1 + selector: +matchLabels: + app: drupal-mysql + template: +metadata: + labels: +app: drupal-mysql +spec: + containers: +- name: drupal-mysql-container + image: mysql:5.7 + env: + - name: MYSQL_ROOT_PASSWORD +valueFrom: + secretKeyRef: +name: mysql-pass +key: password + ports: + - containerPort: 3306 +name: mysql + volumeMounts: + - name: mysql-persistent-storage +mountPath: /var/lib/mysql +subPath: dbdata + volumes: +- name: mysql-persistent-storage + persistentVolumeClaim: +claimName: drupal-mysql-pvc + + + + +$ kubectl get pvc +Error from server (Forbidden): persistentvolumeclaims is forbidden: User "ning" cannot list resource "persistentvolumeclaims" in API group "" in the namespace "default" + +$ kubectl get pv +Error from server (Forbidden): persistentvolumes is forbidden: User "ning" cannot list resource "persistentvolumes" in API group "" at the cluster scope + diff --git a/docker/dist/Dockerfile.dist.ubuntu16.04 b/docker/dist/Dockerfile.dist.ubuntu16.04 index 0fe59f9..bcf6f37 100644 --- a/docker/dist/Dockerfile.dist.ubuntu16.04 +++ b/docker/dist/Dockerfile.dist.ubuntu16.04 @@ -17,7 +17,9 @@ FROM ubuntu:16.04 -RUN apt-get update +RUN apt-get update && apt-get install -y software-properties-common + +RUN add-apt-repository ppa:openjdk-r/ppa RUN apt-get -y install \ python \ @@ -25,7 +27,9 @@ RUN apt-get -y install \ unzip \ software-properties-common \ curl \ -supervisor \ +supervisor + +RUN apt-get update && apt-get -y install \ openjdk-11-jdk-headless ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64
[incubator-heron] tag 0.20.3-incubating-rc3-test2 created (now 6681dc3)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to tag 0.20.3-incubating-rc3-test2 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. at 6681dc3 (commit) This tag includes the following new commits: new 8d800f5 fix ubuntu 16.04 docker file new 6681dc3 Fix ubuntu docker file The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[incubator-heron] 01/02: fix ubuntu 16.04 docker file
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to tag 0.20.3-incubating-rc3-test2 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git commit 8d800f59625fa639e548b930afdef3ddf3acd86e Author: Ning Wang AuthorDate: Sat May 16 22:58:48 2020 -0700 fix ubuntu 16.04 docker file --- docker/compile/Dockerfile.ubuntu16.04 | 9 - 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docker/compile/Dockerfile.ubuntu16.04 b/docker/compile/Dockerfile.ubuntu16.04 index a3a2265..dbf7b70 100644 --- a/docker/compile/Dockerfile.ubuntu16.04 +++ b/docker/compile/Dockerfile.ubuntu16.04 @@ -21,6 +21,10 @@ FROM ubuntu:16.04 ENV TARGET_PLATFORM ubuntu ENV bazelVersion 3.0.0 +RUN apt-get update && apt-get install -y software-properties-common + +RUN add-apt-repository ppa:openjdk-r/ppa + RUN apt-get update && apt-get -y install \ automake \ build-essential \ @@ -31,12 +35,15 @@ RUN apt-get update && apt-get -y install \ libtool-bin \ libunwind8 \ libunwind-setjmp0-dev \ + pkg-config \ python-dev \ libcppunit-dev \ tree \ zip \ unzip \ - wget \ + wget + +RUN apt-get update && apt-get -y install \ openjdk-11-jdk-headless ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64
[incubator-heron] branch master updated: Set Java SupportedSourceVersion to 11 (#3524)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new e0bb08a Set Java SupportedSourceVersion to 11 (#3524)
e0bb08a is described below
commit e0bb08aa4c75f43a14b9abf0c1891d154dabccd0
Author: Ning Wang
AuthorDate: Wed May 13 21:34:56 2020 -0700
Set Java SupportedSourceVersion to 11 (#3524)
---
.../org/apache/heron/classification/HeronAnnotationProcessor.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/heron/api/src/java/org/apache/heron/classification/HeronAnnotationProcessor.java
b/heron/api/src/java/org/apache/heron/classification/HeronAnnotationProcessor.java
index f600779..09bb91c 100644
---
a/heron/api/src/java/org/apache/heron/classification/HeronAnnotationProcessor.java
+++
b/heron/api/src/java/org/apache/heron/classification/HeronAnnotationProcessor.java
@@ -30,7 +30,7 @@ import javax.lang.model.element.Element;
import javax.lang.model.element.TypeElement;
import javax.tools.Diagnostic.Kind;
-import static javax.lang.model.SourceVersion.RELEASE_8;
+import static javax.lang.model.SourceVersion.RELEASE_11;
/**
* Class that processes usage of {@link InterfaceAudience} and {@link
InterfaceStability} at compile
@@ -39,7 +39,7 @@ import static javax.lang.model.SourceVersion.RELEASE_8;
@SupportedAnnotationTypes({"org.apache.heron.classification.InterfaceStability.Unstable",
"org.apache.heron.classification.InterfaceAudience.Private",
"org.apache.heron.classification.InterfaceAudience.LimitedPrivate"})
-@SupportedSourceVersion(RELEASE_8)
+@SupportedSourceVersion(RELEASE_11)
public class HeronAnnotationProcessor extends AbstractProcessor {
private ProcessingEnvironment env;
[incubator-heron] branch master updated: Adding Ubuntu 20.04 Dockerfiles (#3519)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new c7a5ab4 Adding Ubuntu 20.04 Dockerfiles (#3519) c7a5ab4 is described below commit c7a5ab45e1974d86a19e3eab4a5832a1d8e94600 Author: Nicholas Nezis AuthorDate: Thu Apr 30 19:27:01 2020 -0400 Adding Ubuntu 20.04 Dockerfiles (#3519) --- docker/compile/Dockerfile.ubuntu20.04 | 52 ++ docker/dist/Dockerfile.dist.ubuntu20.04 | 57 + 2 files changed, 109 insertions(+) diff --git a/docker/compile/Dockerfile.ubuntu20.04 b/docker/compile/Dockerfile.ubuntu20.04 new file mode 100644 index 000..c963910 --- /dev/null +++ b/docker/compile/Dockerfile.ubuntu20.04 @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +FROM ubuntu:20.04 + +# This is passed to the heron build command via the --config flag +ENV TARGET_PLATFORM ubuntu +ENV bazelVersion 3.0.0 + +ARG DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && apt-get -y install \ + g++ \ + cmake \ + automake \ + libtool-bin \ + libunwind8 \ + libcppunit-dev \ + patch \ + python-dev \ + pkg-config \ + wget \ + zip \ + virtualenv \ + unzip \ + git \ + curl \ + tree \ + openjdk-11-jdk-headless + +ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64 + +RUN wget -O /tmp/bazel.sh https://github.com/bazelbuild/bazel/releases/download/$bazelVersion/bazel-$bazelVersion-installer-linux-x86_64.sh \ + && chmod +x /tmp/bazel.sh \ + && /tmp/bazel.sh + +ADD bazelrc /root/.bazelrc +ADD scripts/compile-platform.sh /compile-platform.sh diff --git a/docker/dist/Dockerfile.dist.ubuntu20.04 b/docker/dist/Dockerfile.dist.ubuntu20.04 new file mode 100644 index 000..6e22845 --- /dev/null +++ b/docker/dist/Dockerfile.dist.ubuntu20.04 @@ -0,0 +1,57 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +FROM ubuntu:20.04 + +ARG DEBIAN_FRONTEND=noninteractive + +RUN apt-get update +RUN apt-get -y install \ +unzip software-properties-common curl python python3 supervisor openjdk-11-jdk-headless + +ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64 +RUN update-ca-certificates -f + +ADD artifacts /heron + +WORKDIR /heron + +# run heron installers +RUN /heron/heron-install.sh \ +&& rm -f /heron/heron-install.sh + +RUN ln -s /usr/local/heron/dist/heron-core /heron \ +&& mkdir -p /heron/heron-tools \ +&& ln -s /usr/local/heron/bin /heron/heron-tools \ +&& ln -s /usr/local/heron/conf /heron/heron-tools \ +&& ln -s /usr/local/heron/dist /heron/heron-tools \ +&& ln -s /usr/local/heron/lib /heron/heron-tools \ +&& ln -s /usr/local/heron/release.yaml /heron/heron-tools \ +&& ln -s /usr/local/heron/examples /heron \ +&& ln -s /usr/local/heron/release.yaml /heron + +ENV HERON_HOME /heron/heron-core/ +RUN export HERON_HOME + +# install zookeeper +ARG ZK_DIST=zookeeper-3.4.14 +ADD dist/scripts /opt/heron-docker/scripts + +RUN sh /opt/heron-docker/scripts/install-zookeeper.sh $ZK_DIST + +ADD dist/conf/zookeeper.conf /opt/zookeeper/conf/zookeeper.conf +ADD dist/conf/sandbox.conf /etc/supervisor/conf.d/
[incubator-heron] branch release/0.20.3_incubating_rc3 created (now 7bfa0c7)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch release/0.20.3_incubating_rc3 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. at 7bfa0c7 Fix debian9 docker file and clean up centos8 and ubuntu18.04 (#3518) No new revisions were added by this update.
[incubator-heron] annotated tag 0.20.3-incubating-rc3 updated (0b510ff -> 40ae369)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to annotated tag 0.20.3-incubating-rc3 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. *** WARNING: tag 0.20.3-incubating-rc3 was modified! *** from 0b510ff (tag) to 40ae369 (tag) tagging 7bfa0c732d5c32439cfe1ec07d9fb6ea9994dc34 (commit) replaces 0.20.3-incubating-rc3-test by Ning Wang on Thu Apr 23 09:24:36 2020 -0700 - Log - Version 0.20.3-incubating release candidate 3 --- from 923b8b9 Add cppunit-devel for Bazel 3 (#3513) add 78d7145 Adding missing pkg-config package to Debian 9 build (#3517) add ec7a87a Update issue templates (#3379) add 7bfa0c7 Fix debian9 docker file and clean up centos8 and ubuntu18.04 (#3518) No new revisions were added by this update. Summary of changes: .github/ISSUE_TEMPLATE/bug_report.md | 31 +++ .github/ISSUE_TEMPLATE/custom.md | 10 ++ .github/ISSUE_TEMPLATE/feature_request.md | 20 docker/base/Dockerfile.base.debian9 | 2 +- docker/compile/Dockerfile.debian9 | 1 + docker/test/Dockerfile.centos7| 3 ++- docker/test/Dockerfile.ubuntu18.04| 3 ++- 7 files changed, 67 insertions(+), 3 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/custom.md create mode 100644 .github/ISSUE_TEMPLATE/feature_request.md
[incubator-heron] branch master updated: Fix debian9 docker file and clean up centos8 and ubuntu18.04 (#3518)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 7bfa0c7 Fix debian9 docker file and clean up centos8 and ubuntu18.04 (#3518) 7bfa0c7 is described below commit 7bfa0c732d5c32439cfe1ec07d9fb6ea9994dc34 Author: Ning Wang AuthorDate: Thu Apr 23 09:23:21 2020 -0700 Fix debian9 docker file and clean up centos8 and ubuntu18.04 (#3518) --- docker/base/Dockerfile.base.debian9 | 2 +- docker/test/Dockerfile.centos7 | 3 ++- docker/test/Dockerfile.ubuntu18.04 | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docker/base/Dockerfile.base.debian9 b/docker/base/Dockerfile.base.debian9 index 27892fc..3a4ba7f 100644 --- a/docker/base/Dockerfile.base.debian9 +++ b/docker/base/Dockerfile.base.debian9 @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -FROM openjdk:8-jdk-slim +FROM openjdk:11-jdk-slim-stretch RUN apt-get -y update && apt-get -y install \ netcat-openbsd \ diff --git a/docker/test/Dockerfile.centos7 b/docker/test/Dockerfile.centos7 index 8122b73..3a36273 100644 --- a/docker/test/Dockerfile.centos7 +++ b/docker/test/Dockerfile.centos7 @@ -19,7 +19,7 @@ FROM centos:centos7 # This is passed to the heron build command via the --config flag ENV TARGET_PLATFORM centos -ENV bazelVersion 0.26.0 +ENV bazelVersion 3.0.0 RUN yum -y upgrade RUN yum -y install \ @@ -33,6 +33,7 @@ RUN yum -y install \ git \ kernel-devel \ libtool \ + cppunit-devel \ make \ patch \ python-devel \ diff --git a/docker/test/Dockerfile.ubuntu18.04 b/docker/test/Dockerfile.ubuntu18.04 index e9c2e82..2586947 100644 --- a/docker/test/Dockerfile.ubuntu18.04 +++ b/docker/test/Dockerfile.ubuntu18.04 @@ -19,7 +19,7 @@ FROM ubuntu:18.04 # This is passed to the heron build command via the --config flag ENV TARGET_PLATFORM ubuntu -ENV bazelVersion 0.26.0 +ENV bazelVersion 3.0.0 RUN apt-get update && apt-get -y install \ g++ \ @@ -27,6 +27,7 @@ RUN apt-get update && apt-get -y install \ automake \ libtool-bin \ libunwind8 \ + libcppunit-dev \ patch \ python-dev \ python3-dev \
[incubator-heron] annotated tag 0.20.3-incubating-rc3 updated (923b8b9 -> 0b510ff)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to annotated tag 0.20.3-incubating-rc3 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. *** WARNING: tag 0.20.3-incubating-rc3 was modified! *** from 923b8b9 (commit) to 0b510ff (tag) tagging 923b8b9ef22a31ed73fdc0ad2e004e0eeb407681 (commit) replaces 0.20.3-incubating-rc2 by Ning Wang on Tue Apr 21 00:45:49 2020 -0700 - Log - Apache Heron release 0.20.3-incubating release candidate 3 --- No new revisions were added by this update. Summary of changes:
[incubator-heron] annotated tag 0.20.3-incubating-rc3-test updated (923b8b9 -> 97e17b3)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to annotated tag 0.20.3-incubating-rc3-test in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. *** WARNING: tag 0.20.3-incubating-rc3-test was modified! *** from 923b8b9 (commit) to 97e17b3 (tag) tagging 923b8b9ef22a31ed73fdc0ad2e004e0eeb407681 (commit) replaces 0.20.3-incubating-rc2 by Ning Wang on Mon Apr 20 19:02:52 2020 -0700 - Log - Apache Heron 0.20.3-incubating release candidate 3 --- No new revisions were added by this update. Summary of changes:
[incubator-heron] branch master updated: Add cppunit-devel for Bazel 3 (#3513)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 923b8b9 Add cppunit-devel for Bazel 3 (#3513) 923b8b9 is described below commit 923b8b9ef22a31ed73fdc0ad2e004e0eeb407681 Author: choi se AuthorDate: Mon Apr 20 01:58:35 2020 +0900 Add cppunit-devel for Bazel 3 (#3513) * Add cppunit-devel for centos Dockerfile * Add cppunit-devel for Dockerfile --- docker/compile/Dockerfile.centos7 | 1 + docker/compile/Dockerfile.debian9 | 1 + docker/compile/Dockerfile.ubuntu14.04 | 1 + docker/compile/Dockerfile.ubuntu16.04 | 1 + 4 files changed, 4 insertions(+) diff --git a/docker/compile/Dockerfile.centos7 b/docker/compile/Dockerfile.centos7 index 5e3bc49..c117ee9 100644 --- a/docker/compile/Dockerfile.centos7 +++ b/docker/compile/Dockerfile.centos7 @@ -36,6 +36,7 @@ RUN yum -y install \ make \ patch \ python-devel \ + cppunit-devel \ zip \ unzip \ wget \ diff --git a/docker/compile/Dockerfile.debian9 b/docker/compile/Dockerfile.debian9 index 9572652..fa2b2bc 100644 --- a/docker/compile/Dockerfile.debian9 +++ b/docker/compile/Dockerfile.debian9 @@ -30,6 +30,7 @@ RUN apt-get update && apt-get -y install \ git \ libtool \ libtool-bin \ + libcppunit-dev \ python-dev \ python3-dev \ software-properties-common \ diff --git a/docker/compile/Dockerfile.ubuntu14.04 b/docker/compile/Dockerfile.ubuntu14.04 index 8ba7bea..922cd89 100644 --- a/docker/compile/Dockerfile.ubuntu14.04 +++ b/docker/compile/Dockerfile.ubuntu14.04 @@ -34,6 +34,7 @@ RUN apt-get update && apt-get -y install \ git \ libtool \ python-dev \ + libcppunit-dev \ zip \ unzip \ wget \ diff --git a/docker/compile/Dockerfile.ubuntu16.04 b/docker/compile/Dockerfile.ubuntu16.04 index 0befaae..a3a2265 100644 --- a/docker/compile/Dockerfile.ubuntu16.04 +++ b/docker/compile/Dockerfile.ubuntu16.04 @@ -32,6 +32,7 @@ RUN apt-get update && apt-get -y install \ libunwind8 \ libunwind-setjmp0-dev \ python-dev \ + libcppunit-dev \ tree \ zip \ unzip \
[incubator-heron] branch master updated: Fix style
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 0f27ef4 Fix style
0f27ef4 is described below
commit 0f27ef46aa4fa0871c75b398892982261e90b8d7
Author: Ning Wang
AuthorDate: Mon Apr 6 23:53:59 2020 -0700
Fix style
---
.../src/java/org/apache/heron/apiserver/utils/ConfigUtils.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/heron/tools/apiserver/src/java/org/apache/heron/apiserver/utils/ConfigUtils.java
b/heron/tools/apiserver/src/java/org/apache/heron/apiserver/utils/ConfigUtils.java
index e6aa022..c3de5e9 100644
---
a/heron/tools/apiserver/src/java/org/apache/heron/apiserver/utils/ConfigUtils.java
+++
b/heron/tools/apiserver/src/java/org/apache/heron/apiserver/utils/ConfigUtils.java
@@ -139,7 +139,8 @@ public final class ConfigUtils {
) {
stateManagerReader = Files.newBufferedReader(stateManagerPath);
- final Map overrides = (Map) new Yaml(new
SafeConstructor()).load(overrideReader);
+ final Map overrides =
+ (Map) new Yaml(new
SafeConstructor()).load(overrideReader);
final Map stateMangerConfig =
(Map) new Yaml(new
SafeConstructor()).load(stateManagerReader);
// update the state manager config with the overrides
[incubator-heron] branch master updated: Use safe constructor in yaml deserialization
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 9d77767 Use safe constructor in yaml deserialization
9d77767 is described below
commit 9d7776702750743f536300dd0a2fa7555ac16bf3
Author: Ning Wang
AuthorDate: Sun Apr 5 17:21:31 2020 -0700
Use safe constructor in yaml deserialization
---
.../java/org/apache/heron/common/config/ConfigReader.java | 5 +++--
.../org/apache/heron/metricsmgr/MetricsSinksConfig.java | 3 ++-
.../org/apache/heron/apiserver/utils/ConfigUtils.java | 7 ---
.../apache/heron/apiserver/utils/ConfigUtilsTests.java| 15 ---
4 files changed, 17 insertions(+), 13 deletions(-)
diff --git
a/heron/common/src/java/org/apache/heron/common/config/ConfigReader.java
b/heron/common/src/java/org/apache/heron/common/config/ConfigReader.java
index cb0a3ca..a17c72f 100644
--- a/heron/common/src/java/org/apache/heron/common/config/ConfigReader.java
+++ b/heron/common/src/java/org/apache/heron/common/config/ConfigReader.java
@@ -32,6 +32,7 @@ import java.util.logging.Level;
import java.util.logging.Logger;
import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.constructor.SafeConstructor;
/**
* Loads config file in Yaml file format.
@@ -79,7 +80,7 @@ public final class ConfigReader {
try {
FileInputStream fin = new FileInputStream(new File(fileName));
try {
- Yaml yaml = new Yaml();
+ Yaml yaml = new Yaml(new SafeConstructor());
propsYaml = (Map) yaml.load(fin);
LOG.log(Level.FINE, "Successfully read config file {0}", fileName);
} finally {
@@ -104,7 +105,7 @@ public final class ConfigReader {
public static Map loadStream(InputStream inputStream) {
LOG.fine("Reading config stream");
-Yaml yaml = new Yaml();
+Yaml yaml = new Yaml(new SafeConstructor());
Map propsYaml = (Map)
yaml.load(inputStream);
LOG.fine("Successfully read config");
diff --git
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsSinksConfig.java
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsSinksConfig.java
index ff989cd..f0655bb 100644
---
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsSinksConfig.java
+++
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsSinksConfig.java
@@ -29,6 +29,7 @@ import java.util.List;
import java.util.Map;
import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.constructor.SafeConstructor;
import org.apache.heron.common.basics.TypeUtils;
@@ -62,7 +63,7 @@ public class MetricsSinksConfig {
return Collections.emptyMap();
}
-Yaml yaml = new Yaml();
+Yaml yaml = new Yaml(new SafeConstructor());
try (InputStream inputStream = new FileInputStream(configFile)) {
return (Map) yaml.load(inputStream);
}
diff --git
a/heron/tools/apiserver/src/java/org/apache/heron/apiserver/utils/ConfigUtils.java
b/heron/tools/apiserver/src/java/org/apache/heron/apiserver/utils/ConfigUtils.java
index 8a8269b..e6aa022 100644
---
a/heron/tools/apiserver/src/java/org/apache/heron/apiserver/utils/ConfigUtils.java
+++
b/heron/tools/apiserver/src/java/org/apache/heron/apiserver/utils/ConfigUtils.java
@@ -30,6 +30,7 @@ import java.util.Properties;
import org.yaml.snakeyaml.DumperOptions;
import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.constructor.SafeConstructor;
import org.apache.heron.api.exception.InvalidTopologyException;
import org.apache.heron.api.generated.TopologyAPI;
@@ -110,7 +111,7 @@ public final class ConfigUtils {
try (Writer writer = Files.newBufferedWriter(tempOverridesPath)) {
overrideReader = Files.newBufferedReader(overridesPath);
final Map currentOverrides =
- (Map) new Yaml().load(overrideReader);
+ (Map) new Yaml(new
SafeConstructor()).load(overrideReader);
currentOverrides.putAll(overrides);
// write updated overrides
@@ -138,9 +139,9 @@ public final class ConfigUtils {
) {
stateManagerReader = Files.newBufferedReader(stateManagerPath);
- final Map overrides = (Map) new
Yaml().load(overrideReader);
+ final Map overrides = (Map) new Yaml(new
SafeConstructor()).load(overrideReader);
final Map stateMangerConfig =
- (Map) new Yaml().load(stateManagerReader);
+ (Map) new Yaml(new
SafeConstructor()).load(stateManagerReader);
// update the state manager config with the overrides
for (Map.Entry entry : overrides.entrySet()) {
// does this key have an override?
diff --git
a/heron/tools/apiserver/tests/java/org/apache/heron/apiserver/utils/ConfigUtilsTests.java
b/heron/tools/apiserver/tests/java/org/apache/heron/apiserver/utils/ConfigUtilsTests.java
index 7fd110c..2fb
[incubator-heron] branch master updated: Support custom metrics rules for PrometheusSink (#3493)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new b069e45 Support custom metrics rules for PrometheusSink (#3493)
b069e45 is described below
commit b069e450a78c2f77ba5427c3f1aceda1e881e9f2
Author: choi se
AuthorDate: Thu Apr 2 14:14:28 2020 +0900
Support custom metrics rules for PrometheusSink (#3493)
* Support Java 11
* config travis to use oracle jdk 11
* Java 11 support (#3399)
* Support Java 11
* config travis to use oracle jdk 11
* Add check jdk version
* Fix command arguments.
Change insert gc_options
Update list
Fix gc-logging
* Add missing parameter
* typo
* Add pause time
* wip
* Support jmx_exporter format configuration.
* Fix checkstyle
* Remove unused
* Java 11 support (#3399)
* Support Java 11
* config travis to use oracle jdk 11
* Add check jdk version
* Fix command arguments.
Change insert gc_options
Update list
Fix gc-logging
* wip
* Support jmx_exporter format configuration.
* Fix checkstyle
* Remove unused
* Update kafkaOffset metrics
* Add Rules
* Make log/sink/consume Streamlet component support setName and
setNumPartitions (#3459)
* Patch to fix cppcheck with newer glibc (#3471)
* Add documents for setting up a docker based development environment
(#3475)
* Improve concurrency for needed parts. (#3107)
* Change concurrent Map
* Change concurrent Map
* HashMap changes for unneeded parts.
* HashMap changes for unneeded parts.
* Review changes
* Changes HashMap for unneeded parts.
* Improve concurrency for needed parts.
* Remove unused imports.
* Remove unused imports.
* Remove unused imports.
* Fix NPE
(cherry picked from commit 545d3814b315c29d3e396309a2ededaad193ec32)
* Fix WhitespaceAround
* Add dummy Object
* Fix ConstantName
(cherry picked from commit 8d6d5067072e92d6e276f93e18297ddedc647c6c)
* Update kafkaOffset metrics
* Add Rules
* Update line is longer than 100 characters
* Update line is longer than 100 characters
* Add attrNameSnakeCase or other metrics fix
* fix checkstyle
Co-authored-by: Ning Wang
Co-authored-by: Ning Wang
Co-authored-by: Nicholas Nezis
---
.../heron/metricsmgr/sink/PrometheusSink.java | 236 ++---
.../heron/metricsmgr/sink/PrometheusSinkTests.java | 124 ++-
2 files changed, 324 insertions(+), 36 deletions(-)
diff --git
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java
index 0db442a..56a244b 100644
---
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java
+++
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java
@@ -20,10 +20,17 @@
package org.apache.heron.metricsmgr.sink;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.logging.Level;
import java.util.logging.Logger;
+import java.util.regex.Matcher;
import java.util.regex.Pattern;
import com.google.common.cache.Cache;
@@ -33,6 +40,9 @@ import org.apache.heron.spi.metricsmgr.metrics.MetricsInfo;
import org.apache.heron.spi.metricsmgr.metrics.MetricsRecord;
import org.apache.heron.spi.metricsmgr.sink.SinkContext;
+import static java.lang.String.format;
+import static
org.apache.heron.metricsmgr.sink.PrometheusSink.Prometheus.sanitizeMetricName;
+
/**
* A web sink that exposes and endpoint that Prometheus can scrape
*
@@ -57,6 +67,7 @@ public class PrometheusSink extends AbstractWebSink {
// This is the cache that is used to serve the metrics
private Cache> metricsCache;
+ private List rules = new ArrayList();
private String cluster;
private String role;
@@ -66,6 +77,26 @@ public class PrometheusSink extends AbstractWebSink {
super();
}
+ private enum Type {
+COUNTER,
+GAUGE,
+SUMMARY,
+HISTOGRAM,
+UNTYPED,
+ }
+
+ private static class Rule {
+public Pattern pattern;
+public String name;
+public String value;
+public Double valueFactor = 1.0;
+public String help;
+public boolean attrNameSnakeCase;
+public Type t
[incubator-heron] branch master updated: Update Debian docker build with JDK11 and correct debian version (#3487)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 43c7c74 Update Debian docker build with JDK11 and correct debian version (#3487) 43c7c74 is described below commit 43c7c748967e4fd15d0aeabd63e97ab48ad89f8d Author: Windham Wong AuthorDate: Thu Mar 12 09:15:59 2020 +0800 Update Debian docker build with JDK11 and correct debian version (#3487) * Debian9 docker build fix * Fix Debian docker build with JDK11 Co-authored-by: tahlia --- docker/compile/Dockerfile.debian10 | 2 +- docker/compile/Dockerfile.debian9 | 7 ++- docker/dist/Dockerfile.dist.debian9 | 7 +++ 3 files changed, 6 insertions(+), 10 deletions(-) diff --git a/docker/compile/Dockerfile.debian10 b/docker/compile/Dockerfile.debian10 index c240b98..70e15d0 100644 --- a/docker/compile/Dockerfile.debian10 +++ b/docker/compile/Dockerfile.debian10 @@ -31,7 +31,7 @@ RUN apt-get update && apt-get -y install \ libtool \ libtool-bin \ python \ - python2.7-dev \ + python-dev \ software-properties-common \ python-setuptools \ tree \ diff --git a/docker/compile/Dockerfile.debian9 b/docker/compile/Dockerfile.debian9 index 0af0330..223c4eb 100644 --- a/docker/compile/Dockerfile.debian9 +++ b/docker/compile/Dockerfile.debian9 @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -FROM openjdk:8-jdk-slim +FROM openjdk:11-jdk-slim-stretch # This is passed to the heron build command via the --config flag ENV TARGET_PLATFORM debian @@ -37,10 +37,7 @@ RUN apt-get update && apt-get -y install \ tree \ zip \ unzip \ - wget \ - openjdk-11-jdk-headless - -ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64 + wget RUN wget -O /tmp/bazel.sh https://github.com/bazelbuild/bazel/releases/download/$bazelVersion/bazel-$bazelVersion-installer-linux-x86_64.sh \ && chmod +x /tmp/bazel.sh \ diff --git a/docker/dist/Dockerfile.dist.debian9 b/docker/dist/Dockerfile.dist.debian9 index 7ba3645..5d56c1e 100644 --- a/docker/dist/Dockerfile.dist.debian9 +++ b/docker/dist/Dockerfile.dist.debian9 @@ -15,17 +15,16 @@ # specific language governing permissions and limitations # under the License. -FROM openjdk:8-jdk-slim +FROM openjdk:11-jdk-slim-stretch RUN apt-get -y update && apt-get -y install \ netcat-openbsd \ python \ -python3 \ +python-dev \ unzip \ curl \ vim \ -supervisor \ -openjdk-11-jdk-headless && \ +supervisor && \ apt-get clean all && \ rm -rf /var/lib/apt/lists/*
[incubator-heron] branch master updated: Patch to fix cppcheck with newer glibc (#3471)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new a6ec96e Patch to fix cppcheck with newer glibc (#3471)
a6ec96e is described below
commit a6ec96e8102693c417f44bf69c7805615f06b917
Author: Nicholas Nezis
AuthorDate: Sat Feb 29 05:49:20 2020 -0500
Patch to fix cppcheck with newer glibc (#3471)
---
WORKSPACE | 8 +---
third_party/cppcheck/cppcheck-readdir-fix.patch | 12
2 files changed, 17 insertions(+), 3 deletions(-)
diff --git a/WORKSPACE b/WORKSPACE
index ac4393f..065d85b 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -930,10 +930,12 @@ http_archive(
http_archive(
name = "com_github_danmar_cppcheck",
-urls = ["https://github.com/danmar/cppcheck/archive/1.87.zip;],
-strip_prefix = "cppcheck-1.87",
build_file = "@//:third_party/cppcheck/cppcheck.BUILD",
-sha256 =
"b3de7fbdc1a23d7341b55f7f88877e106a76847bd5a07fa721c07310b625318b",
+patch_args = ["-p2"],
+patches = ["//third_party/cppcheck:cppcheck-readdir-fix.patch"],
+sha256 =
"cb0e66cbe2d6b655fce430cfaaa74b83ad11c91f221e3926f1ca3211bb7c906b",
+strip_prefix = "cppcheck-1.90",
+urls = ["https://github.com/danmar/cppcheck/archive/1.90.zip;],
)
http_archive(
diff --git a/third_party/cppcheck/cppcheck-readdir-fix.patch
b/third_party/cppcheck/cppcheck-readdir-fix.patch
new file mode 100644
index 000..bde961a
--- /dev/null
+++ b/third_party/cppcheck/cppcheck-readdir-fix.patch
@@ -0,0 +1,12 @@
+diff -Naur cppcheck/cppcheck-1.90/cli/filelister.cpp
cppcheck-fixed/cppcheck-1.90/cli/filelister.cpp
+--- cppcheck/cppcheck-1.90/cli/filelister.cpp 2020-02-20 22:42:28.0
-0500
cppcheck-fixed/cppcheck-1.90/cli/filelister.cpp2020-02-20
22:44:48.0 -0500
+@@ -194,7 +194,7 @@
+ std::string new_path;
+ new_path.reserve(path.length() + 100);// prealloc some memory to
avoid constant new/deletes in loop
+
+-while ((readdir_r(dir, , _result) == 0) && (dir_result
!= nullptr)) {
++while ((dir_result = readdir(dir)) != NULL) {
+
+ if ((std::strcmp(dir_result->d_name, ".") == 0) ||
+ (std::strcmp(dir_result->d_name, "..") == 0))
[incubator-heron] branch master updated (1c59862 -> f0b6e1b)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 1c59862 Update README for Java 11 (#3464) add f0b6e1b Make log/sink/consume Streamlet component support setName and setNumPartitions (#3459) No new revisions were added by this update. Summary of changes: .../examples/streamlet/StreamletCloneTopology.java | 6 +- .../scala/ScalaClassicalMusicTopology.scala| 1 + .../scala/ScalaIntegerProcessingTopology.scala | 2 + .../java/org/apache/heron/streamlet/Streamlet.java | 8 +- .../org/apache/heron/streamlet/StreamletBase.java | 60 ++ .../apache/heron/streamlet/impl/BuilderImpl.java | 2 +- .../heron/streamlet/impl/StreamletBaseImpl.java| 224 + .../apache/heron/streamlet/impl/StreamletImpl.java | 162 ++- .../heron/streamlet/impl/StreamletShadow.java | 6 +- .../impl/streamlets/ConsumerStreamlet.java | 3 +- .../streamlet/impl/streamlets/LogStreamlet.java| 3 +- .../streamlet/impl/streamlets/SinkStreamlet.java | 3 +- .../apache/heron/streamlet/scala/Streamlet.scala | 22 +- .../heron/streamlet/scala/StreamletBase.scala | 58 ++ .../streamlet/scala/impl/StreamletBaseImpl.scala | 71 +++ .../heron/streamlet/scala/impl/StreamletImpl.scala | 40 ++-- .../heron/streamlet/impl/StreamletImplTest.java| 10 +- 17 files changed, 472 insertions(+), 209 deletions(-) create mode 100644 heron/api/src/java/org/apache/heron/streamlet/StreamletBase.java create mode 100644 heron/api/src/java/org/apache/heron/streamlet/impl/StreamletBaseImpl.java create mode 100644 heron/api/src/scala/org/apache/heron/streamlet/scala/StreamletBase.scala create mode 100644 heron/api/src/scala/org/apache/heron/streamlet/scala/impl/StreamletBaseImpl.scala
[incubator-heron] branch master updated: Update website for java 11 (#3462)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 4379fe1 Update website for java 11 (#3462)
4379fe1 is described below
commit 4379fe1a30241140a771fe1ad6b2708eea994cdd
Author: Ning Wang
AuthorDate: Wed Feb 19 22:52:40 2020 -0800
Update website for java 11 (#3462)
---
website2/docs/compiling-code-organization.md| 5 ++-
website2/docs/compiling-docker.md | 9 +++--
website2/docs/compiling-linux.md| 48 ++---
website2/docs/compiling-overview.md | 4 +--
website2/docs/heron-streamlet-concepts.md | 2 +-
website2/website/scripts/Dockerfile.ubuntu18.04 | 4 +--
6 files changed, 21 insertions(+), 51 deletions(-)
diff --git a/website2/docs/compiling-code-organization.md
b/website2/docs/compiling-code-organization.md
index bba8ac4..053bc48 100644
--- a/website2/docs/compiling-code-organization.md
+++ b/website2/docs/compiling-code-organization.md
@@ -35,12 +35,11 @@ The primary programming languages for Heron are C++, Java,
and Python.
[Topology Master](heron-architecture#topology-master), and
[Stream Manager](heron-architecture#stream-manager).
-* **Java 8** is used primarily for Heron's [topology
+* **Java 11** is used primarily for Heron's [topology
API](heron-topology-concepts), and [Heron
Instance](heron-architecture#heron-instance).
It is currently the only language in which topologies can be written.
Instructions can be found
in [Building Topologies](../../developers/java/topologies), while
documentation for the Java
-API can be found
[here](/api/org/apache/heron/api/topology/package-summary.html). Please note
that Heron topologies do not
-require Java 8 and can be written in Java 7 or later.
+API can be found
[here](/api/org/apache/heron/api/topology/package-summary.html). Please note
that Heron topologies do not require Java 11 and can be written in Java 7 or
later.
* **Python 2** (specifically 2.7) is used primarily for Heron's [CLI
interface](user-manuals-heron-cli) and UI components such as [Heron
UI](user-manuals-heron-ui) and the [Heron
Tracker](user-manuals-heron-tracker-runbook).
diff --git a/website2/docs/compiling-docker.md
b/website2/docs/compiling-docker.md
index 7050102..c33bacd 100644
--- a/website2/docs/compiling-docker.md
+++ b/website2/docs/compiling-docker.md
@@ -169,7 +169,7 @@ RUN apt-get update && apt-get -y install \
wget
```
-### Step 4 --- An installation script for Java 8 and a `JAVA_HOME` environment
variable
+### Step 4 --- An installation script for Java 11 and a `JAVA_HOME`
environment variable
Here's an example:
@@ -178,11 +178,10 @@ RUN \
echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select
true | debconf-set-selections && \
add-apt-repository -y ppa:webupd8team/java && \
apt-get update && \
- apt-get install -y oracle-java8-installer && \
- rm -rf /var/lib/apt/lists/* && \
- rm -rf /var/cache/oracle-jdk8-installer
+ apt-get install -y openjdk-11-jdk-headless && \
+ rm -rf /var/lib/apt/lists/*
-ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
+ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64
```
Step 5 - An installation script for [Bazel](http://bazel.io/) version {{%
bazelVersion %}} or above.
diff --git a/website2/docs/compiling-linux.md b/website2/docs/compiling-linux.md
index 0be85f9..e40cb03 100644
--- a/website2/docs/compiling-linux.md
+++ b/website2/docs/compiling-linux.md
@@ -22,12 +22,12 @@ sidebar_label: Compiling on Linux
Heron can currently be built on the following Linux platforms:
-* [Ubuntu 14.04](#building-on-ubuntu-14.04)
+* [Ubuntu 14.04](#building-on-ubuntu-18.04)
* [CentOS 7](#building-on-centos-7)
## Building on Ubuntu 14.04
-To build Heron on a fresh Ubuntu 14.04 installation:
+To build Heron on a fresh Ubuntu 18.04 installation:
### Step 1 --- Update Ubuntu
@@ -50,19 +50,19 @@ export CC=/usr/bin/gcc
export CCX=/usr/bin/g++
```
-### Step 4 --- Install JDK 8 and set JAVA_HOME
+### Step 4 --- Install JDK 11 and set JAVA_HOME
```bash
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update -y
-$ sudo apt-get install oracle-java8-installer -y
-$ export JAVA_HOME="/usr/lib/jvm/java-8-oracle"
+$ sudo apt-get install openjdk-11-jdk-headless -y
+$ export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"
```
Step 5 - Install Bazel {{% bazelVersion %}}
```bash
-wget -O /tmp/bazel.sh
https://github.com/bazelbuild/bazel/releases/download/0.23.2/bazel-0.23.2-installer-linux-x86_64.sh
+wget -O /tmp/bazel.sh
https://github.com/bazelbuild/bazel/releases/download/0.26.0/bazel-0.26.0-installer-linux-x86_64.sh
chmod +x /tmp/bazel.sh
/tmp/bazel.sh --user
[incubator-heron] branch master updated: Java 11 support (#3454)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 1476076 Java 11 support (#3454)
1476076 is described below
commit 14760762b052ed9a6c8aea2988beb5108ae9fc79
Author: se choi
AuthorDate: Thu Feb 20 01:17:29 2020 +0900
Java 11 support (#3454)
* Support Java 11
* config travis to use oracle jdk 11
* Java 11 support (#3399)
* Support Java 11
* config travis to use oracle jdk 11
* Add check jdk version
* Fix command arguments.
Change insert gc_options
Update list
Fix gc-logging
* Add missing parameter
* typo
* Add pause time
* Add python3 for docker files
* Revert "config travis to use oracle jdk 11"
This reverts commit edaa84f4a6665a1bbd4a93feb67eb67f62476d9a.
* Update unitest
* Update unittest
* Add JDK11 UnitTest
* Add Unittest using Docker.
* Add full test
* Change JDK11
* Ignore PowerMock used for testing is not compatible with Java 11.
* Change JDK11 for travis.yml
* Update python
* Update library requests, pylint
* Revert "Update library requests, pylint"
This reverts commit 7706d34a7b108846b1d4ada8c74e7f6ce65911c0.
* Add ignore
* Refactoring ReflectionUtils
Co-authored-by: Ning Wang
---
.travis.yml| 2 +-
WORKSPACE | 13 +-
docker/compile/Dockerfile.centos7 | 10 +-
docker/compile/Dockerfile.debian9 | 11 +-
docker/compile/Dockerfile.ubuntu14.04 | 12 +-
docker/compile/Dockerfile.ubuntu16.04 | 15 +-
docker/compile/Dockerfile.ubuntu18.04 | 5 +-
docker/dist/Dockerfile.dist.centos7| 6 +-
docker/dist/Dockerfile.dist.debian9| 4 +-
docker/dist/Dockerfile.dist.ubuntu14.04| 5 +-
docker/dist/Dockerfile.dist.ubuntu16.04| 5 +-
docker/dist/Dockerfile.dist.ubuntu18.04| 4 +-
docker/scripts/test-docker.sh | 64 +++
docker/scripts/test-platform.sh| 76
docker/scripts/test-unittest.sh| 175 ++
docker/{compile => test}/Dockerfile.centos7| 11 +-
docker/{compile => test}/Dockerfile.ubuntu18.04| 7 +-
.../apache/heron/eco/submit/EcoSubmitterTest.java | 2 +
heron/api/src/java/BUILD | 3 +-
.../org/apache/heron/api/HeronSubmitterTest.java | 2 +
heron/common/src/java/BUILD| 1 +
.../apache/heron/common/basics/FileUtilsTest.java | 2 +
.../apache/heron/downloader/DLDownloaderTest.java | 2 +
heron/executor/src/python/heron_executor.py| 129 ++
.../tests/python/heron_executor_unittest.py| 196 -
heron/scheduler-core/src/java/BUILD| 1 +
.../apache/heron/scheduler/LaunchRunnerTest.java | 2 +
.../heron/scheduler/RuntimeManagerMainTest.java| 2 +
.../heron/scheduler/RuntimeManagerRunnerTest.java | 2 +
.../apache/heron/scheduler/SchedulerMainTest.java | 2 +
.../apache/heron/scheduler/SubmitterMainTest.java | 2 +
.../heron/scheduler/UpdateTopologyManagerTest.java | 2 +
.../client/HttpServiceSchedulerClientTest.java | 2 +
.../client/SchedulerClientFactoryTest.java | 2 +
.../scheduler/dryrun/SubmitDryRunRenderTest.java | 2 +
.../scheduler/dryrun/UpdateDryRunRenderTest.java | 2 +
.../heron/scheduler/utils/LauncherUtilsTest.java | 2 +
.../heron/scheduler/utils/SchedulerUtilsTest.java | 2 +
.../heron/scheduler/aurora/AuroraLauncherTest.java | 2 +
.../scheduler/aurora/AuroraSchedulerTest.java | 2 +
.../kubernetes/KubernetesLauncherTest.java | 2 +
.../heron/scheduler/local/LocalLauncherTest.java | 2 +
.../scheduler/marathon/MarathonControllerTest.java | 2 +
.../scheduler/marathon/MarathonLauncherTest.java | 2 +
.../heron/scheduler/nomad/NomadSchedulerTest.java | 2 +-
.../heron/scheduler/slurm/SlurmControllerTest.java | 2 +
.../heron/scheduler/slurm/SlurmLauncherTest.java | 2 +
.../heron/scheduler/slurm/SlurmSchedulerTest.java | 2 +
.../scheduler/yarn/HeronExecutorTaskTest.java | 2 +-
.../scheduler/yarn/HeronMasterDriverTest.java | 2 +
heron/simulator/src/java/BUILD | 1 +
.../apache/heron/spi/utils/ReflectionUtils.java| 6 +-
.../apache/heron/spi/common/ConfigLoaderTest.java | 2 +
.../org/apache/heron/spi/common/ConfigTest.java| 3 +-
.../apache/heron/spi/utils/Net
[incubator-heron] branch master updated: Use netcat-bsd in Debian (#3447)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new d43f6bd Use netcat-bsd in Debian (#3447) d43f6bd is described below commit d43f6bd0266d8d20a1883a087f922a7925730c0c Author: Ning Wang AuthorDate: Sun Feb 9 12:18:43 2020 -0800 Use netcat-bsd in Debian (#3447) --- docker/base/Dockerfile.base.debian9 | 2 +- docker/dist/Dockerfile.dist.debian9 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docker/base/Dockerfile.base.debian9 b/docker/base/Dockerfile.base.debian9 index ebbccf4..27892fc 100644 --- a/docker/base/Dockerfile.base.debian9 +++ b/docker/base/Dockerfile.base.debian9 @@ -18,7 +18,7 @@ FROM openjdk:8-jdk-slim RUN apt-get -y update && apt-get -y install \ -netcat \ +netcat-openbsd \ python \ unzip \ curl \ diff --git a/docker/dist/Dockerfile.dist.debian9 b/docker/dist/Dockerfile.dist.debian9 index 27fba8a..97953d9 100644 --- a/docker/dist/Dockerfile.dist.debian9 +++ b/docker/dist/Dockerfile.dist.debian9 @@ -18,7 +18,7 @@ FROM openjdk:8-jdk-slim RUN apt-get -y update && apt-get -y install \ -netcat \ +netcat-openbsd \ python \ unzip \ curl \
[incubator-heron] branch master updated: Improve docker dev env script (#3441)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 3b66e97 Improve docker dev env script (#3441)
3b66e97 is described below
commit 3b66e971c811b4258b3bd375ef5b0066857a373b
Author: Ning Wang
AuthorDate: Sun Feb 2 22:48:42 2020 -0800
Improve docker dev env script (#3441)
---
docker/scripts/{dev-env.sh => dev-env-create.sh} | 40 +++---
docker/scripts/dev-env-run.sh| 42
2 files changed, 62 insertions(+), 20 deletions(-)
diff --git a/docker/scripts/dev-env.sh b/docker/scripts/dev-env-create.sh
old mode 100755
new mode 100644
similarity index 74%
rename from docker/scripts/dev-env.sh
rename to docker/scripts/dev-env-create.sh
index 63eec60..0926cfa
--- a/docker/scripts/dev-env.sh
+++ b/docker/scripts/dev-env-create.sh
@@ -16,39 +16,38 @@
# specific language governing permissions and limitations
# under the License.
-# This is a script to start a docker container that has all
-# tools needed to compire Heron. Developer should be able to
-# compile Heron in the container without any setup works.
-# usage:
+# This is a script to create/start a docker container that has all
+# tools needed to build Heron. Developer should be able to
+# build Heron in the container without any other setup works.
+#
+# Usage:
# To create a clean development environment with docker and run it,
# execute the following scripts in the source directory of Heron:
-# sh docker/scripts/dev-env.sh
+# sh docker/scripts/dev-env-create.sh CONTAINER_NAME [OS]
#
-# After the container is started, build Heron with bazel
+# After the container is started, you can build Heron with bazel
# (ubuntu config is used in the example):
# ./bazel_configure.py
# bazel build --config=ubuntu heron/...
-#
-# To enter an existing container with a new shell, find the container
-# ID with this command first:
-# docker ps -a
-# The image name looks like: "heron-dev:ubuntu18.04".=
-# After the container is found, execute the following commands to start
-# the container in case it is not started yet, and then start a new
-# terminal in the container:
-# docker container start CONTAINER_ID
-# docker exec -it CONTAINER_ID bash
-#
+# bazel build --config=ubuntu scripts/packages:binpkgs
set -o nounset
set -o errexit
+case $# in
+ 0)
+echo "Missing arguments."
+echo "Usage: $0 [OS]"
+exit 1
+;;
+esac
+
# Default platform is ubuntu18.04. Other available platforms
# include centos7, debian9
-TARGET_PLATFORM=${1:-"ubuntu18.04"}
-SCRATCH_DIR=${2:-"$HOME/.heron-docker"}
+TARGET_PLATFORM=${2:-"ubuntu18.04"}
+SCRATCH_DIR="$HOME/.heron-docker"
REPOSITORY="heron-dev"
-
+CONTAINER_NAME=$1
realpath() {
echo "$(cd "$(dirname "$1")"; pwd)/$(basename "$1")"
@@ -83,6 +82,7 @@ docker build -t $REPOSITORY:$TARGET_PLATFORM -f $DOCKER_FILE
$SCRATCH_DIR
echo "Creating and starting container and mapping the current dir to /heron"
docker container run -it \
+--name $CONTAINER_NAME --rm \
-e TARGET_PLATFORM=$TARGET_PLATFORM \
-e SCRATCH_DIR="/scratch" \
-v $PROJECT_DIR:/heron \
diff --git a/docker/scripts/dev-env-run.sh b/docker/scripts/dev-env-run.sh
new file mode 100755
index 000..669d2d7
--- /dev/null
+++ b/docker/scripts/dev-env-run.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a script to start a new session of the docker container
+# created with the dev-env-create.sh script.
+# Usage:
+# sh docker/scripts/dev-env-run.sh CONTAINER_NAME
+#
+# In case you forgot the container name, you can use this command
+# to list all the existing containers (name is the last column).
+# docker container ls -a
+
+set -o nounset
+set -o errexit
+
+case $# in
+ 0)
+echo "Missing argument."
+echo "Usage: $0 "
+exit 1
+;;
+esac
+
+CONTAINER_NAME=$1
+
+docker container start $CONTAINER_NAME
+docker exec -it $CONTAINER_NAME bash
\ No newline at end of file
[incubator-heron] branch master updated (0599224 -> b964cf4)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 0599224 Update license files for JSXTransformer.js (#3437) add b964cf4 explicitly making the site indexable (#3442) No new revisions were added by this update. Summary of changes: .gitmodules| 4 website/public | 1 - website2/package-lock.json | 3 +++ website2/website/siteConfig.js | 51 +- 4 files changed, 8 insertions(+), 51 deletions(-) delete mode 16 website/public create mode 100644 website2/package-lock.json
[incubator-heron] branch master updated: Updated rules_scala to fix https issue (#3436)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new b16973a Updated rules_scala to fix https issue (#3436)
b16973a is described below
commit b16973a7741ac2780202e888b5ada994d5230552
Author: Nicholas Nezis
AuthorDate: Thu Jan 16 03:04:53 2020 -0500
Updated rules_scala to fix https issue (#3436)
---
WORKSPACE | 7 +++
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/WORKSPACE b/WORKSPACE
index e29118d..b8d7bc6 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -998,14 +998,13 @@ http_archive(
)
# scala integration
-rules_scala_version = "300b4369a0a56d9e590d9fea8a73c3913d758e12" # May 27 -
update this as needed
+rules_scala_version = "358ab829626c6c2d34ec27f856485d3121e299c7" # Jan 15
2020 - update this as needed
http_archive(
name = "io_bazel_rules_scala",
-urls = ["https://github.com/bazelbuild/rules_scala/archive/%s.zip; %
rules_scala_version],
-type = "zip",
strip_prefix = "rules_scala-%s" % rules_scala_version,
-sha256 =
"7f35ee7d96b22f6139b81da3a8ba5fb816e1803ed097f7295b85b7a56e4401c7",
+type = "zip",
+url = "https://github.com/bazelbuild/rules_scala/archive/%s.zip; %
rules_scala_version,
)
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
[incubator-heron] branch master updated: update readme (#3429)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 2f0c1db update readme (#3429) 2f0c1db is described below commit 2f0c1dbc457eb76fde678916db4310ad07927454 Author: Josh Fischer AuthorDate: Wed Jan 8 02:31:26 2020 -0600 update readme (#3429) * update readme * fix typo --- README.md | 23 --- 1 file changed, 4 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 09a6aa1..85a7b61 100644 --- a/README.md +++ b/README.md @@ -24,16 +24,14 @@ Heron is realtime analytics platform developed by Twitter. It has a wide array [Heron in Apache Incubation](http://incubator.apache.org/projects/heron.html) -[Heron Website](https://apache.github.io/incubator-heron) - ### Documentation -[https://apache.github.io/incubator-heron](https://apache.github.io/incubator-heron) +[https://heron.incubator.apache.org/](https://heron.incubator.apache.org/) Heron Requirements: * Java JDK 1.8 * Python 2.7 - * Bazel latest + * Bazel 0.26.0 ## Contact @@ -52,27 +50,14 @@ Heron is realtime analytics platform developed by Twitter. It has a wide array [Bay Area Heron Meetup](https://www.meetup.com/Apache-Heron-Bay-Area), *We meet on Third Monday of Every Month in Palo Alto.* - Twitter -Heron on Twitter: [@heronstreaming](https://twitter.com/heronstreaming) - ## For more information: -* Official Heron documentation located at [https://apache.github.io/incubator-heron](https://apache.github.io/incubator-heron) +* Official Heron documentation located at [https://heron.incubator.apache.org/](https://heron.incubator.apache.org/) * Official Heron resources, including Conference & Journal Papers, Videos, Blog Posts and selected Press located at [Heron Resources](https://apache.github.io/incubator-heron/docs/resources/) * [Twitter Heron: Stream Processing at Scale](http://dl.acm.org/citation.cfm?id=2742788) (academic paper) * [Twitter Heron: Stream Processing at Scale](https://www.youtube.com/watch?v=pUaFOuGgmco) (YouTube video) * [Flying Faster with Twitter Heron](https://blog.twitter.com/2015/flying-faster-with-twitter-heron) (blog post) - -## Update - -We recently merged updates to run Heron natively using [Mesos](http://mesos.apache.org/) in [AWS](https://aws.amazon.com/), Mesos/[Aurora](http://aurora.apache.org/) in AWS, and locally on a laptop. - -We also added beta testing for [Apache YARN](https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/YARN.html) support using [Apache REEF](http://reef.apache.org/), in addition to the Simple Linux Utility for Resource Management [(SLURM)](http://slurm.schedmd.com/). - -We are working to add support for [Mesosphere DC/OS](https://dcos.io/) and [Kubernetes](http://kubernetes.io/). We will continue to post updates as we progress. - - ## License -Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0 +Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0 \ No newline at end of file
[incubator-heron] branch master updated: Updates to make Zookeeper work (#3424)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new a745e3e Updates to make Zookeeper work (#3424) a745e3e is described below commit a745e3ef9d6668cdd4d5efb3e2a257d5679c8597 Author: Nicholas Nezis AuthorDate: Sat Dec 21 02:38:44 2019 -0500 Updates to make Zookeeper work (#3424) Netcat was not included in the centos image. Also the zookeeper scripts were not executable. --- docker/dist/Dockerfile.dist.centos7 | 1 + docker/dist/scripts/generate-zookeeper-config.sh | 0 docker/dist/scripts/install-zookeeper.sh | 0 docker/dist/scripts/start-zookeeper.sh | 0 docker/dist/scripts/wait-for-zookeeper.sh| 0 5 files changed, 1 insertion(+) diff --git a/docker/dist/Dockerfile.dist.centos7 b/docker/dist/Dockerfile.dist.centos7 index 88b856c..d19acc7 100644 --- a/docker/dist/Dockerfile.dist.centos7 +++ b/docker/dist/Dockerfile.dist.centos7 @@ -22,6 +22,7 @@ RUN yum -y install python; yum clean all RUN yum -y install unzip; yum clean all RUN yum -y install which; yum clean all RUN yum -y install curl; yum clean all +RUN yum -y install nmap-ncat; yum clean all RUN yum -y install python-setuptools; yum clean all RUN easy_install supervisor RUN echo_supervisord_conf > /etc/supervisord.conf diff --git a/docker/dist/scripts/generate-zookeeper-config.sh b/docker/dist/scripts/generate-zookeeper-config.sh old mode 100644 new mode 100755 diff --git a/docker/dist/scripts/install-zookeeper.sh b/docker/dist/scripts/install-zookeeper.sh old mode 100644 new mode 100755 diff --git a/docker/dist/scripts/start-zookeeper.sh b/docker/dist/scripts/start-zookeeper.sh old mode 100644 new mode 100755 diff --git a/docker/dist/scripts/wait-for-zookeeper.sh b/docker/dist/scripts/wait-for-zookeeper.sh old mode 100644 new mode 100755
[incubator-heron] branch master updated (367b44a -> cb8212f)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 367b44a Removing old com.twitter namespace references (#3415) add cb8212f update bazel_config for Mac OS 10.14 (#3420) No new revisions were added by this update. Summary of changes:
[incubator-heron] branch master updated (73f2487 -> 367b44a)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 73f2487 Joshfischer/site build mods (#3422) add 367b44a Removing old com.twitter namespace references (#3415) No new revisions were added by this update. Summary of changes: deploy/kubernetes/helm/templates/tools.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[incubator-heron] branch master updated (a565a6b -> 73f2487)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from a565a6b Updated compile docker images to include `tree` package (#3423) add 73f2487 Joshfischer/site build mods (#3422) No new revisions were added by this update. Summary of changes: .asf.yaml| 6 ++ website2/website/scripts/publish_site.sh | 3 +++ 2 files changed, 9 insertions(+)
[incubator-heron] branch master updated (7ef2e15 -> a565a6b)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 7ef2e15 Fix Helm values.yml typo (#3421) add a565a6b Updated compile docker images to include `tree` package (#3423) No new revisions were added by this update. Summary of changes: docker/compile/Dockerfile.centos7 | 1 + docker/compile/Dockerfile.debian9 | 1 + docker/compile/Dockerfile.ubuntu14.04 | 1 + docker/compile/Dockerfile.ubuntu16.04 | 1 + docker/compile/Dockerfile.ubuntu18.04 | 1 + 5 files changed, 5 insertions(+)
[incubator-heron] branch master updated: Update zk version to 3.4.14 in dist and install zk in ubuntu 16/18 di… (#3411)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 340e654 Update zk version to 3.4.14 in dist and install zk in ubuntu 16/18 di… (#3411) 340e654 is described below commit 340e6546f5dd3c7eb557ffc850736112ec782cf3 Author: Ning Wang AuthorDate: Mon Dec 9 23:55:38 2019 -0800 Update zk version to 3.4.14 in dist and install zk in ubuntu 16/18 di… (#3411) * Update zk version to 3.4.14 in dist and install zk in ubuntu 16/18 dist docker image * fix path * Add back docker ARG --- docker/base/Dockerfile.base.debian9 | 32 ++--- docker/base/scripts/install-zookeeper.sh | 60 docker/dist/Dockerfile.dist.centos7 | 9 + docker/dist/Dockerfile.dist.debian9 | 33 +++--- docker/dist/Dockerfile.dist.ubuntu14.04 | 36 +++ docker/dist/Dockerfile.dist.ubuntu16.04 | 11 +- docker/dist/Dockerfile.dist.ubuntu18.04 | 9 + docker/dist/scripts/install-zookeeper.sh | 60 8 files changed, 159 insertions(+), 91 deletions(-) diff --git a/docker/base/Dockerfile.base.debian9 b/docker/base/Dockerfile.base.debian9 index d45d592..ebbccf4 100644 --- a/docker/base/Dockerfile.base.debian9 +++ b/docker/base/Dockerfile.base.debian9 @@ -27,36 +27,10 @@ RUN apt-get -y update && apt-get -y install \ rm -rf /var/lib/apt/lists/* # install zookeeper -ARG ZK_DIST=zookeeper-3.4.10 +ARG ZK_DIST=zookeeper-3.4.14 +ADD base/scripts /opt/heron-docker/scripts -RUN curl -O "https://archive.apache.org/dist/zookeeper/$ZK_DIST/$ZK_DIST.tar.gz; \ -&& tar -xzf $ZK_DIST.tar.gz -C /opt \ -&& rm -r $ZK_DIST.tar.gz \ -&& mv /opt/$ZK_DIST /opt/zookeeper \ -&& rm -rf /opt/zookeeper/CHANGES.txt \ -/opt/zookeeper/README.txt \ -/opt/zookeeper/NOTICE.txt \ -/opt/zookeeper/CHANGES.txt \ -/opt/zookeeper/README_packaging.txt \ -/opt/zookeeper/build.xml \ -/opt/zookeeper/config \ -/opt/zookeeper/contrib \ -/opt/zookeeper/dist-maven \ -/opt/zookeeper/docs \ -/opt/zookeeper/ivy.xml \ -/opt/zookeeper/ivysettings.xml \ -/opt/zookeeper/recipes \ -/opt/zookeeper/src \ -/opt/zookeeper/$ZK_DIST.jar.asc \ -/opt/zookeeper/$ZK_DIST.jar.md5 \ -/opt/zookeeper/$ZK_DIST.jar.sha1 +RUN sh /opt/heron-docker/scripts/install-zookeeper.sh $ZK_DIST ADD base/conf/zookeeper.conf /opt/zookeeper/conf/zookeeper.conf ADD base/conf/sandbox.conf /etc/supervisor/conf.d/ - -RUN mkdir -p /opt/zookeeper/scripts -ADD base/scripts /opt/zookeeper/scripts -RUN chmod +x /opt/zookeeper/scripts/generate-zookeeper-config.sh && \ -chmod +x /opt/zookeeper/scripts/zookeeper-ruok.sh && \ -chmod +x /opt/zookeeper/scripts/start-zookeeper.sh && \ -chmod +x /opt/zookeeper/scripts/wait-for-zookeeper.sh diff --git a/docker/base/scripts/install-zookeeper.sh b/docker/base/scripts/install-zookeeper.sh new file mode 100644 index 000..8c7be96 --- /dev/null +++ b/docker/base/scripts/install-zookeeper.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +if [ "$#" -ne 1 ]; then + echo "Usage: $0 zookeeper-" + echo "Example:" + echo "$ $0 zookeeper-3.4.14" + exit 1 +fi + +ZK_DIST=$1 + +curl -O "https://archive.apache.org/dist/zookeeper/$ZK_DIST/$ZK_DIST.tar.gz; +tar -xzf ./$ZK_DIST.tar.gz -C /opt +rm ./$ZK_DIST.tar.gz + +mv /opt/$ZK_DIST /opt/zookeeper +rm -rf /opt/zookeeper/CHANGES.txt \ +/opt/zookeeper/README.txt \ +/opt/zookeeper/NOTICE.txt \ +/opt/zookeeper/CHANGES.txt \ +/opt/zookeeper/README_packaging.txt \ +/opt/zookeeper/build.xml \ +/opt/zookeeper/config \ +/opt/zookeeper/contrib \ +/opt/zookeeper/dist-maven \ +/opt/zookeeper/docs \ +/opt/zookeeper/ivy.xml \ +/opt/zookeeper/ivysettings.xml \ +/opt/zookeeper/recipes \ +/opt/zookeeper/src \ +/opt/zookeeper/$ZK_DIST.jar.as
[incubator-heron] branch master updated: Fix jQuery license and add missing license files (#3407)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new d8d811a Fix jQuery license and add missing license files (#3407) d8d811a is described below commit d8d811a910992f9a4f38535597b51fa7b1bc22c8 Author: Ning Wang AuthorDate: Fri Nov 29 23:55:19 2019 -0800 Fix jQuery license and add missing license files (#3407) --- LICENSE | 6 +- licenses/LICENSE-Modernizr.txt| 5 ++ licenses/LICENSE-autogen.sh.txt | 30 +++ LICENSE => licenses/LICENSE-bazel_authors.txt | 124 +- 4 files changed, 40 insertions(+), 125 deletions(-) diff --git a/LICENSE b/LICENSE index 3c18e55..5714bc7 100644 --- a/LICENSE +++ b/LICENSE @@ -222,9 +222,6 @@ See project link for details. -> heron/tools/ui/resources/static/js/bootstrap.js Bootstrap(v3.0.0, https://github.com/twbs/bootstrap/blob/v3.0.0/LICENSE) -> heron/tools/ui/resources/static/css/bootstrap.css -JQuery(https://jquery.org/license/) - -> heron/shell/assets/jquery.js - -> heron/tools/ui/resources/static/js/jquery.min.js JSXTransformer(v0.10.0) -> heron/tools/ui/resources/static/js/JSXTransformer.0.10.0.js React (v0.10.0, https://github.com/facebook/react/blob/0.10-stable/LICENSE) @@ -270,6 +267,9 @@ The following components are provided under the MIT License. See project link fo -> heron/tools/ui/resources/static/js/d3-tip.min.0.6.3.js html5shiv(v3.6.2pre, https://github.com/aFarkas/html5shiv/blob/master/MIT%20and%20GPL2%20licenses.md) -> heron/tools/ui/resources/static/js/html5-trunk.js +JQuery(v1.7.2, v1.8.3 https://jquery.org/license/) + -> heron/shell/assets/jquery.js + -> heron/tools/ui/resources/static/js/jquery.min.js jQuery UI(v1.8.23, https://github.com/jquery/jquery-ui/blob/master/LICENSE.txt) -> heron/tools/ui/resources/static/js/jquery-ui-1.8.23.custom.min.js list.js(v1.1.1, https://github.com/javve/list.js/blob/master/LICENSE) diff --git a/licenses/LICENSE-Modernizr.txt b/licenses/LICENSE-Modernizr.txt new file mode 100644 index 000..e662c78 --- /dev/null +++ b/licenses/LICENSE-Modernizr.txt @@ -0,0 +1,5 @@ +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/licenses/LICENSE-autogen.sh.txt b/licenses/LICENSE-autogen.sh.txt new file mode 100644 index 000..f869f6c --- /dev/null +++ b/licenses/LICENSE-autogen.sh.txt @@ -0,0 +1,30 @@ +Copyright (c) 2005-2009 United States Government as represented by +the U.S. Army Research Laboratory. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following +disclaimer in the documentation and/or other materials provided +with the distribution. + +3. The name of the author may not be used to endorse or promote +products derived from this software without specific prior written +permission. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS +OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, +WHET
[incubator-heron] branch master updated: BUG: bookie statefulset helm template specifies incorrect service name (#3410)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 4bca2e5 BUG: bookie statefulset helm template specifies incorrect
service name (#3410)
4bca2e5 is described below
commit 4bca2e55995b4a1a5c4550c854a4a09ceffd02fa
Author: Scott <47566431+the-scott-h...@users.noreply.github.com>
AuthorDate: Thu Nov 28 02:49:11 2019 -0500
BUG: bookie statefulset helm template specifies incorrect service name
(#3410)
* Update bookie template to use correct service name
* helm add bookie toleration for unready endpoints
---
deploy/kubernetes/helm/templates/bookie.yaml | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/deploy/kubernetes/helm/templates/bookie.yaml
b/deploy/kubernetes/helm/templates/bookie.yaml
index acb359e..7a76c56 100644
--- a/deploy/kubernetes/helm/templates/bookie.yaml
+++ b/deploy/kubernetes/helm/templates/bookie.yaml
@@ -71,7 +71,7 @@ metadata:
component: {{ .Release.Name }}-bookie
spec:
{{- if or (eq .Values.platform "gke") (eq .Values.platform "minikube") }}
- serviceName: {{ .Release.Name }}-bookkeeper
+ serviceName: {{ .Release.Name }}-bookie
replicas: {{ $bookieReplicas }}
{{- end }}
@@ -200,6 +200,10 @@ spec:
apiVersion: v1
kind: Service
metadata:
+ {{- if or (eq .Values.platform "gke") (eq .Values.platform "minikube") }}
+ annotations:
+service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
+ {{- end }}
name: {{ .Release.Name }}-bookie
labels:
app: {{ .Release.Name }}-bookkeeper
[incubator-heron] branch master updated: Add parallelism calculator in Heron UI (#3401)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new d8db1c5 Add parallelism calculator in Heron UI (#3401)
d8db1c5 is described below
commit d8db1c50b499c823e319fd16a4476a02f525a367
Author: Ning Wang
AuthorDate: Fri Nov 15 00:05:26 2019 -0800
Add parallelism calculator in Heron UI (#3401)
---
heron/tools/ui/resources/static/css/main.css | 8 +++-
heron/tools/ui/resources/static/js/topologies.js | 51 +---
heron/tools/ui/resources/templates/topology.html | 59
3 files changed, 110 insertions(+), 8 deletions(-)
diff --git a/heron/tools/ui/resources/static/css/main.css
b/heron/tools/ui/resources/static/css/main.css
index 0ae4425..f13666d 100644
--- a/heron/tools/ui/resources/static/css/main.css
+++ b/heron/tools/ui/resources/static/css/main.css
@@ -2291,4 +2291,10 @@ div#topologydetails.display-stats
div.display-info.display-stats,
div#topologydetails.display-counters div.display-info.display-counters,
div#topologydetails.display-config div.display-info.display-config {
display: block;
-}
\ No newline at end of file
+}
+
+/* Backdrop in bootstrap 3 seems to have solid color and we would like it to
be half transparent */
+div.modal-backdrop.fade.in {
+ opacity: 0.5;
+ filter: alpha(opacity=50);
+}
diff --git a/heron/tools/ui/resources/static/js/topologies.js
b/heron/tools/ui/resources/static/js/topologies.js
index 764e0aa..ff8dbbe 100644
--- a/heron/tools/ui/resources/static/js/topologies.js
+++ b/heron/tools/ui/resources/static/js/topologies.js
@@ -1067,8 +1067,34 @@ var BoltRunningInfo = React.createClass({
});
var setState = this.setState.bind(this);
-const headings = ["Bolt", "Parallelism", "Execute Count", "Failure Count",
- "Capacity Utilization(min)", "Capacity
Utilization(max)"];
+const headings = [
+ { name: "Bolt", sortable: true },
+ { name: "Parallelism", sortable: true },
+ { name: "Execute Count", sortable: true },
+ { name: "Failure Count", sortable: true },
+ { name: "Capacity Utilization(min)", sortable: true },
+ { name: "Capacity Utilization(max)", sortable: true },
+ { name: "Parallelism Calculator", sortable: false }
+];
+
+var openParallelismCalculator = function (e, index) {
+ var row = rows[index];
+
+ var calculator = $("#parallelism-calculator-modal");
+ calculator.find('#modal-component').text(row[0]);
+ calculator.attr('data-execute-count', row[2]);
+ calculator.find('#modal-execute-count').text(row[2]);
+ calculator.find('#target-execute-count').val(row[2]);
+ calculator.attr('data-max-utilization', row[5].replace("%", ""));
+ calculator.find('#modal-max-utilization').text(row[5]);
+ calculator.find('#target-max-utilization').val(row[5].replace("%", ""));
+ calculator.attr('data-parallelism', row[1]);
+ calculator.find('#modal-parallelism').text(row[1]);
+ calculator.find('#target-parallelism').val(row[1]);
+
+ calculator.modal({keyboard: true});
+};
+
return (
@@ -1100,16 +1126,27 @@ var BoltRunningInfo = React.createClass({
reverse: i === sortKey ? (!reverse) : true
});
}
-return {heading};
+
+if (heading.sortable) {
+ return {heading.name};
+} else {
+ return {heading.name};
+}
})}
-{rows.map(function (row) {
+{rows.map(function (row, index) {
return {
-row.map(function (value, i) {
- return {value};
-})};
+ row.map(function (value, i) {
+return {value};
+ })}
+
+ openParallelismCalculator(e,
index)}>
+ Calculator
+
+ ;
})}
diff --git a/heron/tools/ui/resources/templates/topology.html
b/heron/tools/ui/resources/templates/topology.html
index f743c78..c109625 100644
--- a/heron/tools/ui/resources/templates/topology.html
+++ b/heron/tools/ui/resources/templates/topology.html
@@ -182,7 +182,49 @@ under the License.
+
+
+
+
+
+ Parallelism
Calculator:
+
+
+
+
+
+
+Target execute count (current
[incubator-heron] branch master updated: Make topology details section scrollable in Heron UI (#3400)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new ae2d1f7 Make topology details section scrollable in Heron UI (#3400)
ae2d1f7 is described below
commit ae2d1f7487d9b27d9d29a90dd16c31609418732f
Author: Ning Wang
AuthorDate: Mon Nov 11 14:40:04 2019 -0800
Make topology details section scrollable in Heron UI (#3400)
---
heron/tools/ui/resources/static/css/main.css | 5 +++--
heron/tools/ui/resources/templates/topology.html | 16 ++--
2 files changed, 17 insertions(+), 4 deletions(-)
diff --git a/heron/tools/ui/resources/static/css/main.css
b/heron/tools/ui/resources/static/css/main.css
index 5cde820..0ae4425 100644
--- a/heron/tools/ui/resources/static/css/main.css
+++ b/heron/tools/ui/resources/static/css/main.css
@@ -583,8 +583,7 @@ header {
left: 0; }
.dashboard-wrapper {
- /*padding: 30px;*/
- padding: 0px 30px 30px 10px;
+ padding: 0px 10px 20px 10px;
position: relative;
margin: 0px 0px 10px 10px;
background: white;
@@ -2269,6 +2268,7 @@ div#display-navigator {
padding-bottom: 5px;
background-color: transparent;
border-width: 0px;
+ outline: none; /* disable outline after clicked on */
}
#display-navigator .navbar-default button.active {
@@ -2280,6 +2280,7 @@ div#topologydetails {
border-style: solid;
border-width: 1px;
border-color: #dd;
+ overflow-y: scroll;
}
div#topologydetails div.display-info {
diff --git a/heron/tools/ui/resources/templates/topology.html
b/heron/tools/ui/resources/templates/topology.html
index 49961a7..f743c78 100644
--- a/heron/tools/ui/resources/templates/topology.html
+++ b/heron/tools/ui/resources/templates/topology.html
@@ -103,7 +103,7 @@ under the License.
-
+
@@ -281,12 +281,23 @@ under the License.
}
);
+function resizeDetailsSection() {
+ const minSectionHeight = 350; // Minimal height in pixel.
+ var windowH = $(window).height();
+ var planH = $(".plans").height();
+ var detailsH = Math.max(minSectionHeight, windowH - planH - 222); //
Leave 222 pixels for topology info.
+
+ d3.selectAll("div#topologydetails").style('height', detailsH + 'px');
+}
+
function render(planController) {
drawLogicalPlan(planController, logicalPlan, "#logical-plan",
$("#logical-plan").width(), 400, "{{baseUrl}}",
"{{cluster}}", "{{environ}}", "{{topology}}");
- planController.planResized();
drawPhysicalPlan(planController, physicalPlan, packingPlan,
"#physical-plan", $("#physical-plan").width(), 400, "{{baseUrl}}",
"{{cluster}}",
"{{environ}}", "{{topology}}");
+
+ resizeDetailsSection();
+
planController.planResized();
}
@@ -297,6 +308,7 @@ under the License.
}, 100));
render(planController);
drawStatsTable(planController, "{{baseUrl}}", "{{cluster}}",
"{{environ}}", "{{topology}}", physicalPlan, logicalPlan);
+
planController.planDrawn();
}
};
[incubator-heron] branch master updated: Support Catalina for Mac (#3398)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 909c7a0 Support Catalina for Mac (#3398)
909c7a0 is described below
commit 909c7a005f43ca826e9e1209ecb3fa0ae88bf1a8
Author: se choi
AuthorDate: Mon Nov 11 15:59:40 2019 +0900
Support Catalina for Mac (#3398)
---
bazel_configure.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/bazel_configure.py b/bazel_configure.py
index ec254e5..02e67f1 100755
--- a/bazel_configure.py
+++ b/bazel_configure.py
@@ -191,7 +191,8 @@ def discover_version(path):
# on mac, /usr/bin/cpp --version returns this:
# Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
- mac_line = re.search('^(Apple LLVM version\s+[\d\.]+)\s+\(clang.*',
first_line)
+ # Apple clang version 11.0.0 (clang-1100.0.33.12)
+ mac_line = re.search('^(Apple (LLVM|clang) version\s+[\d\.]+)\s+\(clang.*',
first_line)
if mac_line:
version = get_trailing_version(mac_line.group(1))
if version:
[incubator-heron] branch java_11_support updated: Java 11 support (#3399)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch java_11_support
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/java_11_support by this push:
new 1294a04 Java 11 support (#3399)
1294a04 is described below
commit 1294a04ee3aafe26fcfe109e5d20537b417213b3
Author: se choi
AuthorDate: Mon Nov 11 15:57:57 2019 +0900
Java 11 support (#3399)
* Support Java 11
* config travis to use oracle jdk 11
* Add check jdk version
---
heron/executor/src/python/heron_executor.py | 117 ++--
1 file changed, 40 insertions(+), 77 deletions(-)
diff --git a/heron/executor/src/python/heron_executor.py
b/heron/executor/src/python/heron_executor.py
index 01f59a9..aac24c2 100755
--- a/heron/executor/src/python/heron_executor.py
+++ b/heron/executor/src/python/heron_executor.py
@@ -432,21 +432,7 @@ class HeronExecutor(object):
# for instance, the default -Xmx in Twitter mesos
machine is around 18GB
'-Xmx1024M',
'-XX:+PrintCommandLineFlags',
- '-verbosegc',
- '-XX:+PrintGCDetails',
- '-XX:+PrintGCTimeStamps',
- '-XX:+PrintGCDateStamps',
- '-XX:+PrintGCCause',
- '-XX:+UseGCLogFileRotation',
- '-XX:NumberOfGCLogFiles=5',
- '-XX:GCLogFileSize=100M',
- '-XX:+PrintPromotionFailure',
- '-XX:+PrintTenuringDistribution',
- '-XX:+PrintHeapAtGC',
- '-XX:+HeapDumpOnOutOfMemoryError',
- '-XX:+UseConcMarkSweepGC',
- '-XX:+PrintCommandLineFlags',
- '-Xloggc:log-files/gc.' + metricsManagerId + '.log',
+ self._get_java_gc_instance_cmd(metricsManagerId),
'-Djava.net.preferIPv4Stack=true',
'-cp',
self.metrics_manager_classpath,
@@ -473,21 +459,7 @@ class HeronExecutor(object):
# for instance, the default -Xmx in Twitter mesos
machine is around 18GB
'-Xmx1024M',
'-XX:+PrintCommandLineFlags',
- '-verbosegc',
- '-XX:+PrintGCDetails',
- '-XX:+PrintGCTimeStamps',
- '-XX:+PrintGCDateStamps',
- '-XX:+PrintGCCause',
- '-XX:+UseGCLogFileRotation',
- '-XX:NumberOfGCLogFiles=5',
- '-XX:GCLogFileSize=100M',
- '-XX:+PrintPromotionFailure',
- '-XX:+PrintTenuringDistribution',
- '-XX:+PrintHeapAtGC',
- '-XX:+HeapDumpOnOutOfMemoryError',
- '-XX:+UseConcMarkSweepGC',
- '-XX:+PrintCommandLineFlags',
- '-Xloggc:log-files/gc.metricscache.log',
+ self._get_java_gc_instance_cmd('metricscache'),
'-Djava.net.preferIPv4Stack=true',
'-cp',
self.metricscache_manager_classpath,
@@ -515,21 +487,7 @@ class HeronExecutor(object):
# for instance, the default -Xmx in Twitter mesos machine
is around 18GB
'-Xmx1024M',
'-XX:+PrintCommandLineFlags',
- '-verbosegc',
- '-XX:+PrintGCDetails',
- '-XX:+PrintGCTimeStamps',
- '-XX:+PrintGCDateStamps',
- '-XX:+PrintGCCause',
- '-XX:+UseGCLogFileRotation',
- '-XX:NumberOfGCLogFiles=5',
- '-XX:GCLogFileSize=100M',
- '-XX:+PrintPromotionFailure',
- '-XX:+PrintTenuringDistribution',
- '-XX:+PrintHeapAtGC',
- '-XX:+HeapDumpOnOutOfMemoryError',
- '-XX:+UseConcMarkSweepGC',
- '-XX:+PrintCommandLineFlags',
- '-Xloggc:log-files/gc.healthmgr.log',
+ self._get_java_gc_instance_cmd('healthmgr'),
'-Djava.net.preferIPv4Stack=true',
'-cp', self.health_manager_classpath,
healthmgr_main_class,
@@ -619,6 +577,41 @@ class HeronExecutor(object):
def _get_jvm_instance_cmd(self):
return Command(os.path.join(self.heron_java_home, 'bin/java'),
self.shell_env)
+ def _get_java_major_version(self):
+return int(self._get_jvm_version().split(".")[0])
+
+ def _get_java_gc_in
[incubator-heron] branch java_11_support updated (4ca4810 -> 0236e4b)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a change to branch java_11_support
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git.
omit 4ca4810 config travis to use oracle jdk 11
omit b52d478 Support Java 11
add 073e4d4 updating the NOTICE (#3374)
add 31b6667 Fix license in health handler py (#3376)
add a383bc8 Clean up license for healthHandler and jarjar help.txt (#3377)
add 066dd8e Add instance resource usage in container (#3375)
add 7987f4a Avoid duplicated metrics requests in Heron UI (#3378)
add c27d2c0 Fix max capacity timeline in Heron UI (#3383)
add c51faf8 makemetricstriggerperiodic (#3384)
add a65d330 fix base url and counters flag; combine metrics requests
(#3385)
add 1dbf1ce Refactor counters sections in Heron UI topology page (#3392)
add 8045a30 Docker site (#3393)
add 135fb76 Helm chart Bookkeeper fix (#3381)
add 1f5f23d Add tables for component running info (#3394)
add bd06dc8 Update the instruction for using the docker based development
environment (#3395)
add 06a55e9 Re-organize UI elements in Heorn UI topology page (#3397)
add 88a3ccd Support Java 11
add 0236e4b config travis to use oracle jdk 11
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4ca4810)
\
N -- N -- N refs/heads/java_11_support (0236e4b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
LICENSE| 5 +-
NOTICE | 2 +-
deploy/kubernetes/helm/templates/bookie.yaml | 3 +
docker/scripts/dev-env.sh | 9 +-
.../heron/healthmgr/HealthManagerMetrics.java | 11 +-
heron/shell/src/python/handlers/healthhandler.py | 23 +-
heron/tools/common/src/python/access/heron_api.py | 9 +-
heron/tools/ui/resources/static/css/main.css | 38 +
heron/tools/ui/resources/static/js/config.js | 2 +-
.../tools/ui/resources/static/js/physical-plan.js | 47 +-
.../ui/resources/static/js/plan-controller.js | 1 +
heron/tools/ui/resources/static/js/topologies.js | 790 +
.../static/js/{colors.js => topology-details.js} | 29 +-
heron/tools/ui/resources/templates/topology.html | 252 ---
...azel_rules_pex-0.3.0.txt => LICENSE-jarjar.txt} | 2 +-
website2/website/package-lock.json | 7 +-
website2/website/package.json | 4 +-
.../website/scripts}/Dockerfile.ubuntu18.04| 16 +-
website2/website/scripts/build-site.sh | 13 +-
.../website}/scripts/compile-docker.sh | 39 +-
website2/website/scripts/python-doc-gen.sh | 16 +-
21 files changed, 964 insertions(+), 354 deletions(-)
copy heron/tools/ui/resources/static/js/{colors.js => topology-details.js}
(53%)
copy licenses/{LICENSE-bazel_rules_pex-0.3.0.txt => LICENSE-jarjar.txt} (99%)
copy {docker/compile => website2/website/scripts}/Dockerfile.ubuntu18.04 (83%)
copy {docker => website2/website}/scripts/compile-docker.sh (61%)
[incubator-heron] branch nwangtw-patch-1 updated (13a59bc -> 09fd23e)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch nwangtw-patch-1 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 13a59bc Update issue templates add 09fd23e Update bug_report.md No new revisions were added by this update. Summary of changes: .github/ISSUE_TEMPLATE/bug_report.md | 9 + 1 file changed, 1 insertion(+), 8 deletions(-)
[incubator-heron] branch master updated: Re-organize UI elements in Heorn UI topology page (#3397)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 06a55e9 Re-organize UI elements in Heorn UI topology page (#3397)
06a55e9 is described below
commit 06a55e9de1b3524df406f8994e5e3b7f4b5f85d8
Author: Ning Wang
AuthorDate: Sat Nov 9 19:36:14 2019 -0800
Re-organize UI elements in Heorn UI topology page (#3397)
---
heron/tools/ui/resources/static/css/main.css | 38
heron/tools/ui/resources/static/js/config.js | 2 +-
heron/tools/ui/resources/static/js/topologies.js | 2 +-
.../ui/resources/static/js/topology-details.js | 41
heron/tools/ui/resources/templates/topology.html | 252 -
5 files changed, 223 insertions(+), 112 deletions(-)
diff --git a/heron/tools/ui/resources/static/css/main.css
b/heron/tools/ui/resources/static/css/main.css
index d8c6570..5cde820 100644
--- a/heron/tools/ui/resources/static/css/main.css
+++ b/heron/tools/ui/resources/static/css/main.css
@@ -2253,3 +2253,41 @@ nav {
.navbar-custom .navbar-link:hover {
color: #ffecb3;
}
+
+/* Topology details */
+div#display-navigator {
+ border-style: solid;
+ border-width: 1px;
+ border-color: #dd;
+ border-bottom-width: 0px;
+}
+
+#display-navigator .navbar-default button {
+ padding-left: 15px;
+ padding-right: 15px;
+ padding-top: 5px;
+ padding-bottom: 5px;
+ background-color: transparent;
+ border-width: 0px;
+}
+
+#display-navigator .navbar-default button.active {
+ background-color: lightskyblue;
+}
+
+/* Select the right topology details to show */
+div#topologydetails {
+ border-style: solid;
+ border-width: 1px;
+ border-color: #dd;
+}
+
+div#topologydetails div.display-info {
+ display: none;
+}
+
+div#topologydetails.display-stats div.display-info.display-stats,
+div#topologydetails.display-counters div.display-info.display-counters,
+div#topologydetails.display-config div.display-info.display-config {
+ display: block;
+}
\ No newline at end of file
diff --git a/heron/tools/ui/resources/static/js/config.js
b/heron/tools/ui/resources/static/js/config.js
index 762fe2b..4e23f2d 100644
--- a/heron/tools/ui/resources/static/js/config.js
+++ b/heron/tools/ui/resources/static/js/config.js
@@ -166,7 +166,7 @@ var ConfigTable = React.createClass({
tableLayout: 'inherit'
};
return (
-
+
{title}
diff --git a/heron/tools/ui/resources/static/js/topologies.js
b/heron/tools/ui/resources/static/js/topologies.js
index 1d1a0f9..764e0aa 100644
--- a/heron/tools/ui/resources/static/js/topologies.js
+++ b/heron/tools/ui/resources/static/js/topologies.js
@@ -598,7 +598,7 @@ var AllMetrics = React.createClass({
instance: this.props.instance,
};
return (
-
+
diff --git a/heron/tools/ui/resources/static/js/topology-details.js
b/heron/tools/ui/resources/static/js/topology-details.js
new file mode 100644
index 000..8add7da
--- /dev/null
+++ b/heron/tools/ui/resources/static/js/topology-details.js
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * Navigator of the topology details section in the topology page.
+ * When a tab is clicked, the corresponding "display-" class
+ * is set to the "topologydetails" element. Then CSS shows/hides
+ * the information.
+ */
+(function () {
+ var selected = "stats"; // Show stats tab at beginning.
+
+ function navigate () {
+this.parentElement.selected=this.id;
+for (var i = 0; i < this.parentElement.children.length; ++i) {
+ this.parentElement.children[i].className = ''; // Hide all
+}
+this.className = 'active'; // Show "this" element.
+
+// Show the right div by setting the class of the outter div.
+d3.selectAll('div#topologydetails').attr('class', 'display-' + this.id);
+ }
+
+ d3.selectAll('.navigator button').on('click', navigate);
+}());
diff --git a/heron/tools/ui/resources/templates/topology.html
b
[incubator-heron] branch master updated: Update the instruction for using the docker based development environment (#3395)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new bd06dc8 Update the instruction for using the docker based development environment (#3395) bd06dc8 is described below commit bd06dc8ad65b9f2bc5a1a9c1d4fca844a03ba941 Author: Ning Wang AuthorDate: Thu Nov 7 18:07:43 2019 -0800 Update the instruction for using the docker based development environment (#3395) --- docker/scripts/dev-env.sh | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docker/scripts/dev-env.sh b/docker/scripts/dev-env.sh index 17e37ed..63eec60 100755 --- a/docker/scripts/dev-env.sh +++ b/docker/scripts/dev-env.sh @@ -30,10 +30,15 @@ # bazel build --config=ubuntu heron/... # # To enter an existing container with a new shell, find the container -# id with this command: +# ID with this command first: # docker ps -a -# And then : +# The image name looks like: "heron-dev:ubuntu18.04".= +# After the container is found, execute the following commands to start +# the container in case it is not started yet, and then start a new +# terminal in the container: +# docker container start CONTAINER_ID # docker exec -it CONTAINER_ID bash +# set -o nounset set -o errexit
[incubator-heron] branch master updated: Add tables for component running info (#3394)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 1f5f23d Add tables for component running info (#3394)
1f5f23d is described below
commit 1f5f23d6fc1cada9d6c48211061d8af1ef302cb2
Author: Ning Wang
AuthorDate: Thu Nov 7 11:06:26 2019 -0800
Add tables for component running info (#3394)
* Add tables for component running info
* format capacity utilization
---
heron/tools/ui/resources/static/js/topologies.js | 436 ++-
1 file changed, 426 insertions(+), 10 deletions(-)
diff --git a/heron/tools/ui/resources/static/js/topologies.js
b/heron/tools/ui/resources/static/js/topologies.js
index 9a423e9..1d1a0f9 100644
--- a/heron/tools/ui/resources/static/js/topologies.js
+++ b/heron/tools/ui/resources/static/js/topologies.js
@@ -29,6 +29,8 @@ setInterval(function() {
countersUrlFlags = {};
}, 6);
+const ParallelismConfig = "topology.component.parallelism";
+
var AllExceptions = React.createClass({
getInitialState: function() {
return {}
@@ -269,7 +271,7 @@ var AllMetrics = React.createClass({
fetchLplan: function () {
if (this.state.lplan === undefined) {
- var fetch_url = this.props.baseUrl +
+ const fetch_url = this.props.baseUrl +
"/topologies/" +
this.props.cluster + "/" +
this.props.environ + "/" +
@@ -392,7 +394,7 @@ var AllMetrics = React.createClass({
fetchCounters: function() {
if (this.props.hasOwnProperty("comp_name")) {
- var fetch_url = this.props.baseUrl + "/topologies/metrics?" +
+ const fetch_url = this.props.baseUrl + "/topologies/metrics?" +
"cluster=" + this.props.cluster + "&" +
"environ=" + this.props.environ + "&" +
"topology=" + this.props.topology + "&" +
@@ -450,16 +452,19 @@ var AllMetrics = React.createClass({
// Fetch all metrics for all the spouts.
// Must have pplan and metricnames to continue.
if (this.state.pplan && this.state.lplan) {
+const fetch_url = this.props.baseUrl + "/topologies/metrics?" +
+ "cluster=" + this.props.cluster + "&" +
+ "environ=" + this.props.environ + "&" +
+ "topology=" + this.props.topology;
+
var spoutNames = [];
for (var name in this.state.lplan.spouts) {
if (this.state.lplan.spouts.hasOwnProperty(name)) {
spoutNames.push(name);
}
}
-var fetch_url = this.props.baseUrl + "/topologies/metrics?" +
- "cluster=" + this.props.cluster + "&" +
- "environ=" + this.props.environ + "&" +
- "topology=" + this.props.topology;
+
+// Load metrics for spouts.
for (var i = 0; i < spoutNames.length; i++) {
var spout = spoutNames[i];
for (var j = 0; j < this.state.lplan.spouts[spout].outputs.length;
j++) {
@@ -481,6 +486,35 @@ var AllMetrics = React.createClass({
}
}
}
+// Load metrics for bolts.
+var boltNames = [];
+for (var name in this.state.lplan.bolts) {
+ if (this.state.lplan.bolts.hasOwnProperty(name)) {
+boltNames.push(name);
+ }
+}
+
+for (var i = 0; i < boltNames.length; i++) {
+ var bolt = boltNames[i];
+ for (var j = 0; j < this.state.lplan.bolts[bolt].outputs.length;
j++) {
+var streamName =
this.state.lplan.bolts[bolt].outputs[j].stream_name;
+for (var timeRange in this.timeRanges) {
+ metricnameargs = "";
+ for (var boltMetric in this.boltMetricsInput) {
+if (this.boltMetricsInput.hasOwnProperty(boltMetric)) {
+ metricnameargs += "=" + boltMetric + streamName;
+}
+ }
+ if (this.timeRanges.hasOwnProperty(timeRange) && metricnameargs
!= "") {
+var url = fetch_url + metricnameargs +
+ "=" + bolt +
+ "=" + this.timeRanges[timeRange];
+
+this.fetchCountersURL(url, timeRange);
+ }
+}
+ }
+}
}
}
},
@@ -566,6 +600,8 @@ var AllMetrics = React.createClass({
return (
+
+
@@ -573,6 +609,7 @@ var AllMetrics = React.createClass({
}
});
+
var TopologyCounters = React.createClass({
capitalize: function(astr) {
if (astr) {
@@ -5
[incubator-heron] branch master updated: Helm chart Bookkeeper fix (#3381)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 135fb76 Helm chart Bookkeeper fix (#3381)
135fb76 is described below
commit 135fb760b00739dbb0f655fb91937a4edd0cb8a0
Author: Nicholas Nezis
AuthorDate: Thu Nov 7 03:11:16 2019 -0500
Helm chart Bookkeeper fix (#3381)
* Kubernetes Statefulset bug fix
* Update bookie.yaml
Updated to only use hostname with Statefulset
---
deploy/kubernetes/helm/templates/bookie.yaml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/deploy/kubernetes/helm/templates/bookie.yaml
b/deploy/kubernetes/helm/templates/bookie.yaml
index fecc7f1..acb359e 100644
--- a/deploy/kubernetes/helm/templates/bookie.yaml
+++ b/deploy/kubernetes/helm/templates/bookie.yaml
@@ -41,6 +41,9 @@ data:
BK_indexDirectories: "/bookkeeper/data/ledgers"
BK_zkServers: {{ .Release.Name }}-zookeeper:{{ .Values.zookeeper.clientPort
}}
BK_autoRecoveryDaemonEnabled: "true"
+ {{- if or (eq .Values.platform "gke") (eq .Values.platform "minikube") }}
+ BK_useHostNameAsBookieID: "true"
+ {{- end }}
# TODO: Issue 458: https://github.com/apache/bookkeeper/issues/458
{{- if eq .Values.bookkeeper.prometheus.enabled true }}
BK_enableStatistics: "true"
[incubator-heron] branch java_11_support created (now 4ca4810)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch java_11_support in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. at 4ca4810 config travis to use oracle jdk 11 No new revisions were added by this update.
[incubator-heron] branch master updated: Refactor counters sections in Heron UI topology page (#3392)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 1dbf1ce Refactor counters sections in Heron UI topology page (#3392)
1dbf1ce is described below
commit 1dbf1ce1860725827004c7047bf714b87c9df6d7
Author: Ning Wang
AuthorDate: Thu Oct 31 09:07:56 2019 -0700
Refactor counters sections in Heron UI topology page (#3392)
---
heron/tools/ui/resources/static/js/topologies.js | 220 +++
1 file changed, 142 insertions(+), 78 deletions(-)
diff --git a/heron/tools/ui/resources/static/js/topologies.js
b/heron/tools/ui/resources/static/js/topologies.js
index 0f25192..9a423e9 100644
--- a/heron/tools/ui/resources/static/js/topologies.js
+++ b/heron/tools/ui/resources/static/js/topologies.js
@@ -565,6 +565,7 @@ var AllMetrics = React.createClass({
};
return (
+
@@ -572,7 +573,7 @@ var AllMetrics = React.createClass({
}
});
-var ComponentCounters = React.createClass({
+var TopologyCounters = React.createClass({
capitalize: function(astr) {
if (astr) {
return astr.charAt(0).toUpperCase() + astr.slice(1);
@@ -581,73 +582,7 @@ var ComponentCounters = React.createClass({
},
getTitle: function () {
-if (this.props.info.comp_name) {
- var comp_type = this.capitalize(this.props.info.comp_type);
- var title = " " + comp_type + " Counters - " + this.props.info.comp_name;
- if (this.props.info.comp_spout_type !== undefined
- && this.props.info.comp_spout_type !== "default") {
-title += " - " + this.props.info.comp_spout_type + "/" +
this.props.info.comp_spout_source;
- }
- return title;
-}
-return " Topology Counters";
- },
-
- getComponentMetricsRows: function () {
-var metrics = {};
-if (this.props.info.metrics.hasOwnProperty(this.props.info.comp_name)) {
- metrics = this.props.info.metrics[this.props.info.comp_name];
-}
-var aggregatedMetrics = {};
-var timeRanges = ["10 mins", "1 hr", "3 hrs", "All Time"];
-
-for (var time in metrics) {
- if (metrics.hasOwnProperty(time)) {
-for (var metricname in metrics[time]) {
- if (metrics[time].hasOwnProperty(metricname) &&
- metricname !== "__interval" &&
- metricname !== "Uptime (ddd:hh:mm:ss)") {
-for (var stream in metrics[time][metricname]) {
- if (metrics[time][metricname].hasOwnProperty(stream)) {
-displayName = metricname + " (" + stream + ")";
-var value = 0;
-var allMetrics = metrics[time][metricname][stream].metrics;
-var aggregationType =
metrics[time][metricname][stream].aggregationType;
-var count = 0;
-for (var m in allMetrics) {
- if (allMetrics.hasOwnProperty(m)) {
-value += Number(allMetrics[m]);
-count++;
- }
-}
-if (aggregationType === AVG && count > 0) {
- value /= count;
-}
-if (!aggregatedMetrics.hasOwnProperty(displayName)) {
- aggregatedMetrics[displayName] = {};
-}
-value /= (metrics[time][metricname][stream].scaleDevisor || 1);
-aggregatedMetrics[displayName][time] = value;
- }
-}
- }
-}
- }
-}
-
-var rows = [];
-for (var name in aggregatedMetrics) {
- if (aggregatedMetrics.hasOwnProperty(name)) {
-var row = [];
-row.push(name);
-for (var i = 0; i < timeRanges.length; i++) {
- var time = timeRanges[i];
- row.push(Number(Number(aggregatedMetrics[name][time] ||
0).toFixed(2)));
-}
-rows.push(row);
- }
-}
-return rows;
+return "Topology Counters";
},
getTopologyMetricsRows: function () {
@@ -718,26 +653,154 @@ var ComponentCounters = React.createClass({
},
render: function () {
+if (this.props.info.comp_name) {
+ // A component is selected, return empty div.
+ return ()
+}
var title = this.getTitle();
+var rows = this.getTopologyMetricsRows();
var timeRanges = ["10 mins", "1 hr", "3 hrs", "All Time"];
var headings = ["Metrics"];
headings.push.apply(headings, timeRanges);
-var rows = [];
-var extraLinks = [];
+return (
+
+
+
+{title}
+
+
+
+
+
[incubator-heron] branch master updated: fix base url and counters flag; combine metrics requests (#3385)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new a65d330 fix base url and counters flag; combine metrics requests
(#3385)
a65d330 is described below
commit a65d3309b30f0468ef47e3a73f4ad334cee73e86
Author: Ning Wang
AuthorDate: Wed Oct 30 16:02:39 2019 -0700
fix base url and counters flag; combine metrics requests (#3385)
---
.../ui/resources/static/js/plan-controller.js | 1 +
heron/tools/ui/resources/static/js/topologies.js | 50 --
2 files changed, 28 insertions(+), 23 deletions(-)
diff --git a/heron/tools/ui/resources/static/js/plan-controller.js
b/heron/tools/ui/resources/static/js/plan-controller.js
index 5482e43..cbb7e13 100644
--- a/heron/tools/ui/resources/static/js/plan-controller.js
+++ b/heron/tools/ui/resources/static/js/plan-controller.js
@@ -110,6 +110,7 @@ function PlanController(baseUrl, cluster, environ,
toponame, physicalPlan, logic
function clearFocus() {
React.renderComponent(
AllMetrics({
+baseUrl: baseUrl,
cluster: cluster,
environ: environ,
topology: toponame
diff --git a/heron/tools/ui/resources/static/js/topologies.js
b/heron/tools/ui/resources/static/js/topologies.js
index 73860d1..0f25192 100644
--- a/heron/tools/ui/resources/static/js/topologies.js
+++ b/heron/tools/ui/resources/static/js/topologies.js
@@ -24,6 +24,10 @@ var SUM = 0,
LAST = 2;
var countersUrlFlags = {};
+setInterval(function() {
+ // Reset fetch flags every minute so that new fetches can go through
+ countersUrlFlags = {};
+}, 6);
var AllExceptions = React.createClass({
getInitialState: function() {
@@ -387,9 +391,8 @@ var AllMetrics = React.createClass({
},
fetchCounters: function() {
-var fetch_url = "";
if (this.props.hasOwnProperty("comp_name")) {
- fetch_url = this.props.baseUrl + "/topologies/metrics?" +
+ var fetch_url = this.props.baseUrl + "/topologies/metrics?" +
"cluster=" + this.props.cluster + "&" +
"environ=" + this.props.environ + "&" +
"topology=" + this.props.topology + "&" +
@@ -432,14 +435,15 @@ var AllMetrics = React.createClass({
}
}
}
- for (var i = 0; i < metricnames.length; i++) {
-for (var timeRange in this.timeRanges) {
- if (this.timeRanges.hasOwnProperty(timeRange)) {
-var url = fetch_url +
- "=" + metricnames[i] +
- "=" + this.timeRanges[timeRange];
-this.fetchCountersURL(url, timeRange);
- }
+ for (var timeRange in this.timeRanges) {
+metricnameargs = "";
+for (var i = 0; i < metricnames.length; i++) {
+ metricnameargs += "=" + metricnames[i];
+}
+if (this.timeRanges.hasOwnProperty(timeRange) && metricnameargs != "")
{
+ var url = fetch_url + metricnameargs +
+"=" + this.timeRanges[timeRange];
+ this.fetchCountersURL(url, timeRange);
}
}
} else if (!this.props.hasOwnProperty("comp_name")) {
@@ -452,28 +456,28 @@ var AllMetrics = React.createClass({
spoutNames.push(name);
}
}
-fetch_url = this.props.baseUrl + "/topologies/metrics?" +
+var fetch_url = this.props.baseUrl + "/topologies/metrics?" +
"cluster=" + this.props.cluster + "&" +
"environ=" + this.props.environ + "&" +
"topology=" + this.props.topology;
for (var i = 0; i < spoutNames.length; i++) {
var spout = spoutNames[i];
- var url = fetch_url + "=" + spout;
for (var j = 0; j < this.state.lplan.spouts[spout].outputs.length;
j++) {
var streamName =
this.state.lplan.spouts[spout].outputs[j].stream_name;
-for (var spoutMetric in this.spoutMetrics) {
- if (this.spoutMetrics.hasOwnProperty(spoutMetric)) {
-var metricname = spoutMetric + streamName;
-for (var timeRange in this.timeRanges) {
- if (this.timeRanges.hasOwnProperty(timeRange)) {
-var url = fetch_url +
- "=" + spout +
- "=" + metricname +
- "=" + this.timeRanges[timeRange];
-this.fetchCountersURL(url, timeRange);
- }
+for (var timeRange in this.timeRanges) {
+ metricnameargs = "";
+ for (var spoutM
[incubator-heron] branch master updated: Fix max capacity timeline in Heron UI (#3383)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new c27d2c0 Fix max capacity timeline in Heron UI (#3383)
c27d2c0 is described below
commit c27d2c0046996bb2f260f83952ad6483b52f60b8
Author: Ning Wang
AuthorDate: Mon Oct 28 23:53:12 2019 -0700
Fix max capacity timeline in Heron UI (#3383)
---
heron/tools/common/src/python/access/heron_api.py | 9 ++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/heron/tools/common/src/python/access/heron_api.py
b/heron/tools/common/src/python/access/heron_api.py
index 709acb4..ebf908d 100644
--- a/heron/tools/common/src/python/access/heron_api.py
+++ b/heron/tools/common/src/python/access/heron_api.py
@@ -867,9 +867,12 @@ class HeronQueryHandler(QueryHandler):
:param multi_ts:
:return:
'''
-if len(multi_ts) > 0 and len(multi_ts[0]["timeline"]) > 0:
- keys = multi_ts[0]["timeline"][0]["data"].keys()
- timelines = ([res["timeline"][0]["data"][key] for key in keys] for res
in multi_ts)
+# Some components don't have specific metrics such as capacity hence the
+# key set is empty. These components are filtered out first.
+filtered_ts = [ts for ts in multi_ts if len(ts["timeline"][0]["data"]) > 0]
+if len(filtered_ts) > 0 and len(filtered_ts[0]["timeline"]) > 0:
+ keys = filtered_ts[0]["timeline"][0]["data"].keys()
+ timelines = ([res["timeline"][0]["data"][key] for key in keys] for res
in filtered_ts)
values = (max(v) for v in zip(*timelines))
return dict(zip(keys, values))
return {}
[incubator-heron] branch master updated: Avoid duplicated metrics requests in Heron UI (#3378)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 7987f4a Avoid duplicated metrics requests in Heron UI (#3378)
7987f4a is described below
commit 7987f4a218fe0f89a7fc83023bbc56691ec192a4
Author: Ning Wang
AuthorDate: Thu Oct 24 21:22:52 2019 -0700
Avoid duplicated metrics requests in Heron UI (#3378)
Same request might be sent three times which gives heron UI pressure and
makes UI slow.
---
heron/tools/ui/resources/static/js/topologies.js | 90 +---
1 file changed, 50 insertions(+), 40 deletions(-)
diff --git a/heron/tools/ui/resources/static/js/topologies.js
b/heron/tools/ui/resources/static/js/topologies.js
index 9fa9f23..73860d1 100644
--- a/heron/tools/ui/resources/static/js/topologies.js
+++ b/heron/tools/ui/resources/static/js/topologies.js
@@ -23,6 +23,8 @@ var SUM = 0,
AVG = 1,
LAST = 2;
+var countersUrlFlags = {};
+
var AllExceptions = React.createClass({
getInitialState: function() {
return {}
@@ -480,50 +482,58 @@ var AllMetrics = React.createClass({
},
fetchCountersURL: function(url, timeRange) {
-$.ajax({
- url: url,
- dataType: 'json',
- success: function(response) {
-if (response.hasOwnProperty("metrics")) {
- var component = response.component;
- var metrics = this.metrics;
- if (!metrics.hasOwnProperty(component)) {
-metrics[response.component] = {};
- }
- if (!metrics[component].hasOwnProperty(timeRange)) {
-metrics[component][timeRange] = {};
- }
- for (var name in response.metrics) {
-if (response.metrics.hasOwnProperty(name)) {
- var metricname = name;
- // Handle __jvm-uptime-secs as a special case.
- if (name !== "__jvm-uptime-secs") {
-metricname = name.split("/")[0] + "/";
- }
- var displayName = this.supportedMetricNames[metricname].name;
- if (!metrics[component][timeRange].hasOwnProperty(displayName)) {
-metrics[component][timeRange][displayName] = {};
- }
- var tmpMetrics = {
-metrics: response.metrics[name],
-scaleDevisor:
this.supportedMetricNames[metricname].scaleDevisor,
-aggregationType:
this.supportedMetricNames[metricname].aggregationType
- };
- if (name === "__jvm-uptime-secs") {
-metrics[component][timeRange][displayName][""] = tmpMetrics;
- } else {
-metrics[component][timeRange][displayName][name.split("/")[1]]
= tmpMetrics;
+// This function might be called multiple times with the same url. Note
+// that the timeRange value is inlcuded in the url so it can be ignored in
+// the check. We check if the request has been made before first and
+// skip if this is a duplicated request.
+if (!countersUrlFlags[url]) {
+ countersUrlFlags[url] = true; // Set the flag
+
+ $.ajax({
+url: url,
+dataType: 'json',
+success: function(response) {
+ if (response.hasOwnProperty("metrics")) {
+var component = response.component;
+var metrics = this.metrics;
+if (!metrics.hasOwnProperty(component)) {
+ metrics[response.component] = {};
+}
+if (!metrics[component].hasOwnProperty(timeRange)) {
+ metrics[component][timeRange] = {};
+}
+for (var name in response.metrics) {
+ if (response.metrics.hasOwnProperty(name)) {
+var metricname = name;
+// Handle __jvm-uptime-secs as a special case.
+if (name !== "__jvm-uptime-secs") {
+ metricname = name.split("/")[0] + "/";
+}
+var displayName = this.supportedMetricNames[metricname].name;
+if
(!metrics[component][timeRange].hasOwnProperty(displayName)) {
+ metrics[component][timeRange][displayName] = {};
+}
+var tmpMetrics = {
+ metrics: response.metrics[name],
+ scaleDevisor:
this.supportedMetricNames[metricname].scaleDevisor,
+ aggregationType:
this.supportedMetricNames[metricname].aggregationType
+};
+if (name === "__jvm-uptime-secs") {
+ metrics[component][timeRange][displayName][""] = tmpMetrics;
+} else {
+
metrics[component][timeRange][displayName][name.spli
[incubator-heron] branch nwangtw-patch-1 updated (066dd8e -> 13a59bc)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch nwangtw-patch-1 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from 066dd8e Add instance resource usage in container (#3375) add 13a59bc Update issue templates No new revisions were added by this update. Summary of changes: .github/ISSUE_TEMPLATE/bug_report.md | 38 +++ .github/ISSUE_TEMPLATE/custom.md | 10 .github/ISSUE_TEMPLATE/feature_request.md | 20 3 files changed, 68 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/custom.md create mode 100644 .github/ISSUE_TEMPLATE/feature_request.md
[incubator-heron] branch nwangtw-patch-1 created (now 066dd8e)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch nwangtw-patch-1 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. at 066dd8e Add instance resource usage in container (#3375) No new revisions were added by this update.
[incubator-heron] branch master updated: Add instance resource usage in container (#3375)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 066dd8e Add instance resource usage in container (#3375)
066dd8e is described below
commit 066dd8e66367e154604d7d939aad8d1cf19591ee
Author: Ning Wang
AuthorDate: Thu Oct 24 17:34:46 2019 -0700
Add instance resource usage in container (#3375)
* Add instance resource usage in container
* update comment and trigger build
---
.../tools/ui/resources/static/js/physical-plan.js | 47 +-
1 file changed, 38 insertions(+), 9 deletions(-)
diff --git a/heron/tools/ui/resources/static/js/physical-plan.js
b/heron/tools/ui/resources/static/js/physical-plan.js
index 49fe75f..cf43232 100644
--- a/heron/tools/ui/resources/static/js/physical-plan.js
+++ b/heron/tools/ui/resources/static/js/physical-plan.js
@@ -63,16 +63,23 @@
container.children = _.sortBy(container.children, 'name');
// Parse index
container.index = parseInt(container.id.split('-')[1]);
- // Search for resource config in packing plan
+ // Search for container and instance resource config in packing plan
container.required_resources = {
-'cpu': 0,
-'disk': 0,
-'ram': 0
+'cpu': 'unknown',
+'disk': 'unknown',
+'ram': 'unknown'
};
for (var i in containerPlan) {
var packing = containerPlan[i];
if (packing.id === container.index) {
container.required_resources = packing.required_resources;
+ // Get instance resources
+ var instance_resources = {};
+ for (var j in packing.instances) {
+var inst = packing.instances[j];
+instance_resources[inst.component_name + '_' + inst.task_id] =
inst.instance_resources;
+ }
+ container.instance_resources = instance_resources;
}
}
});
@@ -86,6 +93,18 @@
return d.children.length;
});
+function formatByteSize(byteSize) {
+ if (byteSize > 1024 * 1024 * 1024 / 2) {
+return (byteSize / (1024 * 1024 * 1024)).toFixed(2) + 'GB';
+ } else if (byteSize > 1024 * 1024 / 2) {
+return (byteSize / (1024 * 1024)).toFixed(2) + 'MB';
+ } else if (byteSize > 1024 / 2) {
+return (byteSize / 1024).toFixed(2) + 'KB';
+ } else {
+return byteSize + 'B';
+ }
+}
+
/**
* Config paramaters for container/heron instance layout
*/
@@ -159,15 +178,25 @@
});
var container_tip = d3.tip()
-.attr('class', 'd3-tip main text-center container')
+.attr('class', 'd3-tip main container')
.offset([8, 0])
.direction('s')
.html(function (container) {
+ var instance_cpu = '';
+ var instance_ram = '';
+ var instance_disk = '';
+ if (container.instance_resources) {
+for (var inst in container.instance_resources) {
+ instance_cpu += ' - ' + inst + ': ' +
container.instance_resources[inst].cpu + '';
+ instance_ram += ' - ' + inst + ': ' +
formatByteSize(container.instance_resources[inst].ram) + '';
+ instance_disk += ' - ' + inst + ': ' +
formatByteSize(container.instance_resources[inst].disk) + '';
+}
+ }
return '' +
- 'cpu required: ' + container.required_resources.cpu + '' +
- 'ram required: ' + (container.required_resources.ram /
1048576).toFixed(2) + 'MB' +
- 'disk required: ' + (container.required_resources.disk /
1048576).toFixed(2) + 'MB' +
- '';
+ 'cpu required: ' + container.required_resources.cpu +
'' + instance_cpu +
+ 'ram required: ' +
formatByteSize(container.required_resources.ram) + '' +
instance_ram +
+ 'disk required: ' +
formatByteSize(container.required_resources.disk) + '' +
instance_disk +
+ '';
});
var containerGs = svg.selectAll('.physical-plan')
[incubator-heron] branch master updated: Clean up license for healthHandler and jarjar help.txt (#3377)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new a383bc8 Clean up license for healthHandler and jarjar help.txt (#3377) a383bc8 is described below commit a383bc894e052eea7a1473dc5577471155572056 Author: Ning Wang AuthorDate: Thu Oct 24 17:29:56 2019 -0700 Clean up license for healthHandler and jarjar help.txt (#3377) --- LICENSE| 3 +- LICENSE => licenses/LICENSE-jarjar.txt | 123 + 2 files changed, 2 insertions(+), 124 deletions(-) diff --git a/LICENSE b/LICENSE index 0498e42..3c18e55 100644 --- a/LICENSE +++ b/LICENSE @@ -248,10 +248,9 @@ See project link for details. -> scripts/packages/tests_template_bin.sh -> scripts/packages/bin_common.sh -> third_party/java/bazel/extra_actions_base.proto -Google +jarjar(https://github.com/google/jarjar/blob/master/LICENSE) -> third_party/java/jarjar/src/main/resources/com/tonicsystems/jarjar/help.txt - Third party MIT licenses diff --git a/LICENSE b/licenses/LICENSE-jarjar.txt similarity index 64% copy from LICENSE copy to licenses/LICENSE-jarjar.txt index 0498e42..d645695 100644 --- a/LICENSE +++ b/licenses/LICENSE-jarjar.txt @@ -1,3 +1,4 @@ + Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ @@ -199,125 +200,3 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. - - - - APACHE HERON SUBCOMPONENTS: - - The Apache Heron project contains subcomponents with separate copyright - notices and license terms. Your use of the source code for the these - subcomponents is subject to the terms and conditions of the following - licenses. - - -Third party Apache 2.0 licenses - - -The following components are provided under the Apache 2.0 License. -See project link for details. - -Bootstrap(v2.0.3, https://github.com/twbs/bootstrap/blob/v2.0.3/LICENSE) - -> heron/shell/assets/bootstrap.css -Bootstrap(v2.2.2, https://github.com/twbs/bootstrap/blob/v2.2.2/LICENSE) - -> heron/tools/ui/resources/static/js/bootstrap.js -Bootstrap(v3.0.0, https://github.com/twbs/bootstrap/blob/v3.0.0/LICENSE) - -> heron/tools/ui/resources/static/css/bootstrap.css -JQuery(https://jquery.org/license/) - -> heron/shell/assets/jquery.js - -> heron/tools/ui/resources/static/js/jquery.min.js -JSXTransformer(v0.10.0) - -> heron/tools/ui/resources/static/js/JSXTransformer.0.10.0.js -React (v0.10.0, https://github.com/facebook/react/blob/0.10-stable/LICENSE) - -> heron/tools/ui/resources/static/js/react.0.10.0.js - bazel_jar_jar(https://github.com/johnynek/bazel_jar_jar/blob/master/LICENSE) - -> tools/rules/jarjar_rules.bzl -gerrit(https://gerrit.googlesource.com/gerrit/+/master/COPYING) - -> tools/rules/javadoc.bzl - k8s-zookeeper-docker(https://github.com/muxinc/k8s-zookeeper-docker/blob/master/LICENSE) - -> docker/base/scripts/generate-zookeeper-config.sh - bazel_rules_pex(v0.3.0,https://github.com/benley/bazel_rules_pex/blob/master/LICENSE.txt) - -> tools/rules/pex/pex_rules.bzl - -> tools/rules/pex/wrapper/pex_wrapper.py -bazel_authors(https://github.com/bazelbuild/bazel/blob/master/LICENSE) - -> scripts/setup-eclipse.sh - -> scripts/detect_os_type.sh - -> scripts/get_all_heron_paths.sh - -> scripts/packages/template_bin.sh - -> scripts/packages/self_extract_binary.bzl - -> scripts/packages/package_info_generator.sh - -> scripts/packages/tests_template_bin.sh - -> scripts/packages/bin_common.sh - -> third_party/java/bazel/extra_actions_base.proto -Google - -> third_party/java/jarjar/src/main/resources/com/tonicsystems/jarjar/help.txt - - - -Third party MIT licenses - - -The following components are provided under the MIT License. See project link for details. - -Bootstrap(v3.1.1, https://github.com/twbs/bootstrap/blob/v3.1.1/LICENSE) - -> heron/tools/ui/resources/static/js/bootstrap.min
[incubator-heron] branch master updated: Fix license in health handler py (#3376)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 31b6667 Fix license in health handler py (#3376) 31b6667 is described below commit 31b66674df8425cfd2812b70d33aa7018d6f6b95 Author: Ning Wang AuthorDate: Wed Oct 23 08:58:56 2019 -0700 Fix license in health handler py (#3376) --- LICENSE | 2 -- heron/shell/src/python/handlers/healthhandler.py | 23 +-- 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/LICENSE b/LICENSE index 360aa7f..0498e42 100644 --- a/LICENSE +++ b/LICENSE @@ -248,8 +248,6 @@ See project link for details. -> scripts/packages/tests_template_bin.sh -> scripts/packages/bin_common.sh -> third_party/java/bazel/extra_actions_base.proto -Twitter - -> heron/shell/src/python/handlers/healthhandler.py Google -> third_party/java/jarjar/src/main/resources/com/tonicsystems/jarjar/help.txt diff --git a/heron/shell/src/python/handlers/healthhandler.py b/heron/shell/src/python/handlers/healthhandler.py index 26bd667..2455ad5 100644 --- a/heron/shell/src/python/handlers/healthhandler.py +++ b/heron/shell/src/python/handlers/healthhandler.py @@ -1,19 +1,22 @@ #!/usr/bin/env python # -*- encoding: utf-8 -*- -# Copyright 2016 Twitter. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at # #http://www.apache.org/licenses/LICENSE-2.0 # -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. """ healthhandler.py """ import tornado.web
[incubator-heron] branch master updated: adding missing licenses. Updating NOTICE (#3370)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 4a600ee adding missing licenses. Updating NOTICE (#3370) 4a600ee is described below commit 4a600ee756e791a4ea8368b4a7189a81071e81d9 Author: Josh Fischer AuthorDate: Mon Oct 21 17:00:53 2019 -0500 adding missing licenses. Updating NOTICE (#3370) --- NOTICE | 2 +- licenses/LICENSE-Docusaurus-1.13.0.txt | 21 +++ licenses/LICENSE-bazel_rules_pex-0.3.0.txt | 202 + 3 files changed, 224 insertions(+), 1 deletion(-) diff --git a/NOTICE b/NOTICE index 8212508..e16c305 100644 --- a/NOTICE +++ b/NOTICE @@ -1,5 +1,5 @@ Apache Heron -Copyright 2019 The Apache Software Foundation +Copyright 2018-2019 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). diff --git a/licenses/LICENSE-Docusaurus-1.13.0.txt b/licenses/LICENSE-Docusaurus-1.13.0.txt new file mode 100644 index 000..4f5f321 --- /dev/null +++ b/licenses/LICENSE-Docusaurus-1.13.0.txt @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2017-present, Facebook, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. \ No newline at end of file diff --git a/licenses/LICENSE-bazel_rules_pex-0.3.0.txt b/licenses/LICENSE-bazel_rules_pex-0.3.0.txt new file mode 100644 index 000..7a4a3ea --- /dev/null +++ b/licenses/LICENSE-bazel_rules_pex-0.3.0.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Deriva
[incubator-heron] branch master updated: Show an arrow head in logical plan (#3367)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new a25b087 Show an arrow head in logical plan (#3367)
a25b087 is described below
commit a25b087773d5348ecc9e42e7d69403056f61b45f
Author: Ning Wang
AuthorDate: Mon Oct 21 10:44:36 2019 -0700
Show an arrow head in logical plan (#3367)
---
heron/tools/ui/resources/static/js/logical-plan.js | 36 --
1 file changed, 33 insertions(+), 3 deletions(-)
diff --git a/heron/tools/ui/resources/static/js/logical-plan.js
b/heron/tools/ui/resources/static/js/logical-plan.js
index ac300c9..d74651a 100644
--- a/heron/tools/ui/resources/static/js/logical-plan.js
+++ b/heron/tools/ui/resources/static/js/logical-plan.js
@@ -250,6 +250,24 @@
var svg = outerSvg.append("g")
.attr("tranform", "translate(" + padding.left + "," +
padding.top + ")");
+var defs = svg.append("defs")
+
+// Arrow head
+defs.append("marker")
+.attr({
+ "id": "arrow",
+ "viewBox": "0 -5 10 10",
+ "refX": 5,
+ "refY": 0,
+ "markerWidth": 2,
+ "markerHeight": 2,
+ "orient": "auto"
+})
+.append("path")
+ .attr("d", "M0,-5L10,0L0,5")
+ .attr("class","arrowHead")
+ .style("fill", linestyle.color);
+
spoutsArr = [];
boltsArr = [];
@@ -363,6 +381,8 @@
.attr("class", "topnode")
.style("fill", "black");
+var compCircleRadius = 17;
+
// Links
node.each(function (n) {
d3.select(this)
@@ -371,6 +391,7 @@
.enter()
.append("path")
.attr('class', 'link')
+.attr("marker-end", "url(#arrow)")
.attr("stroke-width", linestyle.boldwidth)
.attr("stroke", linestyle.color)
.attr("fill", "none")
@@ -378,7 +399,16 @@
var p0 = edge.source;
var p3 = edge.target;
var m = (p0.x + p3.x) / 2;
- var p = [p0, {x: m, y: p0.y}, {x: m, y: p3.y}, p3];
+ var p0x = p0.x + compCircleRadius;
+ var p0y = p0.y;
+ var p3x = p3.x - compCircleRadius;
+ var p3y = p3.y;
+ var p = [
+{x: p0x, y: p0y},
+{x: m, y: p0y},
+{x: m, y: p3y},
+{x: p3x, y: p3.y}
+ ];
return "M" + p[0].x + " " + p[0].y +
"C" + p[1].x + " " + p[1].y +
" " + p[2].x + " " + p[2].y +
@@ -400,7 +430,7 @@
.attr('class', 'background')
.attr("r", function (d) {
if (d.isReal) {
-return d.r = 17;
+return d.r = compCircleRadius;
}
return d.r = 0;
})
@@ -410,7 +440,7 @@
.attr("class", "node")
.attr("r", function (d) {
if (d.isReal) {
-return d.r = 15;
+return d.r = compCircleRadius - 2;
}
return d.r = 0;
})
[incubator-heron] branch master updated: Change urls in WORKSPACE file from http to https (#3369)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 7dbe872 Change urls in WORKSPACE file from http to https (#3369) 7dbe872 is described below commit 7dbe87238bc3e6f3a8248c53a38d875c2a0ea70f Author: Ning Wang AuthorDate: Sun Oct 20 23:34:36 2019 -0700 Change urls in WORKSPACE file from http to https (#3369) --- WORKSPACE | 19 +-- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/WORKSPACE b/WORKSPACE index bfb17d3..5853ef5 100644 --- a/WORKSPACE +++ b/WORKSPACE @@ -37,7 +37,12 @@ hk2_api = "2.5.0-b32" maven_server( name = "default", - url = "http://central.maven.org/maven2/;, + url = "https://repo1.maven.org/maven2/;, +) + +maven_server( + name = "maven_twttr_com", + url = "https://maven.twttr.com;, ) maven_jar( @@ -45,11 +50,6 @@ maven_jar( artifact = "org.apache.avro:avro:1.7.4" ) -maven_server( - name = "maven_twttr_com", - url = "http://maven.twttr.com;, -) - maven_jar( name = "antlr_antlr", artifact = "antlr:antlr:2.7.7", @@ -177,7 +177,7 @@ maven_jar( http_jar( name = "org_apache_httpcomponents_http_client_test", - url = "http://central.maven.org/maven2/org/apache/httpcomponents/httpclient/; + + url = "https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/; + http_client_version + "/httpclient-" + http_client_version + "-tests.jar" ) @@ -769,7 +769,6 @@ maven_jar( maven_jar( name = "commons_codec_commons_codec", artifact = "commons-codec:commons-codec:1.9", - repository = "http://central.maven.org/maven2/;, sha1 = "9ce04e34240f674bc72680f8b843b1457383161a", ) @@ -864,7 +863,7 @@ http_archive( http_archive( name = "org_nongnu_libunwind", -urls = ["http://download.savannah.nongnu.org/releases/libunwind/libunwind-1.1.tar.gz;], +urls = ["https://download.savannah.nongnu.org/releases/libunwind/libunwind-1.1.tar.gz;], strip_prefix = "libunwind-1.1", build_file = "@//:third_party/libunwind/libunwind.BUILD", sha256 = "9dfe0fcae2a866de9d3942c66995e4b460230446887dbdab302d41a8aee8d09a", @@ -872,7 +871,7 @@ http_archive( http_archive( name = "org_apache_zookeeper", -urls = ["http://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz;], +urls = ["https://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz;], strip_prefix = "zookeeper-3.4.14", build_file = "@//:third_party/zookeeper/zookeeper.BUILD", sha256 = "b14f7a0fece8bd34c7fffa46039e563ac5367607c612517aa7bd37306afbd1cd",
[incubator-heron] branch master updated: License work (#3368)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new de2fb39 License work (#3368) de2fb39 is described below commit de2fb3939728ca4cc4e37ca5b76460203de9d875 Author: Josh Fischer AuthorDate: Sun Oct 20 18:53:34 2019 -0500 License work (#3368) * update correct year in notice * update LICENSE file --- LICENSE | 41 + NOTICE | 2 +- 2 files changed, 42 insertions(+), 1 deletion(-) diff --git a/LICENSE b/LICENSE index 02e56a2..7129002 100644 --- a/LICENSE +++ b/LICENSE @@ -235,6 +235,24 @@ See project link for details. -> tools/rules/javadoc.bzl k8s-zookeeper-docker(https://github.com/muxinc/k8s-zookeeper-docker/blob/master/LICENSE) -> docker/base/scripts/generate-zookeeper-config.sh + bazel_rules_pex(v0.3.0,https://github.com/benley/bazel_rules_pex/blob/master/LICENSE.txt) + -> tools/rules/pex/pex_rules.bzl + -> tools/rules/pex/wrapper/pex_wrapper.py +bazel_authors(https://github.com/bazelbuild/bazel/blob/master/LICENSE) + -> scripts/setup-eclipse.sh + -> scripts/detect_os_type.sh + -> scripts/get_all_heron_paths.sh + -> scripts/packages/template_bin.sh + -> scripts/packages/self_extract_binary.bzl + -> scripts/packages/package_info_generator.sh + -> scripts/packages/tests_template_bin.sh + -> scripts/packages/bin_common.sh + -> third_party/java/bazel/extra_actions_base.proto +Twitter + -> heron/shell/src/python/handlers/healthhandler.py +Google + -> third_party/java/jarjar/src/main/resources/com/tonicsystems/jarjar/help.txt + Third party MIT licenses @@ -264,6 +282,17 @@ The following components are provided under the MIT License. See project link fo normalize.css(v2.1.0, v3.0.1, https://github.com/necolas/normalize.css/blob/master/LICENSE.md) -> heron/tools/ui/resources/static/css/bootstrap.css -> heron/tools/ui/resources/static/css/bootstrap.min.css +Docusaurus(v1.13.0, https://github.com/facebook/docusaurus/blob/master/LICENSE) + -> website2/website/core/Footer.js + -> website2/website/siteConfig.js + -> website2/website/pages/en/users.js + -> website2/website/pages/en/index.js + -> website2/website/pages/en/help.js + -> website2/website/pages/en/versions.js +Sizzle(https://github.com/jquery/sizzle/blob/master/LICENSE.txt) + -> heron/shell/assets/jquery.js +Modernizr(http://www.modernizr.com) + -> heron/tools/ui/resources/static/js/bootstrap.js Third party BSD 3-Clause licenses @@ -282,6 +311,8 @@ See project link for details. -> third_party/python/cpplint/cpplint.py d3(v3.4.11, https://github.com/d3/d3/blob/master/LICENSE) -> heron/tools/ui/resources/static/js/d3.min.3.4.11.js +Sizzle(https://github.com/jquery/sizzle/blob/master/LICENSE.txt) + -> heron/shell/assets/jquery.js Third party Boost Software License, Version 1.0 licenses @@ -292,3 +323,13 @@ See project link for details. kashmir(https://github.com/Corvusoft/kashmir-dependency/blob/master/LICENSE_1_0.txt) -> third_party/kashmir/* + + +Third party GNU General Public License, Version 3.0 licenses + + +The following components are provided under the GNU General Public License, Version 3.0. +See project link for details. + +Sizzle(https://github.com/jquery/sizzle/blob/master/LICENSE.txt) + -> heron/shell/assets/jquery.js diff --git a/NOTICE b/NOTICE index 0ef7a26..8212508 100644 --- a/NOTICE +++ b/NOTICE @@ -1,5 +1,5 @@ Apache Heron -Copyright 2018 The Apache Software Foundation +Copyright 2019 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/).
[incubator-heron] branch master updated: Add a script to start docker container with development environment (#3366)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new d950db5 Add a script to start docker container with development
environment (#3366)
d950db5 is described below
commit d950db583b8097443bd0de29338d8b0d25ed63f4
Author: Ning Wang
AuthorDate: Fri Oct 18 23:05:48 2019 -0700
Add a script to start docker container with development environment (#3366)
* Add a script to start docker container with development environment
* update
* update comments for starting a new shell
* map ports and 8889
---
docker/compile/Dockerfile.centos7 | 6 +--
docker/compile/Dockerfile.ubuntu18.04 | 8 ++--
docker/scripts/dev-env.sh | 87 +++
3 files changed, 95 insertions(+), 6 deletions(-)
diff --git a/docker/compile/Dockerfile.centos7
b/docker/compile/Dockerfile.centos7
index 3865b5b..3eda6c4 100644
--- a/docker/compile/Dockerfile.centos7
+++ b/docker/compile/Dockerfile.centos7
@@ -41,9 +41,9 @@ RUN yum -y install \
zip \
unzip \
wget \
- which
-
-RUN yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel
+ which \
+ java-1.8.0-openjdk \
+ java-1.8.0-openjdk-devel
ENV JAVA_HOME /usr/lib/jvm/java-1.8.0
diff --git a/docker/compile/Dockerfile.ubuntu18.04
b/docker/compile/Dockerfile.ubuntu18.04
index 28b8f23..48a62ef 100644
--- a/docker/compile/Dockerfile.ubuntu18.04
+++ b/docker/compile/Dockerfile.ubuntu18.04
@@ -31,9 +31,11 @@ RUN apt-get update && apt-get -y install \
python-dev \
wget \
zip \
- unzip
-
-RUN apt-get -y install openjdk-8-jdk-headless
+ virtualenv \
+ unzip \
+ git \
+ curl \
+ openjdk-8-jdk-headless
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
diff --git a/docker/scripts/dev-env.sh b/docker/scripts/dev-env.sh
new file mode 100755
index 000..17e37ed
--- /dev/null
+++ b/docker/scripts/dev-env.sh
@@ -0,0 +1,87 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a script to start a docker container that has all
+# tools needed to compire Heron. Developer should be able to
+# compile Heron in the container without any setup works.
+# usage:
+# To create a clean development environment with docker and run it,
+# execute the following scripts in the source directory of Heron:
+# sh docker/scripts/dev-env.sh
+#
+# After the container is started, build Heron with bazel
+# (ubuntu config is used in the example):
+# ./bazel_configure.py
+# bazel build --config=ubuntu heron/...
+#
+# To enter an existing container with a new shell, find the container
+# id with this command:
+# docker ps -a
+# And then :
+# docker exec -it CONTAINER_ID bash
+
+set -o nounset
+set -o errexit
+
+# Default platform is ubuntu18.04. Other available platforms
+# include centos7, debian9
+TARGET_PLATFORM=${1:-"ubuntu18.04"}
+SCRATCH_DIR=${2:-"$HOME/.heron-docker"}
+REPOSITORY="heron-dev"
+
+
+realpath() {
+ echo "$(cd "$(dirname "$1")"; pwd)/$(basename "$1")"
+}
+
+DOCKER_DIR=$(dirname $(dirname $(realpath $0)))
+PROJECT_DIR=$(dirname $DOCKER_DIR)
+
+verify_dockerfile_exists() {
+ if [ ! -f $1 ]; then
+echo "The Dockerfiler $1 does not exist"
+exit 1
+ fi
+}
+
+dockerfile_path_for_platform() {
+ echo "$DOCKER_DIR/compile/Dockerfile.$1"
+}
+
+copy_extra_files() {
+ mkdir -p $SCRATCH_DIR/scripts
+ cp $PROJECT_DIR/tools/docker/bazel.rc $SCRATCH_DIR/bazelrc
+ cp $DOCKER_DIR/scripts/compile-docker.sh
$SCRATCH_DIR/scripts/compile-platform.sh
+}
+
+DOCKER_FILE=$(dockerfile_path_for_platform $TARGET_PLATFORM)
+verify_dockerfile_exists $DOCKER_FILE
+copy_extra_files
+
+echo "Building docker image for Heron development environment on
$TARGET_PLATFORM"
+docker build -t $REPOSITORY:$TARGET_PLATFORM -f $DOCKER_FILE $SCRATCH_DIR
+
+echo "Creating and starting container and mapping the current dir to /heron"
+docker con
[incubator-heron] branch master updated: publish site script (#3365)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new bf4fce8 publish site script (#3365)
bf4fce8 is described below
commit bf4fce897607f4217b7243acd0233db99d7a4849
Author: Josh Fischer
AuthorDate: Fri Oct 18 10:52:16 2019 -0500
publish site script (#3365)
* update htacces and create new publish site script
* adding publish_site script to makefile.
* adding fully qualified path
---
.htaccess| 2 +-
website2/website/Makefile| 3 +++
website2/website/scripts/publish_site.sh | 36
3 files changed, 40 insertions(+), 1 deletion(-)
diff --git a/.htaccess b/.htaccess
index 6ff83b2..bdcbac5 100644
--- a/.htaccess
+++ b/.htaccess
@@ -1 +1 @@
-Redirect /index.html https://apache.github.io/incubator-heron/
+Redirect /index.html https://heron.incubator.apache.org
diff --git a/website2/website/Makefile b/website2/website/Makefile
index 1465eb0..7c25991 100644
--- a/website2/website/Makefile
+++ b/website2/website/Makefile
@@ -11,3 +11,6 @@ buildsite:
@scripts/javadocs.sh
@scripts/python-doc-gen.sh 0.0.0
@scripts/build-site.sh
+
+publish:
+ @scripts/publish_site.sh
diff --git a/website2/website/scripts/publish_site.sh
b/website2/website/scripts/publish_site.sh
new file mode 100755
index 000..575b399
--- /dev/null
+++ b/website2/website/scripts/publish_site.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ROOT_DIR=$(git rev-parse --show-toplevel)
+WORK_DIR=${ROOT_DIR}/generated-site/content
+ME=`basename $0`
+
+# push all of the results to asf-site branch
+git checkout asf-site
+git clean -f -d
+git pull origin asf-site
+
+rm -rf ${ROOT_DIR}/content
+mkdir ${ROOT_DIR}/content
+
+cp -a $WORK_DIR/* ${ROOT_DIR}/content
+cp -a ${ROOT_DIR}/.htaccess ${ROOT_DIR}/content
+
+git add ${ROOT_DIR}/content
+git commit -m "git-site-role commit from $ME"
+git push origin asf-site
[incubator-heron] branch master updated: Refine site build (#3363)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 03199a9 Refine site build (#3363) 03199a9 is described below commit 03199a939a69b3d46285d1afe45a816f8824837c Author: Josh Fischer AuthorDate: Tue Oct 15 23:21:19 2019 -0500 Refine site build (#3363) * clean up broken links in MD file. Update gitignore for committing static docs. tweaking static site build scripts * update script * adding page to link to java and py docs. updated build site script to appropriately place static assets. * adding versioned docs for latest release * update versioned docs * comment out blog header link. remove broken links in python docs * fixing heron version references. recreateing versioned docs. * clean up in gitignore. Adding missing ticks in python docs. * clean up --- .gitignore | 3 +- website2/docs/client-api-docs-overview.md | 25 +++ .../getting-started-migrate-storm-topologies.md| 4 +- website2/docs/guides-python-topologies.md | 40 ++-- .../docs/topology-development-streamlet-api.md | 2 - .../docs/topology-development-topology-api-java.md | 2 +- .../topology-development-topology-api-python.md| 2 +- website2/website/Makefile | 2 + website2/website/scripts/build-site.sh | 11 +- website2/website/scripts/python-doc-gen.sh | 12 +- website2/website/sidebars.json | 1 + website2/website/siteConfig.js | 4 +- .../client-api-docs-overview.md| 26 +++ .../cluster-config-instance.md | 2 +- .../cluster-config-metrics.md | 2 +- .../cluster-config-overview.md | 2 +- .../cluster-config-stream.md | 2 +- .../cluster-config-system-level.md | 2 +- .../cluster-config-tmaster.md | 2 +- .../compiling-code-organization.md | 2 +- .../compiling-docker.md| 2 +- .../compiling-linux.md | 2 +- .../compiling-osx.md | 2 +- .../compiling-overview.md | 2 +- .../compiling-running-tests.md | 2 +- .../deployment-api-server.md | 2 +- .../deployment-configuration.md| 2 +- .../deployment-overview.md | 2 +- .../extending-heron-metric-sink.md | 2 +- .../extending-heron-scheduler.md | 2 +- .../getting-started-local-single-node.md | 5 +- .../getting-started-migrate-storm-topologies.md| 6 +- .../getting-started-troubleshooting-guide.md | 2 +- .../guides-data-model.md | 2 +- .../guides-effectively-once-java-topologies.md | 2 +- .../guides-packing-algorithms.md | 2 +- .../guides-python-topologies.md| 42 ++--- .../guides-simulator-mode.md | 2 +- .../guides-topology-tuning.md | 2 +- .../guides-troubeshooting-guide.md | 2 +- .../guides-tuple-serialization.md | 2 +- .../guides-ui-guide.md | 2 +- .../heron-architecture.md | 2 +- .../heron-delivery-semantics.md| 2 +- .../heron-design-goals.md | 2 +- .../heron-resources-resources.md | 2 +- .../heron-streamlet-concepts.md| 2 +- .../heron-topology-concepts.md | 2 +- .../observability-graphite.md | 2 +- .../observability-prometheus.md| 2 +- .../observability-scribe.md| 2 +- .../schedulers-aurora-cluster.md | 2 +- .../schedulers-aurora-local.md | 2 +- .../schedulers-k8s-by-hand.md | 2 +- .../schedulers-k8s-with-helm.md| 2 +- .../schedulers-local.md| 2 +- .../schedulers-mesos-local-mac.md | 2 +- .../schedulers-nomad.md| 2 +- .../schedulers-slurm.md| 2 +- .../schedulers-standalone.md | 2 +- .../schedulers-yarn.md | 2 +- .../state-managers-local-fs.md | 2 +- .../state-managers-zookeeper.md| 2 +- .../topology-development-eco-api.md| 2 +- .../topology
[incubator-heron] branch master updated: Add metrics for tracking bp caused by local instances and remote stmgrs (#3362)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 08a3631 Add metrics for tracking bp caused by local instances and
remote stmgrs (#3362)
08a3631 is described below
commit 08a3631913e36df92aa45f4e637ac64d4162382a
Author: Ning Wang
AuthorDate: Wed Oct 9 11:46:27 2019 -0700
Add metrics for tracking bp caused by local instances and remote stmgrs
(#3362)
---
heron/stmgr/src/cpp/manager/instance-server.cpp | 19 +--
heron/stmgr/src/cpp/manager/instance-server.h | 5 +++--
heron/stmgr/src/cpp/manager/stmgr-server.cpp| 22 ++
heron/stmgr/src/cpp/manager/stmgr-server.h | 4 +++-
4 files changed, 41 insertions(+), 9 deletions(-)
diff --git a/heron/stmgr/src/cpp/manager/instance-server.cpp
b/heron/stmgr/src/cpp/manager/instance-server.cpp
index 9a752de..047c634 100644
--- a/heron/stmgr/src/cpp/manager/instance-server.cpp
+++ b/heron/stmgr/src/cpp/manager/instance-server.cpp
@@ -67,13 +67,16 @@ const sp_string METRIC_FAIL_TUPLES_TO_INSTANCES_LOST =
"__fail_tuples_to_workers
// Num bytes lost since instances is not connected
const sp_string METRIC_BYTES_TO_INSTANCES_LOST = "__bytes_to_workers_lost";
+// Time spent in back pressure caused by instances managed by this stmgr.
+const sp_string METRIC_TIME_SPENT_BACK_PRESSURE_CAUSED_BY_LOCAL_INSTANCE =
+"__time_spent_back_pressure_by_local_instance";
// Time spent in back pressure aggregated - back pressure initiated by us +
// others
const sp_string METRIC_TIME_SPENT_BACK_PRESSURE_AGGR =
"__server/__time_spent_back_pressure_aggr";
// Time spent in back pressure because of a component id. The comp id will be
-// appended
-// to the string below
+// appended to the string below
const sp_string METRIC_TIME_SPENT_BACK_PRESSURE_COMPID =
"__time_spent_back_pressure_by_compid/";
+
// Prefix for connection buffer's metrics
const sp_string CONNECTION_BUFFER_BY_INSTANCEID =
"__connection_buffer_by_instanceid/";
// Prefix for connection buffer's length metrics. This is different
@@ -117,6 +120,9 @@ InstanceServer::InstanceServer(
metrics_manager_client_->register_metric("__server",
instance_server_metrics_);
metrics_manager_client_->register_metric(METRIC_TIME_SPENT_BACK_PRESSURE_AGGR,
back_pressure_metric_aggr_);
+ back_pressure_metric_caused_by_local_instances_ =
make_shared();
+
metrics_manager_client_->register_metric(METRIC_TIME_SPENT_BACK_PRESSURE_CAUSED_BY_LOCAL_INSTANCE,
+
back_pressure_metric_caused_by_local_instances_);
spouts_under_back_pressure_ = false;
// Update queue related metrics every 10 seconds
@@ -180,6 +186,8 @@ InstanceServer::~InstanceServer() {
metrics_manager_client_->unregister_metric("__server");
metrics_manager_client_->unregister_metric(METRIC_TIME_SPENT_BACK_PRESSURE_AGGR);
+ metrics_manager_client_->unregister_metric(
+ METRIC_TIME_SPENT_BACK_PRESSURE_CAUSED_BY_LOCAL_INSTANCE);
// cleanup the instance info
for (auto iter = instance_info_.begin(); iter != instance_info_.end();
++iter) {
@@ -549,6 +557,8 @@ sp_string InstanceServer::GetInstanceName(Connection*
_connection) {
return "";
}
+// This function is called when the buffer in the connection is full (the
instance is not consuming
+// tuples fast enough).
void InstanceServer::StartBackPressureConnectionCb(Connection* _connection) {
// The connection will notify us when we can stop the back pressure
_connection->setCausedBackPressure();
@@ -568,6 +578,8 @@ void
InstanceServer::StartBackPressureConnectionCb(Connection* _connection) {
if (!stmgr_->DidAnnounceBackPressure()) {
stmgr_->SendStartBackPressureToOtherStMgrs();
+// Start backpressure from local instances metric
+back_pressure_metric_caused_by_local_instances_->Start();
}
// Indicate which instance component had back pressure
@@ -578,6 +590,7 @@ void
InstanceServer::StartBackPressureConnectionCb(Connection* _connection) {
StartBackPressureOnSpouts();
}
+// This function is called when the buffer in the connection is empty (the
tuples are drained).
void InstanceServer::StopBackPressureConnectionCb(Connection* _connection) {
_connection->unsetCausedBackPressure();
@@ -605,6 +618,8 @@ void
InstanceServer::StopBackPressureConnectionCb(Connection* _connection) {
if (!stmgr_->DidAnnounceBackPressure()) {
stmgr_->SendStopBackPressureToOtherStMgrs();
+// Stop backpressure from local instances metric
+back_pressure_metric_caused_by_local_instances_->Stop();
}
AttemptStopBackPressureFromSpouts();
}
diff --git a/heron/stmgr/src/cpp
[incubator-heron] branch master updated: udpated yaml and updated tools (#3360)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new b707c7b udpated yaml and updated tools (#3360)
b707c7b is described below
commit b707c7b09e56e2de10abd07929713a4049d9e5cb
Author: Jim Mantheiy, Jr
AuthorDate: Wed Oct 9 02:40:51 2019 -0400
udpated yaml and updated tools (#3360)
* udpated yaml and updated tools
* Update values.yaml.template
fixed based on feedback
* Update values.yaml.template
dropped the configs down to 5/15 based on feedback
---
deploy/kubernetes/helm/templates/tools.yaml | 2 +-
deploy/kubernetes/helm/values.yaml.template | 15 ---
2 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/deploy/kubernetes/helm/templates/tools.yaml
b/deploy/kubernetes/helm/templates/tools.yaml
index 91f4f06..72f92dd 100644
--- a/deploy/kubernetes/helm/templates/tools.yaml
+++ b/deploy/kubernetes/helm/templates/tools.yaml
@@ -111,7 +111,7 @@ spec:
- >-
heron-ui
--port=8889
- --base_url=/api/v1/namespaces/{{ .Release.Namespace
}}/services/{{ .Release.Name }}-ui:8889/proxy
+ --base_url={{ .Values.heron.url }}
- name: heron-apiserver
image: {{ .Values.image }}
imagePullPolicy: {{ .Values.imagePullPolicy }}
diff --git a/deploy/kubernetes/helm/values.yaml.template
b/deploy/kubernetes/helm/values.yaml.template
index e934867..37269e6 100644
--- a/deploy/kubernetes/helm/values.yaml.template
+++ b/deploy/kubernetes/helm/values.yaml.template
@@ -39,7 +39,8 @@ jobReplicas: 1
# amount of memory to provide for API server
apiServerMemory: 512M
-
+heron:
+ url:/api/v1/namespaces/{{ .Release.Namespace }}/services/{{ .Release.Name
}}-ui:8889/proxy
# Topologies uploader
uploader:
class: dlog # s3
@@ -53,18 +54,18 @@ uploader:
packing: RoundRobin # ResourceCompliantRR, FirstFitDecreasing
# Number of replicas for storage bookies, memory and storage requirements
-bookieReplicas: 1
+bookieReplicas: 3
bookieCpuMin: 100m
bookieCpuMax: 250m
-bookieHeapMemory: 256M
-bookieDirectMemory: 128M
+bookieHeapMemory: 512M
+bookieDirectMemory: 256M
bookieReadCacheSize: 32
bookieWriteCacheSize: 32
-bookieJournalCapacity: 1G
-bookieStorageCapacity: 3G
+bookieJournalCapacity: 5G
+bookieStorageCapacity: 15G
# Number of replicas for zookeeper
-zkReplicas: 1
+zkReplicas: 3
bookkeeper:
image: apache/bookkeeper:4.7.3
[incubator-heron] branch master updated: Hook up a packing plan endpoint in Heron UI (#3359)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new b697f83 Hook up a packing plan endpoint in Heron UI (#3359)
b697f83 is described below
commit b697f838e74843c81d757531aa19d7c4526211a6
Author: Ning Wang
AuthorDate: Tue Oct 8 09:01:44 2019 -0700
Hook up a packing plan endpoint in Heron UI (#3359)
* Hook up a packing plan endpoint in Heron UI
* add unit test
---
heron/tools/common/src/python/access/heron_api.py | 20 +
heron/tools/tracker/src/python/tracker.py | 2 +-
.../tools/tracker/tests/python/tracker_unittest.py | 19
heron/tools/ui/src/python/handlers/api/__init__.py | 1 +
heron/tools/ui/src/python/handlers/api/topology.py | 26 ++
heron/tools/ui/src/python/main.py | 2 ++
6 files changed, 69 insertions(+), 1 deletion(-)
diff --git a/heron/tools/common/src/python/access/heron_api.py
b/heron/tools/common/src/python/access/heron_api.py
index 1ab571c..709acb4 100644
--- a/heron/tools/common/src/python/access/heron_api.py
+++ b/heron/tools/common/src/python/access/heron_api.py
@@ -35,6 +35,7 @@ TOPOLOGIES_STATS_URL_FMT= "%s/states" %
TOPOLOGIES_URL_FMT
EXECUTION_STATE_URL_FMT = "%s/executionstate" % TOPOLOGIES_URL_FMT
LOGICALPLAN_URL_FMT = "%s/logicalplan"% TOPOLOGIES_URL_FMT
PHYSICALPLAN_URL_FMT= "%s/physicalplan" % TOPOLOGIES_URL_FMT
+PACKINGPLAN_URL_FMT = "%s/packingplan"% TOPOLOGIES_URL_FMT
SCHEDULER_LOCATION_URL_FMT = "%s/schedulerlocation" % TOPOLOGIES_URL_FMT
METRICS_URL_FMT = "%s/metrics"% TOPOLOGIES_URL_FMT
@@ -268,6 +269,25 @@ def get_instances(cluster, environ, topology, role=None):
@tornado.gen.coroutine
+def get_packing_plan(cluster, environ, topology, role=None):
+ '''
+ Get the packing plan state of a topology in a cluster from tracker
+ :param cluster:
+ :param environ:
+ :param topology:
+ :param role:
+ :return:
+ '''
+ params = dict(cluster=cluster, environ=environ, topology=topology)
+ if role is not None:
+params['role'] = role
+ request_url = tornado.httputil.url_concat(
+ create_url(PACKINGPLAN_URL_FMT), params)
+ raise tornado.gen.Return((yield fetch_url_as_json(request_url)))
+
+
+
+@tornado.gen.coroutine
def get_physical_plan(cluster, environ, topology, role=None):
'''
Get the physical plan state of a topology in a cluster from tracker
diff --git a/heron/tools/tracker/src/python/tracker.py
b/heron/tools/tracker/src/python/tracker.py
index 3ce814d..15df468 100644
--- a/heron/tools/tracker/src/python/tracker.py
+++ b/heron/tools/tracker/src/python/tracker.py
@@ -598,7 +598,7 @@ class Tracker(object):
packingPlan["id"] = topology.packing_plan.id
packingPlan["container_plans"] = containers
-return json.dumps(packingPlan)
+return packingPlan
def setTopologyInfo(self, topology):
"""
diff --git a/heron/tools/tracker/tests/python/tracker_unittest.py
b/heron/tools/tracker/tests/python/tracker_unittest.py
index 82afd64..136f5d6 100644
--- a/heron/tools/tracker/tests/python/tracker_unittest.py
+++ b/heron/tools/tracker/tests/python/tracker_unittest.py
@@ -250,3 +250,22 @@ class TrackerTest(unittest.TestCase):
{'topology.component.parallelism': '1'})
self.assertEqual(pplan['instances'], {})
self.assertEqual(pplan['stmgrs'], {})
+
+ def test_extract_packing_plan(self):
+# Create topology
+pb_pplan = MockProto().create_mock_simple_packing_plan()
+topology = Topology('topology_name', 'ExclamationTopology')
+topology.set_packing_plan(pb_pplan)
+# Extract packing plan
+packing_plan = self.tracker.extract_packing_plan(topology)
+self.assertEqual(packing_plan['id'], 'ExclamationTopology')
+self.assertEqual(packing_plan['container_plans'][0]['id'], 1)
+self.assertEqual(packing_plan['container_plans'][0]['required_resources'],
+ {'disk': 2048L, 'ram': 1024L, 'cpu': 1.0})
+self.assertEqual(packing_plan['container_plans'][0]['instances'][0],
+ {
+ 'component_index': 1,
+'component_name': u'word',
+'instance_resources': {'cpu': 1.0, 'disk': 2048L,
'ram': 1024L},
+'task_id': 1
+ })
diff --git a/heron/tools/ui/src/python/handlers/api/__init__.py
b/heron/tools/ui/src/python/handlers/api/__init__.py
index c9e17e0..e36109d 100644
--- a/he
[incubator-heron] branch master updated: Add a delay for showing tooltip, format visstyle css file (#3358)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 28dc357 Add a delay for showing tooltip, format visstyle css file
(#3358)
28dc357 is described below
commit 28dc357f3dc17884f820b1dfaf53d92f7e5fdc7b
Author: Ning Wang
AuthorDate: Fri Oct 4 23:06:03 2019 -0700
Add a delay for showing tooltip, format visstyle css file (#3358)
---
heron/tools/ui/resources/static/css/visstyle.css | 105 ---
1 file changed, 53 insertions(+), 52 deletions(-)
diff --git a/heron/tools/ui/resources/static/css/visstyle.css
b/heron/tools/ui/resources/static/css/visstyle.css
index 3d48880..34bee52 100644
--- a/heron/tools/ui/resources/static/css/visstyle.css
+++ b/heron/tools/ui/resources/static/css/visstyle.css
@@ -139,6 +139,7 @@ a:active, a:focus {
.d3-tip {
max-width: 400px;
+ transition-delay: 0.25s;
}
path.regular, rect.regular {
@@ -241,19 +242,19 @@ tr.highlighted {
}
.topotable>thead>tr>th, .topotable>tbody>tr>th, .topotable>tfoot>tr>th,
.topotable>thead>tr>td, .topotable>tbody>tr>td, .topotable>tfoot>tr>td {
-font-size: 14px;
-padding: 8px;
-line-height: 1.42857143;
-vertical-align: top;
-border-top: 1px solid #dd;
+ font-size: 14px;
+ padding: 8px;
+ line-height: 1.42857143;
+ vertical-align: top;
+ border-top: 1px solid #dd;
}
.break-all {
-word-break: break-all;
+ word-break: break-all;
}
.no-break {
-white-space: nowrap
+ white-space: nowrap
}
.strong {
@@ -261,16 +262,16 @@ white-space: nowrap
}
.topotable .topobutton {
-padding: 2px;
+ padding: 2px;
}
.topotable .topobutton a {
-margin-left: 8px;
-visibility: hidden;
+ margin-left: 8px;
+ visibility: hidden;
}
.topotable tr:hover .topobutton a {
-visibility: visible;
+ visibility: visible;
}
.vcenter {
display: inline-block;
@@ -279,76 +280,76 @@ visibility: visible;
}
.action-button-group {
-position: absolute;
-right: 10px;
-top: 2px;
+ position: absolute;
+ right: 10px;
+ top: 2px;
}
.sort.asc {
-background-color: lightgrey;
+ background-color: lightgrey;
}
.sort.desc {
-background-color: lightgrey;
+ background-color: lightgrey;
}
/* Sort arrows */
.sort:after {
-width: 0;
-height: 0;
-border-left: 5px solid transparent;
-border-right: 5px solid transparent;
-border-bottom: 5px solid transparent;
-border-top: 5px solid transparent;
-content:"";
-position: relative;
-top:-10px;
-right:-5px;
+ width: 0;
+ height: 0;
+ border-left: 5px solid transparent;
+ border-right: 5px solid transparent;
+ border-bottom: 5px solid transparent;
+ border-top: 5px solid transparent;
+ content:"";
+ position: relative;
+ top:-10px;
+ right:-5px;
}
.sort.asc:after {
-width: 0;
-height: 0;
-border-left: 5px solid transparent;
-border-right: 5px solid transparent;
-border-bottom: 5px solid transparent;
-border-top: 5px solid black;
-content:"";
-position: relative;
-top:10px;
-right:-5px;
+ width: 0;
+ height: 0;
+ border-left: 5px solid transparent;
+ border-right: 5px solid transparent;
+ border-bottom: 5px solid transparent;
+ border-top: 5px solid black;
+ content:"";
+ position: relative;
+ top:10px;
+ right:-5px;
}
.sort.desc:after {
-width: 0;
-height: 0;
-border-left: 5px solid transparent;
-border-right: 5px solid transparent;
-border-bottom: 5px solid black;
-border-top: 5px solid transparent;
-content:"";
-position: relative;
-top:-10px;
-right:-5px;
+ width: 0;
+ height: 0;
+ border-left: 5px solid transparent;
+ border-right: 5px solid transparent;
+ border-bottom: 5px solid black;
+ border-top: 5px solid transparent;
+ content:"";
+ position: relative;
+ top:-10px;
+ right:-5px;
}
.search-result {
-color: grey;
-font-size: 14px;
-line-height: 30px;
+ color: grey;
+ font-size: 14px;
+ line-height: 30px;
}
/* When the topology is dead */
.dead {
-color: red;
+ color: red;
}
/* When the topology is weird state */
.weird {
-color: orange;
+ color: orange;
}
/* When the topology is starting */
.starting {
-color: green;
+ color: green;
}
[incubator-heron] branch master updated: Remove website2 and CONTRIBUTING.md from release archive files (#3357)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new b71572d Remove website2 and CONTRIBUTING.md from release archive files (#3357) b71572d is described below commit b71572dc0c8b1f3785299bd02e5d160bf43d8d8a Author: Ning Wang AuthorDate: Fri Oct 4 23:04:06 2019 -0700 Remove website2 and CONTRIBUTING.md from release archive files (#3357) --- .gitattributes | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitattributes b/.gitattributes index 65d1ff7..157f783 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,4 +1,6 @@ # Files and directories with the attribute export-ignore won’t be added to # archive files. See http://git-scm.com/docs/gitattributes for details. /.gitattributes export-ignore +/CONTRIBUTING.md export-ignore /website/ export-ignore +/website2/export-ignore
[incubator-heron] branch master updated: Add a cap to metrics communicator in TMasterSink and MetricsCacheSink (#3355)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 5a3dee5 Add a cap to metrics communicator in TMasterSink and
MetricsCacheSink (#3355)
5a3dee5 is described below
commit 5a3dee54adf60b706a8af1a800c7ec75d1042ea0
Author: Ning Wang
AuthorDate: Fri Oct 4 15:44:20 2019 -0700
Add a cap to metrics communicator in TMasterSink and MetricsCacheSink
(#3355)
* Add a cap to metrics communicator in TMasterSink and MetricsCacheSink
* refactor while loop
---
.../sink/metricscache/MetricsCacheClient.java | 13 +--
.../sink/metricscache/MetricsCacheSink.java| 19 +++
.../metricsmgr/sink/tmaster/TMasterClient.java | 12 --
.../heron/metricsmgr/sink/tmaster/TMasterSink.java | 17 ++
.../sink/metricscache/MetricsCacheSinkTest.java| 27 ++
.../metricsmgr/sink/tmaster/TMasterSinkTest.java | 27 ++
6 files changed, 111 insertions(+), 4 deletions(-)
diff --git
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/metricscache/MetricsCacheClient.java
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/metricscache/MetricsCacheClient.java
index b95ef2e..5e0a3e9 100644
---
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/metricscache/MetricsCacheClient.java
+++
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/metricscache/MetricsCacheClient.java
@@ -93,12 +93,21 @@ public class MetricsCacheClient extends HeronClient
implements Runnable {
Runnable task = new Runnable() {
@Override
public void run() {
-while (!publishMetricsCommunicator.isEmpty()) {
- TopologyMaster.PublishMetrics publishMetrics =
publishMetricsCommunicator.poll();
+TopologyMaster.PublishMetrics publishMetrics;
+while (true) {
+ synchronized (publishMetricsCommunicator) {
+publishMetrics = publishMetricsCommunicator.poll();
+ }
+ if (publishMetrics == null) {
+break; // No metrics left
+ }
+
LOG.info(String.format("%d Metrics, %d Exceptions to send to
MetricsCache",
publishMetrics.getMetricsCount(),
publishMetrics.getExceptionsCount()));
LOG.fine("Publish Metrics sending to MetricsCache: " +
publishMetrics.toString());
+
sendMessage(publishMetrics);
+
}
}
};
diff --git
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/metricscache/MetricsCacheSink.java
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/metricscache/MetricsCacheSink.java
index 4d2357f..0f049d1 100644
---
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/metricscache/MetricsCacheSink.java
+++
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/metricscache/MetricsCacheSink.java
@@ -83,6 +83,8 @@ import org.apache.heron.spi.metricsmgr.sink.SinkContext;
public class MetricsCacheSink implements IMetricsSink {
private static final Logger LOG =
Logger.getLogger(MetricsCacheSink.class.getName());
+ private static final int MAX_COMMUNICATOR_SIZE = 128;
+
// These configs would be read from metrics-sink-configs.yaml
private static final String KEY_TMASTER_LOCATION_CHECK_INTERVAL_SEC =
"metricscache-location-check-interval-sec";
@@ -232,10 +234,27 @@ public class MetricsCacheSink implements IMetricsSink {
metricsCommunicator.offer(publishMetrics.build());
+
+
// Update metrics
sinkContext.exportCountMetric(RECORD_PROCESS_COUNT, 1);
sinkContext.exportCountMetric(METRICS_COUNT,
publishMetrics.getMetricsCount());
sinkContext.exportCountMetric(EXCEPTIONS_COUNT,
publishMetrics.getExceptionsCount());
+
+checkCommunicator(metricsCommunicator, MAX_COMMUNICATOR_SIZE);
+ }
+
+ // Check if the communicator is full/overflow. Poll and drop extra elements
that
+ // are over the queue limit from the head.
+ public static void
checkCommunicator(Communicator communicator,
+int maxSize) {
+synchronized (communicator) {
+ int size = communicator.size();
+
+ for (int i = 0; i < size - maxSize; ++i) {
+communicator.poll();
+ }
+}
}
@Override
diff --git
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/tmaster/TMasterClient.java
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/tmaster/TMasterClient.java
index 7c7354f..48e0a32 100644
---
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/tmaster/TMasterClient.java
+++
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/tmaster/TMasterClient.java
@@ -88,11 +88,19 @@ public class TMasterClient extends HeronClient implements
Runnable {
Runnable task =
[incubator-heron] branch master updated: Separate gc log files for metrics manage and checkpoint manager by id (#3354)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 9c7cd21 Separate gc log files for metrics manage and checkpoint manager by id (#3354) 9c7cd21 is described below commit 9c7cd219770d648513cc11c776ef02e55a0b2b8c Author: Ning Wang AuthorDate: Thu Oct 3 12:23:14 2019 -0700 Separate gc log files for metrics manage and checkpoint manager by id (#3354) * Separate gc log files for metrics manage and checkpoint manager by id * Fix unit tests --- heron/executor/src/python/heron_executor.py| 7 --- heron/executor/tests/python/heron_executor_unittest.py | 4 ++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/heron/executor/src/python/heron_executor.py b/heron/executor/src/python/heron_executor.py index 25bec6f..01f59a9 100755 --- a/heron/executor/src/python/heron_executor.py +++ b/heron/executor/src/python/heron_executor.py @@ -446,7 +446,7 @@ class HeronExecutor(object): '-XX:+HeapDumpOnOutOfMemoryError', '-XX:+UseConcMarkSweepGC', '-XX:+PrintCommandLineFlags', - '-Xloggc:log-files/gc.metricsmgr.log', + '-Xloggc:log-files/gc.' + metricsManagerId + '.log', '-Djava.net.preferIPv4Stack=true', '-cp', self.metrics_manager_classpath, @@ -845,6 +845,7 @@ class HeronExecutor(object): ckptmgr_main_class = 'org.apache.heron.ckptmgr.CheckpointManager' ckptmgr_ram_mb = self.checkpoint_manager_ram / (1024 * 1024) +ckptmgr_id = self.ckptmgr_ids[self.shard] ckptmgr_cmd = [os.path.join(self.heron_java_home, "bin/java"), '-Xms%dM' % ckptmgr_ram_mb, '-Xmx%dM' % ckptmgr_ram_mb, @@ -863,14 +864,14 @@ class HeronExecutor(object): '-XX:+HeapDumpOnOutOfMemoryError', '-XX:+UseConcMarkSweepGC', '-XX:+UseConcMarkSweepGC', - '-Xloggc:log-files/gc.ckptmgr.log', + '-Xloggc:log-files/gc.' + ckptmgr_id + '.log', '-Djava.net.preferIPv4Stack=true', '-cp', self.checkpoint_manager_classpath, ckptmgr_main_class, '-t' + self.topology_name, '-i' + self.topology_id, - '-c' + self.ckptmgr_ids[self.shard], + '-c' + ckptmgr_id, '-p' + self.checkpoint_manager_port, '-f' + self.stateful_config_file, '-o' + self.override_config_file, diff --git a/heron/executor/tests/python/heron_executor_unittest.py b/heron/executor/tests/python/heron_executor_unittest.py index d1debb0..7fbd800 100644 --- a/heron/executor/tests/python/heron_executor_unittest.py +++ b/heron/executor/tests/python/heron_executor_unittest.py @@ -109,12 +109,12 @@ class HeronExecutorTest(unittest.TestCase): "-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=100M " \ "-XX:+PrintPromotionFailure -XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC " \ "-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+PrintCommandLineFlags " \ - "-Xloggc:log-files/gc.metricsmgr.log -Djava.net.preferIPv4Stack=true " \ + "-Xloggc:log-files/gc.metricsmgr-%d.log -Djava.net.preferIPv4Stack=true " \ "-cp metricsmgr_classpath org.apache.heron.metricsmgr.MetricsManager " \ "--id=metricsmgr-%d --port=metricsmgr_port " \ "--topology=topname --cluster=cluster --role=role --environment=environ --topology-id=topid " \ "--system-config-file=%s --override-config-file=%s --sink-config-file=metrics_sinks_config_file" %\ - (container_id, INTERNAL_CONF_PATH, OVERRIDE_PATH) + (container_id, container_id, INTERNAL_CONF_PATH, OVERRIDE_PATH) def get_expected_metricscachemgr_command(): return "heron_java_home/bin/java -Xmx1024M -XX:+PrintCommandLineFlags -verbosegc " \
[incubator-heron] branch master updated: Make unknown arguments a warning instead of an error in heron executor (#3353)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 0876b56 Make unknown arguments a warning instead of an error in heron
executor (#3353)
0876b56 is described below
commit 0876b56d49f06e587bd14e41c54ef730b35221eb
Author: Ning Wang
AuthorDate: Wed Oct 2 16:55:27 2019 -0700
Make unknown arguments a warning instead of an error in heron executor
(#3353)
---
heron/executor/src/python/heron_executor.py | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/heron/executor/src/python/heron_executor.py
b/heron/executor/src/python/heron_executor.py
index 9380ef5..25bec6f 100755
--- a/heron/executor/src/python/heron_executor.py
+++ b/heron/executor/src/python/heron_executor.py
@@ -376,9 +376,8 @@ class HeronExecutor(object):
parsed_args, unknown_args = parser.parse_known_args(args[1:])
if unknown_args:
- Log.error('Unknown argument: %s' % unknown_args[0])
- parser.print_help()
- sys.exit(1)
+ Log.warn('Unknown arguments found!!! They are: %s' % unknown_args)
+ Log.warn(parser.format_help())
return parsed_args
[incubator-heron] branch master updated: add parallelism in component, fix stream tooltip offset (#3352)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 2d2d7cf add parallelism in component, fix stream tooltip offset
(#3352)
2d2d7cf is described below
commit 2d2d7cfd336d0b94788fb3d91270c052d46c89ef
Author: Ning Wang
AuthorDate: Tue Oct 1 17:55:27 2019 -0700
add parallelism in component, fix stream tooltip offset (#3352)
---
heron/tools/ui/resources/static/css/visstyle.css | 13 --
heron/tools/ui/resources/static/js/logical-plan.js | 50 ++
2 files changed, 41 insertions(+), 22 deletions(-)
diff --git a/heron/tools/ui/resources/static/css/visstyle.css
b/heron/tools/ui/resources/static/css/visstyle.css
index 6ad85ad..3d48880 100644
--- a/heron/tools/ui/resources/static/css/visstyle.css
+++ b/heron/tools/ui/resources/static/css/visstyle.css
@@ -142,19 +142,19 @@ a:active, a:focus {
}
path.regular, rect.regular {
-opacity: 1;
+ opacity: 1;
}
path.fade, rect.fade {
-opacity: 0.2;
+ opacity: 0.2;
}
circle.regular {
-opacity: 1;
+ opacity: 1;
}
circle.fade {
-opacity: 0.3;
+ opacity: 0.3;
}
text.regular {
@@ -165,6 +165,11 @@ text.fade {
display: none;
}
+text.fade.fade-half {
+ display: block;
+ opacity: 0.5;
+}
+
circle.background {
opacity: 1;
}
diff --git a/heron/tools/ui/resources/static/js/logical-plan.js
b/heron/tools/ui/resources/static/js/logical-plan.js
index aa68ff2..ac300c9 100644
--- a/heron/tools/ui/resources/static/js/logical-plan.js
+++ b/heron/tools/ui/resources/static/js/logical-plan.js
@@ -256,7 +256,8 @@
// create the spout array
for (var i in topology.spouts) {
spoutsArr.push({
-"name": i
+"name": i,
+"parallelism":
topology.spouts[i]["config"]["topology.component.parallelism"]
});
}
@@ -264,6 +265,7 @@
for (var i in topology.bolts) {
boltsArr.push({
"name": i,
+ "parallelism":
topology.bolts[i]["config"]["topology.component.parallelism"],
"inputComponents": topology.bolts[i]["inputComponents"],
"inputStreams": topology.bolts[i]["inputs"]
});
@@ -346,10 +348,12 @@
var connection_tip = d3.tip()
.attr('class', 'd3-tip main text-center connection')
-.offset([10, 0])
+.offset(function () {
+ return [10 - this.getBBox().height / 2, 0];
+})
.direction('s')
-.html(function (d) {
- return d.streams;
+.html(function (edge) {
+ return edge.streams;
});
var node = svg.selectAll(".topnode")
@@ -385,10 +389,15 @@
});
// Component
-node.append("circle")
+var g = node.append("g")
+.attr("transform", function(d){return "translate("+d.x+","+d.y+")"})
+.on("click", planController.logicalComponentClicked)
+.on("dblclick", planController.logicalComponentClicked)
+.on("mouseover", planController.logicalComponentHoverOver)
+.on("mouseout", planController.logicalComponentHoverOut);
+
+g.append("circle")
.attr('class', 'background')
-.attr("cx", function (d) { return d.x; })
-.attr("cy", function (d) { return d.y; })
.attr("r", function (d) {
if (d.isReal) {
return d.r = 17;
@@ -397,10 +406,8 @@
})
.style('fill', 'white');
-node.append("circle")
+g.append("circle")
.attr("class", "node")
-.attr("cx", function (d) { return d.cx = d.x; })
-.attr("cy", function (d) { return d.cy = d.y; })
.attr("r", function (d) {
if (d.isReal) {
return d.r = 15;
@@ -413,17 +420,24 @@
d.defaultColor = color(d.name);
d.color = d.color || d.defaultColor;
return d.color;
-})
-.on("click", planController.logicalComponentClicked)
-.on("dblclick", planController.logicalComponentClicked)
-.on("mouseover", planController.logicalComponentHoverOver)
-.on("mouseout", planController.logicalComponentHoverOut);
+});
+
+// Component parallelism, always visible
+g.append("text")
+.attr("id", function(d) { return "parallelism+" + d.name; })
+.attr("y", function (d) { return 4; })
+.attr("class", "fade fade-half")
+.style("text-anchor", "middle")
+
[incubator-heron] branch master updated: Add metrics for communicator size in metrics manager (#3351)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 98e35ae Add metrics for communicator size in metrics manager (#3351)
98e35ae is described below
commit 98e35ae59a71910c6baa9f28a99db8e3d4386796
Author: Ning Wang
AuthorDate: Tue Oct 1 11:40:20 2019 -0700
Add metrics for communicator size in metrics manager (#3351)
---
.../apache/heron/metricsmgr/MetricsManager.java| 11 +++---
.../heron/metricsmgr/MetricsManagerServer.java | 42 --
.../heron/metricsmgr/MetricsManagerServerTest.java | 13 ---
3 files changed, 44 insertions(+), 22 deletions(-)
diff --git
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsManager.java
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsManager.java
index 8d4ab5e..076a504 100644
--- a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsManager.java
+++ b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsManager.java
@@ -199,7 +199,7 @@ public class MetricsManager {
: TypeUtils.getInteger(restartAttempts));
// Update the list of Communicator in Metrics Manager Server
- metricsManagerServer.addSinkCommunicator(sinkExecutor.getCommunicator());
+ metricsManagerServer.addSinkCommunicator(sinkId,
sinkExecutor.getCommunicator());
}
}
@@ -537,12 +537,11 @@ public class MetricsManager {
// If the thread name is a key of SinkExecutors, then it is a thread
running IMetricsSink
if (sinkExecutors.containsKey(thread.getName())) {
sinkId = thread.getName();
-// Remove the old sink executor
-SinkExecutor oldSinkExecutor = sinkExecutors.remove(sinkId);
// Remove the unneeded Communicator bind with Metrics Manager Server
-
metricsManagerServer.removeSinkCommunicator(oldSinkExecutor.getCommunicator());
+metricsManagerServer.removeSinkCommunicator(sinkId);
-// Close the sink
+// Remove the old sink executor and close the sink
+SinkExecutor oldSinkExecutor = sinkExecutors.remove(sinkId);
SysUtils.closeIgnoringExceptions(oldSinkExecutor);
thisSinkRetryAttempts = sinksRetryAttempts.remove(sinkId);
@@ -565,7 +564,7 @@ public class MetricsManager {
sinksRetryAttempts.put(sinkId, thisSinkRetryAttempts);
// Update the list of Communicator in Metrics Manager Server
-
metricsManagerServer.addSinkCommunicator(newSinkExecutor.getCommunicator());
+metricsManagerServer.addSinkCommunicator(sinkId,
newSinkExecutor.getCommunicator());
// Restart it
executors.execute(newSinkExecutor);
diff --git
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsManagerServer.java
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsManagerServer.java
index a9ff54f..8b9e6ab 100644
---
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsManagerServer.java
+++
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/MetricsManagerServer.java
@@ -22,10 +22,11 @@ package org.apache.heron.metricsmgr;
import java.net.SocketAddress;
import java.nio.channels.SocketChannel;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashMap;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
-import java.util.concurrent.CopyOnWriteArrayList;
import java.util.logging.Level;
import java.util.logging.Logger;
@@ -61,8 +62,10 @@ public class MetricsManagerServer extends HeronServer {
private static final String SERVER_EXCEPTIONS_RECEIVED =
"exceptions-received";
private static final String SERVER_NEW_TMASTER_LOCATION =
"new-tmaster-location";
private static final String SERVER_TMASTER_LOCATION_RECEIVED =
"tmaster-location-received";
+ private static final String SERVER_COMMUNICATOR_OFFER = "communicator-offer";
+ private static final String SERVER_COMMUNICATOR_SIZE = "communicator-size";
- private final List> metricsSinkCommunicators;
+ private final Map>
metricsSinkCommunicators;
// A map from MetricPublisher's immutable SocketAddress to the
MetricPublisher
// We would fetch SocketAddress by using
SocketChannel.socket().getRemoteSocketAddress,
@@ -96,7 +99,7 @@ public class MetricsManagerServer extends HeronServer {
// Since we might mutate the list while iterating it
// Consider that the iteration vastly outnumbers mutation,
// it would barely hurt any performance
-this.metricsSinkCommunicators = new
CopyOnWriteArrayList>();
+this.metricsSinkCommunicators = Collections.synchronizedMap(new
HashMap<>());
this.publisherMap = new HashMap();
@@ -119,14 +122,24 @@ public class MetricsManagerServer extends HeronServer {
regist
[incubator-heron] branch master updated: Clean up rate limit config and add an example topology (#3350)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 381076b Clean up rate limit config and add an example topology (#3350)
381076b is described below
commit 381076b9df0756ee976a808bed7a97be0764082a
Author: Ning Wang
AuthorDate: Tue Oct 1 09:10:51 2019 -0700
Clean up rate limit config and add an example topology (#3350)
---
.../examples/api/ComponentConfigTopology.java | 120 +
.../api/src/java/org/apache/heron/api/Config.java | 30 +-
2 files changed, 145 insertions(+), 5 deletions(-)
diff --git
a/examples/src/java/org/apache/heron/examples/api/ComponentConfigTopology.java
b/examples/src/java/org/apache/heron/examples/api/ComponentConfigTopology.java
new file mode 100644
index 000..132dc8f
--- /dev/null
+++
b/examples/src/java/org/apache/heron/examples/api/ComponentConfigTopology.java
@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.heron.examples.api;
+
+import java.util.Map;
+
+import org.apache.heron.api.Config;
+import org.apache.heron.api.HeronSubmitter;
+import org.apache.heron.api.bolt.BaseRichBolt;
+import org.apache.heron.api.bolt.OutputCollector;
+import org.apache.heron.api.metric.GlobalMetrics;
+import org.apache.heron.api.topology.OutputFieldsDeclarer;
+import org.apache.heron.api.topology.TopologyBuilder;
+import org.apache.heron.api.topology.TopologyContext;
+import org.apache.heron.api.tuple.Tuple;
+import org.apache.heron.api.utils.Utils;
+import org.apache.heron.common.basics.ByteAmount;
+import org.apache.heron.examples.api.spout.TestWordSpout;
+import org.apache.heron.simulator.Simulator;
+
+
+/**
+ * This is a basic example of a Storm topology.
+ */
+public final class ComponentConfigTopology {
+
+ private ComponentConfigTopology() {
+ }
+
+ public static void main(String[] args) throws Exception {
+TopologyBuilder builder = new TopologyBuilder();
+
+builder.setSpout("word", new TestWordSpout(), 2)
+// Set rate limit to 1000 bytes per second (since parallelism is set
to 2,
+// each instance is rate limited as 500 bps).
+.addConfiguration(Config.TOPOLOGY_COMPONENT_OUTPUT_BPS, 1000);
+builder.setBolt("exclaim1", new ExclamationBolt(), 2)
+.shuffleGrouping("word");
+
+Config conf = new Config();
+conf.setDebug(true);
+conf.setMaxSpoutPending(10);
+
+// component resource configuration
+conf.setComponentCpu("word", 0.5);
+conf.setComponentRam("word", ByteAmount.fromMegabytes(512));
+conf.setComponentDisk("word", ByteAmount.fromMegabytes(512));
+conf.setComponentCpu("exclaim1", 0.5);
+conf.setComponentRam("exclaim1", ByteAmount.fromMegabytes(512));
+conf.setComponentDisk("exclaim1", ByteAmount.fromMegabytes(512));
+
+// container resource configuration
+conf.setContainerDiskRequested(ByteAmount.fromGigabytes(2));
+conf.setContainerRamRequested(ByteAmount.fromGigabytes(3));
+conf.setContainerCpuRequested(2);
+
+// Specify the size of RAM padding to per container.
+// Notice, this config will be considered as a hint,
+// and it's up to the packing algorithm to determine whether to apply this
hint
+conf.setContainerRamPadding(ByteAmount.fromGigabytes(2));
+
+if (args != null && args.length > 0) {
+ conf.setNumStmgrs(2);
+ HeronSubmitter.submitTopology(args[0], conf, builder.createTopology());
+} else {
+ Simulator simulator = new Simulator();
+ simulator.submitTopology("test", conf, builder.createTopology());
+ Utils.sleep(1);
+ simulator.killTopology("test");
+ simulator.shutdown();
+}
+ }
+
+ public static class ExclamationBolt extends BaseRichBolt {
+private static final long serialVersionUID = 2165326630789117557L;
+private long nItems;
+private long startTime;
+
+@Override
+@SuppressWarnings("
[incubator-heron] branch master updated: Build kafka bolt maven artifacts (#3341)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 7c04f04 Build kafka bolt maven artifacts (#3341)
7c04f04 is described below
commit 7c04f04f2fa6970f29ddfa2a28f326a0c152a375
Author: SiMing Weng
AuthorDate: Sat Sep 28 23:05:35 2019 -0400
Build kafka bolt maven artifacts (#3341)
* update KafkaSpout
* Revert "update KafkaSpout"
This reverts commit dc64e9ad
* build kafka bolt maven artifacts
---
...eron-kafka-spout.template.pom => heron-kafka.template.pom} | 0
release/maven/maven-pom-version.sh| 11 +--
scripts/ci/build_maven_artifacts.sh | 8
3 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/release/maven/heron-kafka-spout.template.pom
b/release/maven/heron-kafka.template.pom
similarity index 100%
rename from release/maven/heron-kafka-spout.template.pom
rename to release/maven/heron-kafka.template.pom
diff --git a/release/maven/maven-pom-version.sh
b/release/maven/maven-pom-version.sh
index a096dfc..b9e8901 100755
--- a/release/maven/maven-pom-version.sh
+++ b/release/maven/maven-pom-version.sh
@@ -51,9 +51,16 @@ cat ./maven/heron-with-kryo.template.pom | \
sed "s/DESCRIPTION/Heron Simulator/g" \
>> ./heron-simulator-$1.pom
-cat ./maven/heron-kafka-spout.template.pom | \
+cat ./maven/heron-kafka.template.pom | \
sed "s/VERSION/$1/g" | \
sed "s/ARTIFACT_ID/heron-kafka-spout/g" | \
sed "s/NAME/heron-kafka-spout/g" | \
sed "s/DESCRIPTION/Heron Kafka Spout/g" \
->> ./heron-kafka-spout-$1.pom
\ No newline at end of file
+>> ./heron-kafka-spout-$1.pom
+
+cat ./maven/heron-kafka.template.pom | \
+sed "s/VERSION/$1/g" | \
+sed "s/ARTIFACT_ID/heron-kafka-bolt/g" | \
+sed "s/NAME/heron-kafka-bolt/g" | \
+sed "s/DESCRIPTION/Heron Kafka Bolt/g" \
+>> ./heron-kafka-bolt-$1.pom
diff --git a/scripts/ci/build_maven_artifacts.sh
b/scripts/ci/build_maven_artifacts.sh
index 7a3441a..01b8b9d 100644
--- a/scripts/ci/build_maven_artifacts.sh
+++ b/scripts/ci/build_maven_artifacts.sh
@@ -44,6 +44,7 @@ $HOME/bin/bazel build heron/spi/src/java:all
$HOME/bin/bazel build heron/simulator/src/java:all
$HOME/bin/bazel build storm-compatibility/src/java:all
$HOME/bin/bazel build contrib/spouts/kafka/src/java:all
+$HOME/bin/bazel build contrib/bolts/kafka/src/java:all
echo "Found Version Tag $VERSION_TAG"
mkdir -p artifacts/$VERSION_TAG
@@ -64,6 +65,7 @@ mkdir -p $OUTPUT_DIR/$VERSION_TAG/heron-spi
mkdir -p $OUTPUT_DIR/$VERSION_TAG/heron-simulator
mkdir -p $OUTPUT_DIR/$VERSION_TAG/heron-storm
mkdir -p $OUTPUT_DIR/$VERSION_TAG/heron-kafka-spout
+mkdir -p $OUTPUT_DIR/$VERSION_TAG/heron-kafka-bolt
echo "Copy heron-api artifacts ... "
cp ./release/heron-api-$VERSION_TAG.pom $OUTPUT_DIR/$VERSION_TAG/heron-api/
@@ -95,6 +97,12 @@ cp
./bazel-bin/contrib/spouts/kafka/src/java/libheron-kafka-spout-java.jar ./$OU
cp ./bazel-bin/contrib/spouts/kafka/src/java/heron-kafka-spout-javadoc.zip
./$OUTPUT_DIR/$VERSION_TAG/heron-kafka-spout/heron-kafka-spout-$VERSION_TAG-javadoc.jar
cp ./bazel-bin/contrib/spouts/kafka/src/java/libheron-kafka-spout-java-src.jar
./$OUTPUT_DIR/$VERSION_TAG/heron-kafka-spout/heron-kafka-spout-$VERSION_TAG-sources.jar
+echo "Copy heron-kafka-spout artifacts ... "
+cp ./release/heron-kafka-bolt-$VERSION_TAG.pom
./$OUTPUT_DIR/$VERSION_TAG/heron-kafka-bolt/
+cp ./bazel-bin/contrib/bolts/kafka/src/java/libheron-kafka-bolt-java.jar
./$OUTPUT_DIR/$VERSION_TAG/heron-kafka-bolt/heron-kafka-bolt-$VERSION_TAG.jar
+cp ./bazel-bin/contrib/bolts/kafka/src/java/heron-kafka-bolt-javadoc.zip
./$OUTPUT_DIR/$VERSION_TAG/heron-kafka-bolt/heron-kafka-bolt-$VERSION_TAG-javadoc.jar
+cp ./bazel-bin/contrib/bolts/kafka/src/java/libheron-kafka-bolt-java-src.jar
./$OUTPUT_DIR/$VERSION_TAG/heron-kafka-bolt/heron-kafka-bolt-$VERSION_TAG-sources.jar
+
echo "Compress all artifacts into a bundle file ..."
tar -czf "heron-artifacts-${VERSION_TAG}.tar.gz" $OUTPUT_DIR
mv "heron-artifacts-${VERSION_TAG}.tar.gz" $OUTPUT_DIR
[incubator-heron] branch master updated: Upgrade zookeeper libraries to 3.4.14 (#3339)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new dd6da67 Upgrade zookeeper libraries to 3.4.14 (#3339) dd6da67 is described below commit dd6da67da7972791639746dd9190fd919ba6986c Author: Ning Wang AuthorDate: Wed Sep 18 00:00:25 2019 -0700 Upgrade zookeeper libraries to 3.4.14 (#3339) --- WORKSPACE | 8 third_party/zookeeper/zookeeper.BUILD | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/WORKSPACE b/WORKSPACE index 872c975..bfb17d3 100644 --- a/WORKSPACE +++ b/WORKSPACE @@ -429,7 +429,7 @@ maven_jar( maven_jar( name = "org_apache_zookeeper_zookeeper", - artifact = "org.apache.zookeeper:zookeeper:3.4.6", + artifact = "org.apache.zookeeper:zookeeper:3.4.14", ) maven_jar( @@ -872,10 +872,10 @@ http_archive( http_archive( name = "org_apache_zookeeper", -urls = ["http://archive.apache.org/dist/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz;], -strip_prefix = "zookeeper-3.4.10", +urls = ["http://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz;], +strip_prefix = "zookeeper-3.4.14", build_file = "@//:third_party/zookeeper/zookeeper.BUILD", -sha256 = "7f7f5414e044ac11fee2a1e0bc225469f51fb0cdf821e67df762a43098223f27", +sha256 = "b14f7a0fece8bd34c7fffa46039e563ac5367607c612517aa7bd37306afbd1cd", ) http_archive( diff --git a/third_party/zookeeper/zookeeper.BUILD b/third_party/zookeeper/zookeeper.BUILD index bf287c5..b59fdb1 100644 --- a/third_party/zookeeper/zookeeper.BUILD +++ b/third_party/zookeeper/zookeeper.BUILD @@ -26,7 +26,7 @@ genrule( "export TMP_DIR=$$(mktemp -d -t zookeeper.X)", "mkdir -p $$TMP_DIR", "cp -R $$(pwd)/external/org_apache_zookeeper/* $$TMP_DIR", -"cd $$TMP_DIR/src/c", +"cd $$TMP_DIR/zookeeper-client/zookeeper-client-c", "./configure --prefix=$$INSTALL_DIR --enable-shared=no", "make install", "rm -rf $$TMP_DIR",
[incubator-heron] branch master updated: Add SimulatorRunner for Streamlet API in Simulator package (#3336)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new eec8be6 Add SimulatorRunner for Streamlet API in Simulator package
(#3336)
eec8be6 is described below
commit eec8be6eaf20c85ed9b1a1801fc02a7ffaeb413b
Author: Ning Wang
AuthorDate: Wed Sep 4 00:53:59 2019 -0700
Add SimulatorRunner for Streamlet API in Simulator package (#3336)
* Add SimulatorRunner in Streamlet API
* remove unused imports
---
heron/simulator/src/java/BUILD | 2 +-
.../apache/heron/simulator/SimulatorRunner.java| 49 ++
2 files changed, 50 insertions(+), 1 deletion(-)
diff --git a/heron/simulator/src/java/BUILD b/heron/simulator/src/java/BUILD
index 232e170..84569b3 100644
--- a/heron/simulator/src/java/BUILD
+++ b/heron/simulator/src/java/BUILD
@@ -14,7 +14,7 @@ java_doc(
simulator_deps_files = \
heron_java_proto_files() + [
-"//heron/api/src/java:api-java-low-level",
+"//heron/api/src/java:api-java-low-level-functional",
"//heron/common/src/java:basics-java",
"//heron/common/src/java:config-java",
"//heron/common/src/java:utils-java",
diff --git
a/heron/simulator/src/java/org/apache/heron/simulator/SimulatorRunner.java
b/heron/simulator/src/java/org/apache/heron/simulator/SimulatorRunner.java
new file mode 100644
index 000..cfffa9f
--- /dev/null
+++ b/heron/simulator/src/java/org/apache/heron/simulator/SimulatorRunner.java
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.heron.simulator;
+
+import org.apache.heron.api.topology.TopologyBuilder;
+import org.apache.heron.streamlet.Builder;
+import org.apache.heron.streamlet.Config;
+import org.apache.heron.streamlet.impl.BuilderImpl;
+
+/**
+ * SimulatorRunner is used to run a topology with Heron simulator.
+ */
+public final class SimulatorRunner {
+ public SimulatorRunner() { }
+
+ /**
+ * Runs the computation
+ * @param name The name of the topology
+ * @param config Any config that is passed to the topology
+ * @param builder The builder used to keep track of the sources.
+ */
+ public void run(String name, Config config, Builder builder) {
+BuilderImpl bldr = (BuilderImpl) builder;
+TopologyBuilder topologyBuilder = bldr.build();
+
+Simulator simulator = new Simulator();
+simulator.submitTopology(name,
+ config.getHeronConfig(),
+ topologyBuilder.createTopology());
+ }
+}
[incubator-heron] branch master updated: Update output message of restart command to include container id (#3335)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new f133e73 Update output message of restart command to include container
id (#3335)
f133e73 is described below
commit f133e73434c8b3cc6ddab0de43afbf30ecca2124
Author: Ning Wang
AuthorDate: Thu Aug 29 14:21:23 2019 -0700
Update output message of restart command to include container id (#3335)
---
heron/tools/cli/src/python/restart.py | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/heron/tools/cli/src/python/restart.py
b/heron/tools/cli/src/python/restart.py
index 010c6f8..9b731fa 100644
--- a/heron/tools/cli/src/python/restart.py
+++ b/heron/tools/cli/src/python/restart.py
@@ -66,9 +66,13 @@ def run(command, parser, cl_args, unknown_args):
Log.debug("Restart Args: %s", cl_args)
container_id = cl_args['container-id']
+ message = "restart topology"
+ if container_id >= 0:
+message = "restart container " + str(container_id) + " of topology"
+
if cl_args['deploy_mode'] == config.SERVER_MODE:
dict_extra_args = {"container_id": str(container_id)}
-return cli_helper.run_server(command, cl_args, "restart topology",
extra_args=dict_extra_args)
+return cli_helper.run_server(command, cl_args, message,
extra_args=dict_extra_args)
else:
list_extra_args = ["--container_id", str(container_id)]
-return cli_helper.run_direct(command, cl_args, "restart topology",
extra_args=list_extra_args)
+return cli_helper.run_direct(command, cl_args, message,
extra_args=list_extra_args)
[incubator-heron] branch master updated: Add license files (#3331)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 12515ff Add license files (#3331) 12515ff is described below commit 12515ff65f0414dbd363d1913df357338bbf27e7 Author: Ning Wang AuthorDate: Sat Aug 24 00:02:56 2019 -0700 Add license files (#3331) Except: JSXTransformer (from react) --- LICENSE | 8 - licenses/LICENSE-Underscore.js-1.6.0.txt | 23 ++ LICENSE => licenses/LICENSE-bazel_jar_jar.txt | 129 + licenses/LICENSE-cloudpickle.txt | 32 +++ licenses/LICENSE-cpplint.txt | 27 ++ licenses/LICENSE-d3-3.4.11.txt| 27 ++ licenses/LICENSE-d3-tip-0.6.3.txt | 8 + LICENSE => licenses/LICENSE-gerrit.txt| 136 +- licenses/LICENSE-html5shiv-3.6.2pre.md| 372 ++ licenses/LICENSE-jQuery-UI-1.8.23.txt | 43 +++ licenses/LICENSE-k8s-zookeeper-docker.txt | 13 + licenses/LICENSE-kashmir.txt | 23 ++ licenses/LICENSE-list.js-1.1.1.txt| 21 ++ licenses/LICENSE-moment-2.8.3.txt | 22 ++ licenses/LICENSE-normalize.css.md | 21 ++ 15 files changed, 636 insertions(+), 269 deletions(-) diff --git a/LICENSE b/LICENSE index cb67dd0..02e56a2 100644 --- a/LICENSE +++ b/LICENSE @@ -233,14 +233,6 @@ See project link for details. -> tools/rules/jarjar_rules.bzl gerrit(https://gerrit.googlesource.com/gerrit/+/master/COPYING) -> tools/rules/javadoc.bzl -heronpy(Copyright - Parsely, Inc.) - -> heronpy/api/bolt/base_bolt.py - -> heronpy/api/bolt/bolt.py - -> heronpy/api/bolt/base_component.py - -> heronpy/api/bolt/base_spout.py - -> heronpy/api/bolt/stream.py - -> heronpy/api/bolt/generator.py - -> heronpy/api/bolt/transformoperator.py k8s-zookeeper-docker(https://github.com/muxinc/k8s-zookeeper-docker/blob/master/LICENSE) -> docker/base/scripts/generate-zookeeper-config.sh diff --git a/licenses/LICENSE-Underscore.js-1.6.0.txt b/licenses/LICENSE-Underscore.js-1.6.0.txt new file mode 100644 index 000..48404b5 --- /dev/null +++ b/licenses/LICENSE-Underscore.js-1.6.0.txt @@ -0,0 +1,23 @@ +Copyright (c) 2009-2018 Jeremy Ashkenas, DocumentCloud and Investigative +Reporters & Editors + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, +copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/LICENSE b/licenses/LICENSE-bazel_jar_jar.txt similarity index 61% copy from LICENSE copy to licenses/LICENSE-bazel_jar_jar.txt index cb67dd0..4947287 100644 --- a/LICENSE +++ b/licenses/LICENSE-bazel_jar_jar.txt @@ -1,3 +1,4 @@ + Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ @@ -173,130 +174,4 @@ incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "Li
[incubator-heron] annotated tag snapshot_2019_08_21 updated (0e0c86b -> 0825fa2)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to annotated tag snapshot_2019_08_21 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. *** WARNING: tag snapshot_2019_08_21 was modified! *** from 0e0c86b (commit) to 0825fa2 (tag) tagging 0e0c86be72607e0bbf5cf90feb1aa81252072142 (commit) replaces 0.20.1-incubating by Ning Wang on Wed Aug 21 09:00:56 2019 -0700 - Log - snapshot on 2019/08/21 --- No new revisions were added by this update. Summary of changes:
[incubator-heron] branch master updated: increase the process restart retry to 150 (#3326)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 6df60ab increase the process restart retry to 150 (#3326) 6df60ab is described below commit 6df60ab60533362bc55b196752f8ae5599ee3820 Author: Neng Lu AuthorDate: Thu Aug 15 23:17:47 2019 -0700 increase the process restart retry to 150 (#3326) --- heron/executor/src/python/heron_executor.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/heron/executor/src/python/heron_executor.py b/heron/executor/src/python/heron_executor.py index 6f5790a..9380ef5 100755 --- a/heron/executor/src/python/heron_executor.py +++ b/heron/executor/src/python/heron_executor.py @@ -297,7 +297,7 @@ class HeronExecutor(object): self.init_from_parsed_args(parsed_args) self.shell_env = shell_env -self.max_runs = 100 +self.max_runs = 150 self.interval_between_runs = 10 # Read the heron_internals.yaml for logging dir
[incubator-heron] branch master updated: Nwang/improve curator connection error message (#3327)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 6154c12 Nwang/improve curator connection error message (#3327)
6154c12 is described below
commit 6154c12d21557a5c92ccdf42357b2a728b67fefa
Author: Ning Wang
AuthorDate: Thu Aug 15 17:43:48 2019 -0700
Nwang/improve curator connection error message (#3327)
* Improve the error messages for blockUntilConnected(). The current
messages sound like client construction issue
* update two more error messages
---
.../heron/statemgr/zookeeper/curator/CuratorStateManager.java| 9 +
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git
a/heron/statemgrs/src/java/org/apache/heron/statemgr/zookeeper/curator/CuratorStateManager.java
b/heron/statemgrs/src/java/org/apache/heron/statemgr/zookeeper/curator/CuratorStateManager.java
index 78129cc..3e0e63f 100644
---
a/heron/statemgrs/src/java/org/apache/heron/statemgr/zookeeper/curator/CuratorStateManager.java
+++
b/heron/statemgrs/src/java/org/apache/heron/statemgr/zookeeper/curator/CuratorStateManager.java
@@ -118,10 +118,10 @@ public class CuratorStateManager extends
FileSystemStateManager {
try {
if (!client.blockUntilConnected(ZkContext.connectionTimeoutMs(newConfig),
TimeUnit.MILLISECONDS)) {
-throw new RuntimeException("Failed to initialize CuratorClient");
+throw new RuntimeException("Failed to connect to " + connectionString);
}
} catch (InterruptedException e) {
- throw new RuntimeException("Failed to initialize CuratorClient", e);
+ throw new RuntimeException("Interrupted from blockUntilConnected(): " +
connectionString, e);
}
if (ZkContext.isInitializeTree(newConfig)) {
@@ -335,7 +335,8 @@ public class CuratorStateManager extends
FileSystemStateManager {
// Suppress it since forPath() throws Exception
// SUPPRESS CHECKSTYLE IllegalCatch
} catch (Exception e) {
- safeSetException(future, new RuntimeException("Could not getNodeData",
e));
+ safeSetException(future, new RuntimeException(
+ "Could not getNodeData using watcher for path: " + path, e));
}
return future;
@@ -448,7 +449,7 @@ public class CuratorStateManager extends
FileSystemStateManager {
public static void main(String[] args) throws ExecutionException,
InterruptedException,
IllegalAccessException, ClassNotFoundException, InstantiationException {
if (args.length < 2) {
- throw new RuntimeException("Expects arguments:
");
+ throw new RuntimeException("Expects 2 arguments:
");
}
String zookeeperHostname = args[1];
[incubator-heron] branch master updated: Kafka bolt (#3324)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 6a28a52 Kafka bolt (#3324)
6a28a52 is described below
commit 6a28a52aff4bb573ee8a11e9f108f4e3ede2
Author: SiMing Weng
AuthorDate: Thu Aug 8 13:15:12 2019 -0400
Kafka bolt (#3324)
* update KafkaSpout
* Revert "update KafkaSpout"
This reverts commit dc64e9ad
* implement a Kafka Bolt
* add unit tests for Effective Once and At Most Once mode
---
contrib/bolts/kafka/src/java/BUILD | 29 +
.../bolts/kafka/DefaultKafkaProducerFactory.java | 38 ++
.../org/apache/heron/bolts/kafka/KafkaBolt.java| 106
.../heron/bolts/kafka/KafkaProducerFactory.java| 26
.../apache/heron/bolts/kafka/TupleTransformer.java | 43 +++
contrib/bolts/kafka/test/java/BUILD| 23
.../kafka/DefaultKafkaProducerFactoryTest.java | 41 ++
.../apache/heron/bolts/kafka/KafkaBoltTest.java| 137 +
8 files changed, 443 insertions(+)
diff --git a/contrib/bolts/kafka/src/java/BUILD
b/contrib/bolts/kafka/src/java/BUILD
new file mode 100644
index 000..1df060c
--- /dev/null
+++ b/contrib/bolts/kafka/src/java/BUILD
@@ -0,0 +1,29 @@
+licenses(["notice"])
+
+package(default_visibility = ["//visibility:public"])
+
+load("//tools/rules:build_defs.bzl", "DOCLINT_HTML_AND_SYNTAX")
+load("//tools/rules:javadoc.bzl", "java_doc")
+
+java_doc(
+name = "heron-kafka-bolt-javadoc",
+libs = [":heron-kafka-bolt-java"],
+pkgs = ["org/apache/heron/bolts/kafka"],
+title = "Kafka Bolt Documentation",
+)
+
+kafka_bolt_deps = [
+"//storm-compatibility/src/java:storm-compatibility-java-neverlink",
+"//heron/api/src/java:api-java-low-level",
+"//heron/common/src/java:basics-java",
+"//heron/common/src/java:config-java",
+"//third_party/java:logging",
+"@org_apache_kafka_kafka_clients//jar",
+]
+
+java_library(
+name = "heron-kafka-bolt-java",
+srcs = glob(["org/apache/heron/bolts/kafka/**/*.java"]),
+javacopts = DOCLINT_HTML_AND_SYNTAX,
+deps = kafka_bolt_deps,
+)
\ No newline at end of file
diff --git
a/contrib/bolts/kafka/src/java/org/apache/heron/bolts/kafka/DefaultKafkaProducerFactory.java
b/contrib/bolts/kafka/src/java/org/apache/heron/bolts/kafka/DefaultKafkaProducerFactory.java
new file mode 100644
index 000..3fdd223
--- /dev/null
+++
b/contrib/bolts/kafka/src/java/org/apache/heron/bolts/kafka/DefaultKafkaProducerFactory.java
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.heron.bolts.kafka;
+
+import java.util.Map;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+
+@SuppressWarnings("unused")
+public class DefaultKafkaProducerFactory implements
KafkaProducerFactory {
+ private static final long serialVersionUID = -2184222924531815883L;
+ private Map configs;
+
+ public DefaultKafkaProducerFactory(Map configs) {
+this.configs = configs;
+ }
+
+ @Override
+ public Producer create() {
+return new KafkaProducer<>(configs);
+ }
+}
diff --git
a/contrib/bolts/kafka/src/java/org/apache/heron/bolts/kafka/KafkaBolt.java
b/contrib/bolts/kafka/src/java/org/apache/heron/bolts/kafka/KafkaBolt.java
new file mode 100644
index 000..95ce3f8
--- /dev/null
+++ b/contrib/bolts/kafka/src/java/org/apache/heron/bolts/kafka/KafkaBolt.java
@@ -0,0 +1,106 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); yo
[incubator-heron] branch master updated: Optimize hot spots in Stream Manager (#3322)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 12f641b Optimize hot spots in Stream Manager (#3322)
12f641b is described below
commit 12f641b442743dfe4ed3fbadea27e47cd6167263
Author: Dmitry Rusakov
AuthorDate: Wed Aug 7 11:47:04 2019 -0700
Optimize hot spots in Stream Manager (#3322)
* Replace std::unordered_map with tsl::hopscotch_map in tuple-cache.h
* Replaced std::hash with a faster xxhash implementation to speedup the
topology if it uses fields-grouping strategy
* Enabled O3 optimization level be default to guarantee all smart pointers
are inlined
* xxhash migration has been rollbacked since its performance effect is
negligible
* RETRY_ATTEMPTS bumped to 50 since Travis is slow
---
WORKSPACE| 8
heron/stmgr/src/cpp/BUILD| 1 +
heron/stmgr/src/cpp/util/tuple-cache.h | 3 ++-
integration_test/src/python/topology_test_runner/main.py | 2 +-
third_party/hopscotch-hashmap/hopscotch.BUILD| 12
tools/bazel.rc | 9 -
tools/travis/toolchain/CROSSTOOL | 2 +-
7 files changed, 33 insertions(+), 4 deletions(-)
diff --git a/WORKSPACE b/WORKSPACE
index 07ebcab..872c975 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -925,6 +925,14 @@ http_archive(
build_file = "@//:third_party/cppcheck/cppcheck.BUILD",
sha256 =
"b3de7fbdc1a23d7341b55f7f88877e106a76847bd5a07fa721c07310b625318b",
)
+
+http_archive(
+name = "com_github_hopscotch_hashmap",
+build_file = "@//:third_party/hopscotch-hashmap/hopscotch.BUILD",
+sha256 =
"73e301925e1418c5ed930ef37ebdcab2c395a6d1bdaf5a012034bb75307d33f1",
+strip_prefix = "hopscotch-map-2.2.1",
+urls = ["https://github.com/Tessil/hopscotch-map/archive/v2.2.1.tar.gz;],
+)
# end 3rdparty C++ dependencies
# for helm
diff --git a/heron/stmgr/src/cpp/BUILD b/heron/stmgr/src/cpp/BUILD
index ebd14f8..c056023 100644
--- a/heron/stmgr/src/cpp/BUILD
+++ b/heron/stmgr/src/cpp/BUILD
@@ -61,6 +61,7 @@ cc_library(
"//heron/proto:proto-cxx",
"//heron/common/src/cpp/network:network-cxx",
"//heron/common/src/cpp/config:config-cxx",
+"@com_github_hopscotch_hashmap//:hopscotch-hashmap",
],
linkstatic = 1,
)
diff --git a/heron/stmgr/src/cpp/util/tuple-cache.h
b/heron/stmgr/src/cpp/util/tuple-cache.h
index a0e23eb..d25ae9c 100644
--- a/heron/stmgr/src/cpp/util/tuple-cache.h
+++ b/heron/stmgr/src/cpp/util/tuple-cache.h
@@ -20,6 +20,7 @@
#ifndef SRC_CPP_SVCS_STMGR_SRC_UTIL_TUPLE_CACHE_H_
#define SRC_CPP_SVCS_STMGR_SRC_UTIL_TUPLE_CACHE_H_
+#include
#include
#include
#include
@@ -114,7 +115,7 @@ class TupleCache {
TupleList* get(sp_int32 _task_id);
// map from task_id to the TupleList
- std::unordered_map cache_;
+ tsl::hopscotch_map cache_;
std::shared_ptr eventLoop_;
std::function tuple_drainer_;
std::function
diff --git a/integration_test/src/python/topology_test_runner/main.py
b/integration_test/src/python/topology_test_runner/main.py
index 741bade..08fefea 100644
--- a/integration_test/src/python/topology_test_runner/main.py
+++ b/integration_test/src/python/topology_test_runner/main.py
@@ -37,7 +37,7 @@ from heron.statemgrs.src.python.filestatemanager import
FileStateManager
DEFAULT_TEST_CONF_FILE =
"integration_test/src/python/topology_test_runner/resources/test.json"
#seconds
-RETRY_ATTEMPTS = 25
+RETRY_ATTEMPTS = 50
RETRY_INTERVAL = 10
WAIT_FOR_DEACTIVATION = 5
diff --git a/third_party/hopscotch-hashmap/hopscotch.BUILD
b/third_party/hopscotch-hashmap/hopscotch.BUILD
new file mode 100644
index 000..8ef7618
--- /dev/null
+++ b/third_party/hopscotch-hashmap/hopscotch.BUILD
@@ -0,0 +1,12 @@
+licenses(["notice"])
+
+package(default_visibility = ["//visibility:public"])
+
+cc_library(
+name = "hopscotch-hashmap",
+hdrs = glob([
+"include/tsl/*.h",
+]),
+includes = ["include"],
+linkstatic = 1,
+)
diff --git a/tools/bazel.rc b/tools/bazel.rc
index 2346ca4..0e35097 100644
--- a/tools/bazel.rc
+++ b/tools/bazel.rc
@@ -34,6 +34,7 @@ build:centos --linkopt -lpthread
build:centos --linkopt -lrt
build:centos --spawn_strategy=standalone
build:centos --workspace_status_command scripts/release/status.sh
+build:centos --copt=-O3
# For debian
# To use it: bazel build --config=debian
@@ -47,6 +48,7 @@ build:debian --linkopt -lpthread
build:debian --linkopt -lrt
build:debian --spawn_strategy=standalone
build:debian --workspace_statu
[incubator-heron] annotated tag nwang.test.05.23 deleted (was 4756ba8)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to annotated tag nwang.test.05.23 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. *** WARNING: tag nwang.test.05.23 was deleted! *** tag was 4756ba8 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository.
[incubator-heron] annotated tag nwang.test.06.11 deleted (was 4f1e3ea)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to annotated tag nwang.test.06.11 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. *** WARNING: tag nwang.test.06.11 was deleted! *** tag was 4f1e3ea The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository.
[incubator-heron] annotated tag 0.20.0.3 deleted (was 22f758b)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to annotated tag 0.20.0.3 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. *** WARNING: tag 0.20.0.3 was deleted! *** tag was 22f758b The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository.
[incubator-heron] annotated tag 0.20.0.1 deleted (was a581d10)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to annotated tag 0.20.0.1 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. *** WARNING: tag 0.20.0.1 was deleted! *** tag was a581d10 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository.
[incubator-heron] annotated tag 0.20.0.2 deleted (was 5bd6973)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to annotated tag 0.20.0.2 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. *** WARNING: tag 0.20.0.2 was deleted! *** tag was 5bd6973 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository.
[incubator-heron] branch 0.20.1-incubating created (now 40ccec6)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a change to branch 0.20.1-incubating in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. at 40ccec6 Joshfischer/refine docs (#3300) No new revisions were added by this update.
[incubator-heron] branch master updated: adding wait for zookeeper script (#3313)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new a130d5e adding wait for zookeeper script (#3313) a130d5e is described below commit a130d5e54cd8642d7e26584ed2e2f9d2285570be Author: Josh Fischer AuthorDate: Fri Aug 2 01:04:48 2019 -0500 adding wait for zookeeper script (#3313) --- docker/dist/Dockerfile.dist.debian9 | 3 ++- docker/dist/Dockerfile.dist.ubuntu14.04 | 3 ++- docker/dist/scripts/wait-for-zookeeper.sh | 36 +++ 3 files changed, 40 insertions(+), 2 deletions(-) diff --git a/docker/dist/Dockerfile.dist.debian9 b/docker/dist/Dockerfile.dist.debian9 index f183129..94fdaa2 100644 --- a/docker/dist/Dockerfile.dist.debian9 +++ b/docker/dist/Dockerfile.dist.debian9 @@ -82,4 +82,5 @@ RUN mkdir -p /opt/zookeeper/scripts ADD dist/scripts /opt/zookeeper/scripts RUN chmod +x /opt/zookeeper/scripts/generate-zookeeper-config.sh \ && chmod +x /opt/zookeeper/scripts/zookeeper-ruok.sh \ -&& chmod +x /opt/zookeeper/scripts/start-zookeeper.sh +&& chmod +x /opt/zookeeper/scripts/start-zookeeper.sh \ +&& chmod +x /opt/zookeeper/scripts/wait-for-zookeeper.sh diff --git a/docker/dist/Dockerfile.dist.ubuntu14.04 b/docker/dist/Dockerfile.dist.ubuntu14.04 index 04f2602..17930e0 100644 --- a/docker/dist/Dockerfile.dist.ubuntu14.04 +++ b/docker/dist/Dockerfile.dist.ubuntu14.04 @@ -82,6 +82,7 @@ RUN mkdir -p /opt/zookeeper/scripts ADD dist/scripts /opt/zookeeper/scripts RUN chmod +x /opt/zookeeper/scripts/generate-zookeeper-config.sh \ && chmod +x /opt/zookeeper/scripts/zookeeper-ruok.sh \ -&& chmod +x /opt/zookeeper/scripts/start-zookeeper.sh +&& chmod +x /opt/zookeeper/scripts/start-zookeeper.sh \ +&& chmod +x /opt/zookeeper/scripts/wait-for-zookeeper.sh CMD ["supervisord", "-n"] diff --git a/docker/dist/scripts/wait-for-zookeeper.sh b/docker/dist/scripts/wait-for-zookeeper.sh new file mode 100644 index 000..bd6b353 --- /dev/null +++ b/docker/dist/scripts/wait-for-zookeeper.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +if [ "$#" -ne 2 ]; then + echo "Usage: $0 " + exit 1 +fi + +WAIT_ZK_HOST=$1 +WAIT_ZK_PORT=$2 + +while true; do + status=$(echo ruok | nc $WAIT_ZK_HOST $WAIT_ZK_PORT); + writestatus=$(echo isro | nc $WAIT_ZK_HOST $WAIT_ZK_PORT) + if [ "$status" = "imok" ] && [ "$writestatus" == "rw" ]; then +echo "Zookeeper $WAIT_ZK_HOST:$WAIT_ZK_PORT is ready"; +exit 0 + fi; + echo "Zookeeper $WAIT_ZK_HOST:$WAIT_ZK_PORT not ready"; + sleep 4; +done
[incubator-heron] branch master updated: When create a smart pointer we now allocate the most hot/frequent messages in the memory pool instead of the heap (#3316)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new a01930e When create a smart pointer we now allocate the most hot/frequent messages in the memory pool instead of the heap (#3316) a01930e is described below commit a01930e86e3709958db14025716a8598dbd105c2 Author: Dmitry Rusakov AuthorDate: Mon Jul 22 22:28:51 2019 -0700 When create a smart pointer we now allocate the most hot/frequent messages in the memory pool instead of the heap (#3316) --- heron/common/src/cpp/basics/basics.h | 19 +++ heron/common/src/cpp/metrics/metricsmgr-client.cpp | 2 +- heron/common/src/cpp/metrics/metricsmgr-client.h | 4 ++-- heron/common/src/cpp/network/client.h| 14 +++--- heron/common/src/cpp/network/server.h| 12 ++-- heron/common/tests/cpp/network/client_unittest.cpp | 2 +- heron/common/tests/cpp/network/client_unittest.h | 2 +- heron/common/tests/cpp/network/oclient_unittest.cpp | 2 +- heron/common/tests/cpp/network/oclient_unittest.h| 2 +- heron/common/tests/cpp/network/oserver_unittest.cpp | 4 ++-- heron/common/tests/cpp/network/oserver_unittest.h| 5 +++-- heron/common/tests/cpp/network/server_unittest.cpp | 4 ++-- heron/common/tests/cpp/network/server_unittest.h | 5 +++-- heron/instance/src/cpp/boltimpl/bolt-instance.cpp| 4 ++-- heron/instance/src/cpp/boltimpl/bolt-instance.h | 6 +++--- heron/instance/src/cpp/gateway/gateway.cpp | 4 ++-- heron/instance/src/cpp/gateway/gateway.h | 13 +++-- heron/instance/src/cpp/gateway/stmgr-client.cpp | 16 +--- heron/instance/src/cpp/gateway/stmgr-client.h| 15 --- heron/instance/src/cpp/instance-main.cpp | 2 +- heron/instance/src/cpp/slave/instance-base.h | 2 +- heron/instance/src/cpp/slave/slave.cpp | 16 heron/instance/src/cpp/slave/slave.h | 15 --- heron/instance/src/cpp/spoutimpl/spout-instance.cpp | 2 +- heron/instance/src/cpp/spoutimpl/spout-instance.h| 2 +- heron/stmgr/src/cpp/manager/checkpoint-gateway.cpp | 14 +++--- heron/stmgr/src/cpp/manager/checkpoint-gateway.h | 8 heron/stmgr/src/cpp/manager/ckptmgr-client.cpp | 10 +- heron/stmgr/src/cpp/manager/ckptmgr-client.h | 6 +++--- heron/stmgr/src/cpp/manager/instance-server.cpp | 12 ++-- heron/stmgr/src/cpp/manager/instance-server.h| 13 +++-- heron/stmgr/src/cpp/manager/stmgr-client.cpp | 9 + heron/stmgr/src/cpp/manager/stmgr-client.h | 4 ++-- heron/stmgr/src/cpp/manager/stmgr-server.cpp | 10 +- heron/stmgr/src/cpp/manager/stmgr-server.h | 10 +- heron/stmgr/src/cpp/manager/stmgr.cpp| 4 ++-- heron/stmgr/src/cpp/manager/stmgr.h | 4 ++-- heron/stmgr/src/cpp/manager/tmaster-client.cpp | 20 ++-- heron/stmgr/src/cpp/manager/tmaster-client.h | 14 -- .../tests/cpp/server/checkpoint-gateway_unittest.cpp | 2 +- heron/stmgr/tests/cpp/server/dummy_instance.cpp | 18 +- heron/stmgr/tests/cpp/server/dummy_instance.h| 16 heron/stmgr/tests/cpp/server/dummy_metricsmgr.cpp| 6 +++--- heron/stmgr/tests/cpp/server/dummy_metricsmgr.h | 7 --- heron/stmgr/tests/cpp/server/dummy_stmgr.cpp | 10 ++ heron/stmgr/tests/cpp/server/dummy_stmgr.h | 12 ++-- heron/tmaster/src/cpp/manager/ckptmgr-client.cpp | 12 ++-- heron/tmaster/src/cpp/manager/ckptmgr-client.h | 12 ++-- heron/tmaster/src/cpp/manager/tmasterserver.cpp | 12 ++-- heron/tmaster/src/cpp/manager/tmasterserver.h| 12 ++-- .../src/cpp/processor/stmgr-heartbeat-processor.cpp | 4 ++-- .../src/cpp/processor/stmgr-heartbeat-processor.h| 2 +- .../src/cpp/processor/stmgr-register-processor.cpp | 4 ++-- .../src/cpp/processor/stmgr-register-processor.h | 2 +- .../tmaster/src/cpp/processor/tmaster-processor.cpp | 2 +- heron/tmaster/src/cpp/processor/tmaster-processor.h | 4 ++-- heron/tmaster/tests/cpp/server/dummystmgr.cpp| 18 +- heron/tmaster/tests/cpp/server/dummystmgr.h | 12 +++- heron/tmaster/tests/cpp/server/dummytmaster.cpp | 2 +- heron/tmaster/tests/cpp/server/dummytmaster.h| 2 +- 60 files changed, 259 insertions(+), 224 deletions(-) diff --git a/heron/common/src/cpp/basics/basics.h b/heron/common/src/cpp/basics/basics.h index 766a64f..dc5be56 100644 --- a/heron/common/src/cpp/basics/basics.h +++ b/heron/common/src/cpp
[incubator-heron] branch master updated: Flip buffers using helper method (#3311)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new bfe58d1 Flip buffers using helper method (#3311)
bfe58d1 is described below
commit bfe58d1e05cb3f5fb5db2f51d13592d9a65ac477
Author: Rohan Agarwal
AuthorDate: Wed Jul 10 16:45:24 2019 -0700
Flip buffers using helper method (#3311)
* flip buffer using helper method
* make constructor private
* make class final
* move constructor to top of class
* no-op
* empty commit to re-trigger CI
* final try for flaky integration tests
---
.../apache/heron/common/network/BufferHelper.java | 48 ++
.../heron/common/network/IncomingPacket.java | 4 +-
.../heron/common/network/OutgoingPacket.java | 2 +-
3 files changed, 51 insertions(+), 3 deletions(-)
diff --git
a/heron/common/src/java/org/apache/heron/common/network/BufferHelper.java
b/heron/common/src/java/org/apache/heron/common/network/BufferHelper.java
new file mode 100644
index 000..2849923
--- /dev/null
+++ b/heron/common/src/java/org/apache/heron/common/network/BufferHelper.java
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.heron.common.network;
+
+import java.nio.Buffer;
+import java.nio.ByteBuffer;
+
+/**
+ * Helper methods for working with {@link Buffer} objects.
+ */
+final class BufferHelper {
+
+ private BufferHelper() {
+ }
+
+ /**
+ * Flip the provided buffer.
+ *
+ * This wrapper around {@link Buffer#flip()} is required because of
+ * incompatible ABI changes between Java 8 and 11. In Java 8, {@link
ByteBuffer#flip()} returns
+ * a {@link Buffer}, whereas in Java 11, this method returns a {@link
ByteBuffer}.
+ *
+ * If this function is used, any object of {@link Buffer} (and subclasses)
are first cast to
+ * {@link Buffer} objects, then flipped, thus avoiding the binary
incompatibility.
+ *
+ * @param buffer The buffer to flip
+ */
+ static void flip(Buffer buffer) {
+buffer.flip();
+ }
+}
diff --git
a/heron/common/src/java/org/apache/heron/common/network/IncomingPacket.java
b/heron/common/src/java/org/apache/heron/common/network/IncomingPacket.java
index 045602e..b1bf26c 100644
--- a/heron/common/src/java/org/apache/heron/common/network/IncomingPacket.java
+++ b/heron/common/src/java/org/apache/heron/common/network/IncomingPacket.java
@@ -76,7 +76,7 @@ public class IncomingPacket {
}
// We read the header fully
headerRead = true;
- header.flip();
+ BufferHelper.flip(header);
int size = header.getInt();
if (size > limit) {
LOG.log(Level.SEVERE, "packet size " + size + " exceeds limit " +
limit);
@@ -86,7 +86,7 @@ public class IncomingPacket {
}
int retval = readFromChannel(channel, data);
if (retval == 0) {
- data.flip();
+ BufferHelper.flip(data);
}
return retval;
}
diff --git
a/heron/common/src/java/org/apache/heron/common/network/OutgoingPacket.java
b/heron/common/src/java/org/apache/heron/common/network/OutgoingPacket.java
index 2b91077..63983db 100644
--- a/heron/common/src/java/org/apache/heron/common/network/OutgoingPacket.java
+++ b/heron/common/src/java/org/apache/heron/common/network/OutgoingPacket.java
@@ -82,7 +82,7 @@ public class OutgoingPacket {
buffer.put(message.toByteArray());
// Make the buffer ready for writing out
-buffer.flip();
+BufferHelper.flip(buffer);
}
public static int sizeRequiredToPackString(String str) {
[incubator-heron] branch master updated: Allow interpolate more spout config parameters in extra links (#3310)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 912d84b Allow interpolate more spout config parameters in extra links
(#3310)
912d84b is described below
commit 912d84bd4a0fc1fd095d2493a2af2745a3f93223
Author: Xiaoyao Qian
AuthorDate: Wed Jul 10 11:52:47 2019 -0700
Allow interpolate more spout config parameters in extra links (#3310)
* Allow interpolate more spout config parameters in extra links
* new interpolate design
---
.../config/src/yaml/tracker/heron_tracker.yaml | 14 ---
heron/tools/tracker/src/python/config.py | 37 +--
heron/tools/tracker/src/python/tracker.py | 43 +-
3 files changed, 27 insertions(+), 67 deletions(-)
diff --git a/heron/tools/config/src/yaml/tracker/heron_tracker.yaml
b/heron/tools/config/src/yaml/tracker/heron_tracker.yaml
index 15f87c8..1d8cf5d 100644
--- a/heron/tools/config/src/yaml/tracker/heron_tracker.yaml
+++ b/heron/tools/config/src/yaml/tracker/heron_tracker.yaml
@@ -69,17 +69,3 @@ statemgrs:
# formatter: "http://127.0.0.1/viz/${ENVIRON}-${CLUSTER}-${TOPOLOGY};
# - name: "Alerts"
# formatter: "http://127.0.0.1/alerts/${ENVIRON}-${CLUSTER}-${TOPOLOGY};
-#
-# spout.extra.links:
-# - spout.type: "kafka"
-# extra.links:
-# - name: "Viz"
-# formatter:
"http://127.0.0.1/kafka/viz/${ENVIRON}-${CLUSTER}-${TOPOLOGY}-${SPOUT_NAME}-${SPOUT_SOURCE};
-# - name: "Alerts"
-# formatter:
"http://127.0.0.1/kafka/alerts/${ENVIRON}-${CLUSTER}-${TOPOLOGY}-${SPOUT_NAME}-${SPOUT_SOURCE};
-# - spout.type: "default"
-# extra.links:
-# - name: "Viz"
-# formatter:
"http://127.0.0.1/default/viz/${ENVIRON}-${CLUSTER}-${TOPOLOGY}-${SPOUT_NAME}-${SPOUT_SOURCE};
-# - name: "Alerts"
-# formatter:
"http://127.0.0.1/default/alerts/${ENVIRON}-${CLUSTER}-${TOPOLOGY}-${SPOUT_NAME}-${SPOUT_SOURCE};
diff --git a/heron/tools/tracker/src/python/config.py
b/heron/tools/tracker/src/python/config.py
index 6dbfc3d..08887b8 100644
--- a/heron/tools/tracker/src/python/config.py
+++ b/heron/tools/tracker/src/python/config.py
@@ -27,8 +27,6 @@ EXTRA_LINKS_KEY = "extra.links"
EXTRA_LINK_NAME_KEY = "name"
EXTRA_LINK_FORMATTER_KEY = "formatter"
EXTRA_LINK_URL_KEY = "url"
-SPOUT_EXTRA_LINKS_KEY = "spout.extra.links"
-SPOUT_TYPE_KEY = "spout.type"
class Config(object):
"""
@@ -40,11 +38,9 @@ class Config(object):
self.configs = configs
self.statemgr_config = StateMgrConfig()
self.extra_links = []
-self.spout_extra_links = {}
self.load_configs()
- # pylint: disable=line-too-long
def load_configs(self):
"""load config files"""
self.statemgr_config.set_state_locations(self.configs[STATEMGRS_KEY])
@@ -52,10 +48,6 @@ class Config(object):
for extra_link in self.configs[EXTRA_LINKS_KEY]:
self.extra_links.append(self.validate_extra_link(extra_link))
-if SPOUT_EXTRA_LINKS_KEY in self.configs:
- for extra_link in self.configs[SPOUT_EXTRA_LINKS_KEY]:
-self.spout_extra_links[extra_link[SPOUT_TYPE_KEY]] =
[self.validate_extra_link(link) for link in extra_link[EXTRA_LINKS_KEY]]
-
def validate_extra_link(self, extra_link):
"""validate extra link"""
if EXTRA_LINK_NAME_KEY not in extra_link or EXTRA_LINK_FORMATTER_KEY not
in extra_link:
@@ -77,8 +69,6 @@ class Config(object):
"${TOPOLOGY}": "topology",
"${ROLE}": "role",
"${USER}": "user",
-"${SPOUT_NAME}": "spout_name",
-"${SPOUT_SOURCE}": "spout_source",
}
dummy_formatted_url = url_format
for key, value in valid_parameters.items():
@@ -91,34 +81,25 @@ class Config(object):
# No error is thrown, so the format is valid.
return url_format
- def get_formatted_url(self, formatter, execution_state, **additional):
+ def get_formatted_url(self, formatter, execution_state):
"""
+@param formatter: The template string to interpolate
@param execution_state: The python dict representing JSON execution_state
-@param additional: additional kwargs to interpolate
@return Formatted viz url
"""
# Create the parameters based on execution state
-valid_parameters = {
-"${CLUSTER}": execution_state.get("cluster",
- additional.get("cluster",
"${CLUSTER}")),
-"${ENVIRON}": execution_state.get("
[incubator-heron] branch master updated: Add cluster to cmdline options (#3312)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new bffedbf Add cluster to cmdline options (#3312)
bffedbf is described below
commit bffedbf178ca66487083a5f6d9b22308e722f177
Author: Ning Wang
AuthorDate: Wed Jul 10 11:52:16 2019 -0700
Add cluster to cmdline options (#3312)
---
heron/tools/cli/src/python/submit.py | 4
1 file changed, 4 insertions(+)
diff --git a/heron/tools/cli/src/python/submit.py
b/heron/tools/cli/src/python/submit.py
index ca7d5e6..f147619 100644
--- a/heron/tools/cli/src/python/submit.py
+++ b/heron/tools/cli/src/python/submit.py
@@ -463,6 +463,10 @@ def run(command, parser, cl_args, unknown_args):
opts.set_config('cmdline.topology.initial.state', initial_state)
opts.set_config('cmdline.topology.role', cl_args['role'])
opts.set_config('cmdline.topology.environment', cl_args['environ'])
+ opts.set_config('cmdline.topology.cluster', cl_args['cluster'])
+ opts.set_config('cmdline.topology.file_name', cl_args['topology-file-name'])
+ opts.set_config('cmdline.topology.class_name',
cl_args['topology-class-name'])
+ opts.set_config('cmdline.topology.submit_user', cl_args['submit_user'])
# Use CLI release yaml file if the release_yaml_file config is empty
if not cl_args['release_yaml_file']:
[incubator-heron] branch master updated: Annotate 3rd party and UI libraries in rat excludes with licenses (#3309)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new 18d6660 Annotate 3rd party and UI libraries in rat excludes with licenses (#3309) 18d6660 is described below commit 18d6660abca88392ea9d880e68b0a1cf34c008b3 Author: Ning Wang AuthorDate: Mon Jul 8 16:55:35 2019 -0700 Annotate 3rd party and UI libraries in rat excludes with licenses (#3309) * Annotate 3rd party and UI libraries in rat excludes with licenses * Annotate website/website2 --- .rat-excludes | 30 +- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/.rat-excludes b/.rat-excludes index bcdac11..f1d77ec 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -13,28 +13,40 @@ autogen\.sh description heron\.iml +# Directories +bazel-.* +m4 +META-INF +# website and website2 are not part of the release +website + # UI resources +# Apache 2.0 licenses & MIT Licenses jquery.*\.js +# Apache 2.0 licenses & MIT Licenses bootstrap\.min\.css -glyphicons-halflings-regular\.svg bootstrap\.min\.js +# MIT licenses +glyphicons-halflings-regular\.svg +# MIT licenses & BSD 3-Clause licenses d3.*\.js +# MIT licenses html5-trunk\.js +# MIT licenses list\.min.*.js +# MIT licenses moment\.min.*\.js +# MIT licenses underscore-min.*\.js underscore-min.*\.map -# Directories -bazel-.* -m4 -website -META-INF # Thirdparty +# 3-Clause BSD License cpplint\.py +# BSD 3-Clause licenses +cloudpickle.py +# Boost Software License, Version 1.0 kashmir +# BSD License semver - -# Unrecognizable license -cloudpickle.py
[incubator-heron] branch master updated: Feature/helm (#3296)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 882ec8c Feature/helm (#3296)
882ec8c is described below
commit 882ec8c087785e0b89a5daa09ea522d223d40619
Author: Scott <47566431+the-scott-h...@users.noreply.github.com>
AuthorDate: Fri Jul 5 18:10:16 2019 -0400
Feature/helm (#3296)
* fix inconsistent volume names and init container
* package names and spacing fix
* fix string comparison init container
could have also used double brackets
* fix prometheus stats provider class name
* add prometheus, kubectlImage and storageClassName
* add template for kubectl image:version
* add prometheus and storageClassName template logic
* add storageClassName template logic to zookeeper
* fix typo in zookeeper.yaml storageClass logic
* remove quotes from prometheus bool comparison
---
deploy/kubernetes/helm/templates/bookie.yaml| 16 +---
deploy/kubernetes/helm/templates/tools.yaml | 2 +-
deploy/kubernetes/helm/templates/zookeeper.yaml | 3 +++
deploy/kubernetes/helm/values.yaml.template | 8
4 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/deploy/kubernetes/helm/templates/bookie.yaml
b/deploy/kubernetes/helm/templates/bookie.yaml
index 15294e6..fecc7f1 100644
--- a/deploy/kubernetes/helm/templates/bookie.yaml
+++ b/deploy/kubernetes/helm/templates/bookie.yaml
@@ -42,7 +42,11 @@ data:
BK_zkServers: {{ .Release.Name }}-zookeeper:{{ .Values.zookeeper.clientPort
}}
BK_autoRecoveryDaemonEnabled: "true"
# TODO: Issue 458: https://github.com/apache/bookkeeper/issues/458
- #BK_statsProviderClass:
org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
+ {{- if eq .Values.bookkeeper.prometheus.enabled true }}
+ BK_enableStatistics: "true"
+ BK_enableTaskExecutionStats: "true"
+ BK_statsProviderClass:
org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
+ {{- end }}
---
## BookKeeper servers need to access the local disks and the pods
@@ -107,8 +111,8 @@ spec:
- -c
- >-
while true; do
-status=$(echo ruok | nc {{ .Release.Name }}-zookeeper {{
.Values.zookeeper.clientPort }});
-writestatus=$(echo isro | nc {{ .Release.Name }}-zookeeper {{
.Values.zookeeper.clientPort }})
+status=$(echo ruok | nc -w 3 {{ .Release.Name }}-zookeeper {{
.Values.zookeeper.clientPort }});
+writestatus=$(echo isro | nc -w 3 {{ .Release.Name
}}-zookeeper {{ .Values.zookeeper.clientPort }})
if [ "$status" = "imok" ] && [ "$writestatus" = "rw" ]; then
break;
fi;
@@ -162,6 +166,9 @@ spec:
component: {{ .Release.Name }}-bookkeeper
spec:
accessModes: [ "ReadWriteOnce" ]
+{{- if ne .Values.bookkeeper.storageClassName "none" }}
+storageClassName: {{ .Values.bookkeeper.storageClassName }}
+{{- end }}
resources:
requests:
storage: {{ $bookieJournalCapacity }}
@@ -173,6 +180,9 @@ spec:
component: {{ .Release.Name }}-bookkeeper
spec:
accessModes: [ "ReadWriteOnce" ]
+{{- if ne .Values.bookkeeper.storageClassName "none" }}
+storageClassName: {{ .Values.bookkeeper.storageClassName }}
+{{- end }}
resources:
requests:
storage: {{ $bookieStorageCapacity }}
diff --git a/deploy/kubernetes/helm/templates/tools.yaml
b/deploy/kubernetes/helm/templates/tools.yaml
index cc2dbeb..91f4f06 100644
--- a/deploy/kubernetes/helm/templates/tools.yaml
+++ b/deploy/kubernetes/helm/templates/tools.yaml
@@ -158,7 +158,7 @@ spec:
- configMapRef:
name: {{ .Release.Name }}-tools-config
- name: kubectl-proxy
- image: heron/kubectl:latest
+ image: {{ .Values.kubectlImage }}
command: ["sh", "-c"]
args:
- >
diff --git a/deploy/kubernetes/helm/templates/zookeeper.yaml
b/deploy/kubernetes/helm/templates/zookeeper.yaml
index 3a0d842..4e4f5b2 100644
--- a/deploy/kubernetes/helm/templates/zookeeper.yaml
+++ b/deploy/kubernetes/helm/templates/zookeeper.yaml
@@ -146,6 +146,9 @@ spec:
name: datadir
spec:
accessModes: [ "ReadWriteOnce" ]
+ {{- if ne .Values.zookeeper.storageClassName "none" }}
+ storageClassName: {{ .Values.zookeeper.storageClassName }}
+ {{- end }}
resources:
requests:
storage: {{ .Values.zookeeper.storage }}
diff --git a/deploy/kubern
[incubator-heron] branch master updated: Spout source extra links (#3306)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new cbbcd27 Spout source extra links (#3306)
cbbcd27 is described below
commit cbbcd27f5f78c0c70c78c7da7ba049cd950131cf
Author: Xiaoyao Qian
AuthorDate: Wed Jul 3 19:32:25 2019 -0700
Spout source extra links (#3306)
* tracker changes
* UI change
* more condition check
* fix style
---
.../config/src/yaml/tracker/heron_tracker.yaml | 14
heron/tools/tracker/src/python/config.py | 34 ++
.../src/python/handlers/logicalplanhandler.py | 1 +
heron/tools/tracker/src/python/tracker.py | 40 ++
heron/tools/ui/resources/static/js/topologies.js | 32 ++---
5 files changed, 103 insertions(+), 18 deletions(-)
diff --git a/heron/tools/config/src/yaml/tracker/heron_tracker.yaml
b/heron/tools/config/src/yaml/tracker/heron_tracker.yaml
index 1d8cf5d..15f87c8 100644
--- a/heron/tools/config/src/yaml/tracker/heron_tracker.yaml
+++ b/heron/tools/config/src/yaml/tracker/heron_tracker.yaml
@@ -69,3 +69,17 @@ statemgrs:
# formatter: "http://127.0.0.1/viz/${ENVIRON}-${CLUSTER}-${TOPOLOGY};
# - name: "Alerts"
# formatter: "http://127.0.0.1/alerts/${ENVIRON}-${CLUSTER}-${TOPOLOGY};
+#
+# spout.extra.links:
+# - spout.type: "kafka"
+# extra.links:
+# - name: "Viz"
+# formatter:
"http://127.0.0.1/kafka/viz/${ENVIRON}-${CLUSTER}-${TOPOLOGY}-${SPOUT_NAME}-${SPOUT_SOURCE};
+# - name: "Alerts"
+# formatter:
"http://127.0.0.1/kafka/alerts/${ENVIRON}-${CLUSTER}-${TOPOLOGY}-${SPOUT_NAME}-${SPOUT_SOURCE};
+# - spout.type: "default"
+# extra.links:
+# - name: "Viz"
+# formatter:
"http://127.0.0.1/default/viz/${ENVIRON}-${CLUSTER}-${TOPOLOGY}-${SPOUT_NAME}-${SPOUT_SOURCE};
+# - name: "Alerts"
+# formatter:
"http://127.0.0.1/default/alerts/${ENVIRON}-${CLUSTER}-${TOPOLOGY}-${SPOUT_NAME}-${SPOUT_SOURCE};
diff --git a/heron/tools/tracker/src/python/config.py
b/heron/tools/tracker/src/python/config.py
index 8da884f..6dbfc3d 100644
--- a/heron/tools/tracker/src/python/config.py
+++ b/heron/tools/tracker/src/python/config.py
@@ -26,7 +26,9 @@ STATEMGRS_KEY = "statemgrs"
EXTRA_LINKS_KEY = "extra.links"
EXTRA_LINK_NAME_KEY = "name"
EXTRA_LINK_FORMATTER_KEY = "formatter"
-
+EXTRA_LINK_URL_KEY = "url"
+SPOUT_EXTRA_LINKS_KEY = "spout.extra.links"
+SPOUT_TYPE_KEY = "spout.type"
class Config(object):
"""
@@ -38,9 +40,11 @@ class Config(object):
self.configs = configs
self.statemgr_config = StateMgrConfig()
self.extra_links = []
+self.spout_extra_links = {}
self.load_configs()
+ # pylint: disable=line-too-long
def load_configs(self):
"""load config files"""
self.statemgr_config.set_state_locations(self.configs[STATEMGRS_KEY])
@@ -48,6 +52,10 @@ class Config(object):
for extra_link in self.configs[EXTRA_LINKS_KEY]:
self.extra_links.append(self.validate_extra_link(extra_link))
+if SPOUT_EXTRA_LINKS_KEY in self.configs:
+ for extra_link in self.configs[SPOUT_EXTRA_LINKS_KEY]:
+self.spout_extra_links[extra_link[SPOUT_TYPE_KEY]] =
[self.validate_extra_link(link) for link in extra_link[EXTRA_LINKS_KEY]]
+
def validate_extra_link(self, extra_link):
"""validate extra link"""
if EXTRA_LINK_NAME_KEY not in extra_link or EXTRA_LINK_FORMATTER_KEY not
in extra_link:
@@ -69,6 +77,8 @@ class Config(object):
"${TOPOLOGY}": "topology",
"${ROLE}": "role",
"${USER}": "user",
+"${SPOUT_NAME}": "spout_name",
+"${SPOUT_SOURCE}": "spout_source",
}
dummy_formatted_url = url_format
for key, value in valid_parameters.items():
@@ -81,19 +91,29 @@ class Config(object):
# No error is thrown, so the format is valid.
return url_format
- def get_formatted_url(self, execution_state, formatter):
+ def get_formatted_url(self, formatter, execution_state, **additional):
"""
@param execution_state: The python dict representing JSON execution_state
+@param additional: additional kwargs to interpolate
@return Formatted viz url
"""
# Create the parameters based on execution state
valid_parameters = {
-"${CLUSTER}": execution_state["cluster"],
-"${ENVIRON}": execution_state["environ"],
-"${TOPOLOGY}": execution_state["jobna
[incubator-heron] branch master updated: Joshfischer/javadocs (#3302)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 20f438d Joshfischer/javadocs (#3302)
20f438d is described below
commit 20f438d597179db4d030ee791abd92c2cce44fa3
Author: Josh Fischer
AuthorDate: Mon Jul 1 11:24:00 2019 -0500
Joshfischer/javadocs (#3302)
* adding start to make file and javadoc script
* updating javadoc overview
---
.gitignore | 2 +
website2/website/Makefile | 5 ++
website2/website/scripts/javadocs-overview.html | 22 +++
website2/website/scripts/javadocs.sh| 78 +
4 files changed, 107 insertions(+)
diff --git a/.gitignore b/.gitignore
index 635f236..3b81340 100644
--- a/.gitignore
+++ b/.gitignore
@@ -129,6 +129,8 @@ com_crashlytics_export_strings.xml
website/node_modules/
website/public/
/bin/
+website2/website/public
+website2/website/static/api
# Visual Studio Code
.vscode
diff --git a/website2/website/Makefile b/website2/website/Makefile
new file mode 100644
index 000..3e82ae8
--- /dev/null
+++ b/website2/website/Makefile
@@ -0,0 +1,5 @@
+start:
+ yarn start
+
+javadocs:
+ @scripts/javadocs.sh
diff --git a/website2/website/scripts/javadocs-overview.html
b/website2/website/scripts/javadocs-overview.html
new file mode 100644
index 000..4d0a711
--- /dev/null
+++ b/website2/website/scripts/javadocs-overview.html
@@ -0,0 +1,22 @@
+
+
+
+Heron is a realtime, distributed, fault-tolerant stream processing
engine.
+
\ No newline at end of file
diff --git a/website2/website/scripts/javadocs.sh
b/website2/website/scripts/javadocs.sh
new file mode 100755
index 000..5232bbc
--- /dev/null
+++ b/website2/website/scripts/javadocs.sh
@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -e
+
+JAVADOC=javadoc
+FLAGS="-quiet -Xdoclint:none"
+
+HERON_ROOT_DIR=$(git rev-parse --show-toplevel)
+# for display on GitHub website
+JAVADOC_OUTPUT_DIR=$HERON_ROOT_DIR/website2/website/public/api/java
+# for display on local Hugo server
+JAVADOC_OUTPUT_LOCAL_DIR=$HERON_ROOT_DIR/website2/website/static/api
+GEN_PROTO_DIR=$HERON_ROOT_DIR/bazel-bin/heron/proto/_javac
+
+# The path of the customized landing page for the Javadocs
+OVERVIEW_HTML_FILE=$HERON_ROOT_DIR/website2/website/scripts/javadocs-overview.html
+
+# Check if this script is run with Travis flag
+if [ $# -eq 1 ] && [ $1 == "--travis" ]; then
+BAZEL_CMD="bazel --bazelrc=$HERON_ROOT_DIR/tools/travis/bazel.rc build"
+else
+BAZEL_CMD="bazel build"
+fi
+
+(cd $HERON_ROOT_DIR && $BAZEL_CMD \
+ `bazel query 'kind("java_library", "heron/...")'`\
+ `bazel query 'kind("java_test", "heron/...")'` \
+ `bazel query 'kind("java_library", "integration_test/...")'`)
+
+HERON_SRC_FILES=`find $HERON_ROOT_DIR -path "*/org/apache/*" -name "*.java"`
+BACKTYPE_SRC_FILES=`find $HERON_ROOT_DIR -path "*/backtype/storm/*" -name
"*.java"`
+APACHE_SRC_FILES=`find $HERON_ROOT_DIR -path "*/org/apache/storm/*" -name
"*.java"`
+GEN_FILES=`find $GEN_PROTO_DIR -name "*.java"`
+
+rm -rf $JAVADOC_OUTPUT_DIR
+mkdir -p $JAVADOC_OUTPUT_DIR
+
+BIN_JARS=`find $HERON_ROOT_DIR/bazel-incubator-heron/_bin/. -name "*\.jar" |
tr '\n' ':'`
+GEN_JARS=`find $HERON_ROOT_DIR/bazel-genfiles/external/. -name "*\.jar" | tr
'\n' ':'`
+SCRIBE_JARS=`find $HERON_ROOT_DIR/bazel-bin/. -name
"libthrift_scribe_java.jar" | tr '\n' ':'`
+PROTO_JARS=`find $HERON_ROOT_DIR/bazel-bin/heron/proto/. -name "*\.jar" | tr
'\n' ':'`
+CLOSURE_CLASSES="$HERON_ROOT_DIR/bazel-bin/storm-compatibility/src/java/_javac/storm-compatibility-java/libstorm-compatibility-java_classes/."
+
+export CLASSPATH=$BIN_JARS:$GEN_JARS:$SCRIBE_JARS:$PROTO_JARS:$CLOSURE_CLASSES
+
+$JAVADOC $FLAGS \
+ -windowtitle "Hero
