[GitHub] [incubator-heron] thinker0 closed issue #3445: Too many Prometheus metric names.
thinker0 closed issue #3445: Too many Prometheus metric names. URL: https://github.com/apache/incubator-heron/issues/3445 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-heron] branch master updated: Support custom metrics rules for PrometheusSink (#3493)
This is an automated email from the ASF dual-hosted git repository. nwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new b069e45 Support custom metrics rules for PrometheusSink (#3493) b069e45 is described below commit b069e450a78c2f77ba5427c3f1aceda1e881e9f2 Author: choi se AuthorDate: Thu Apr 2 14:14:28 2020 +0900 Support custom metrics rules for PrometheusSink (#3493) * Support Java 11 * config travis to use oracle jdk 11 * Java 11 support (#3399) * Support Java 11 * config travis to use oracle jdk 11 * Add check jdk version * Fix command arguments. Change insert gc_options Update list Fix gc-logging * Add missing parameter * typo * Add pause time * wip * Support jmx_exporter format configuration. * Fix checkstyle * Remove unused * Java 11 support (#3399) * Support Java 11 * config travis to use oracle jdk 11 * Add check jdk version * Fix command arguments. Change insert gc_options Update list Fix gc-logging * wip * Support jmx_exporter format configuration. * Fix checkstyle * Remove unused * Update kafkaOffset metrics * Add Rules * Make log/sink/consume Streamlet component support setName and setNumPartitions (#3459) * Patch to fix cppcheck with newer glibc (#3471) * Add documents for setting up a docker based development environment (#3475) * Improve concurrency for needed parts. (#3107) * Change concurrent Map * Change concurrent Map * HashMap changes for unneeded parts. * HashMap changes for unneeded parts. * Review changes * Changes HashMap for unneeded parts. * Improve concurrency for needed parts. * Remove unused imports. * Remove unused imports. * Remove unused imports. * Fix NPE (cherry picked from commit 545d3814b315c29d3e396309a2ededaad193ec32) * Fix WhitespaceAround * Add dummy Object * Fix ConstantName (cherry picked from commit 8d6d5067072e92d6e276f93e18297ddedc647c6c) * Update kafkaOffset metrics * Add Rules * Update line is longer than 100 characters * Update line is longer than 100 characters * Add attrNameSnakeCase or other metrics fix * fix checkstyle Co-authored-by: Ning Wang Co-authored-by: Ning Wang Co-authored-by: Nicholas Nezis --- .../heron/metricsmgr/sink/PrometheusSink.java | 236 ++--- .../heron/metricsmgr/sink/PrometheusSinkTests.java | 124 ++- 2 files changed, 324 insertions(+), 36 deletions(-) diff --git a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java index 0db442a..56a244b 100644 --- a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java +++ b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java @@ -20,10 +20,17 @@ package org.apache.heron.metricsmgr.sink; import java.io.IOException; +import java.util.ArrayList; import java.util.HashMap; +import java.util.List; import java.util.Map; +import java.util.Set; +import java.util.TreeMap; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; import java.util.logging.Level; import java.util.logging.Logger; +import java.util.regex.Matcher; import java.util.regex.Pattern; import com.google.common.cache.Cache; @@ -33,6 +40,9 @@ import org.apache.heron.spi.metricsmgr.metrics.MetricsInfo; import org.apache.heron.spi.metricsmgr.metrics.MetricsRecord; import org.apache.heron.spi.metricsmgr.sink.SinkContext; +import static java.lang.String.format; +import static org.apache.heron.metricsmgr.sink.PrometheusSink.Prometheus.sanitizeMetricName; + /** * A web sink that exposes and endpoint that Prometheus can scrape * @@ -57,6 +67,7 @@ public class PrometheusSink extends AbstractWebSink { // This is the cache that is used to serve the metrics private Cache> metricsCache; + private List rules = new ArrayList(); private String cluster; private String role; @@ -66,6 +77,26 @@ public class PrometheusSink extends AbstractWebSink { super(); } + private enum Type { +COUNTER, +GAUGE, +SUMMARY, +HISTOGRAM, +UNTYPED, + } + + private static class Rule { +public Pattern pattern; +public String name; +public String value; +public Double valueFactor = 1.0; +public String help; +public boolean attrNameSnakeCase; +public Type type =
[GitHub] [incubator-heron] nwangtw merged pull request #3493: Support custom metrics rules for PrometheusSink
nwangtw merged pull request #3493: Support custom metrics rules for PrometheusSink URL: https://github.com/apache/incubator-heron/pull/3493 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-heron] nwangtw commented on issue #3504: terrible resource is needed when submitting to aurora
nwangtw commented on issue #3504: terrible resource is needed when submitting to aurora URL: https://github.com/apache/incubator-heron/issues/3504#issuecomment-607623734 In aurora, all containers need to have the same size, and container 0 is mainly for tmaster. Therefore the key is to keep the container size under control. In your example, your cpu and memory usage could be lower if you have 2 containers and 1 core/1.5G per container. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-heron] branch nicknezis/bazel-2 updated: Removing Zookeeper autoreconf
This is an automated email from the ASF dual-hosted git repository. nicknezis pushed a commit to branch nicknezis/bazel-2 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/nicknezis/bazel-2 by this push: new c0cf358 Removing Zookeeper autoreconf c0cf358 is described below commit c0cf3582c3ef54e122b9421b28142acd8e136483 Author: Nicholas Nezis AuthorDate: Thu Apr 2 00:20:56 2020 -0400 Removing Zookeeper autoreconf --- third_party/zookeeper/zookeeper.BUILD | 1 - 1 file changed, 1 deletion(-) diff --git a/third_party/zookeeper/zookeeper.BUILD b/third_party/zookeeper/zookeeper.BUILD index d45e758..008cb5b 100644 --- a/third_party/zookeeper/zookeeper.BUILD +++ b/third_party/zookeeper/zookeeper.BUILD @@ -29,7 +29,6 @@ genrule( "mkdir -p $$TMP_DIR", "cp -R $$(pwd)/external/org_apache_zookeeper/* $$TMP_DIR", "cd $$TMP_DIR/zookeeper-client/zookeeper-client-c", -"autoreconf -f -i", "./configure --prefix=$$INSTALL_DIR --enable-shared=no", "make install", "rm -rf $$TMP_DIR",
[incubator-heron] branch nicknezis/bazel-2 updated: Trying out newer Ubuntu version
This is an automated email from the ASF dual-hosted git repository. nicknezis pushed a commit to branch nicknezis/bazel-2 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/nicknezis/bazel-2 by this push: new f553b93 Trying out newer Ubuntu version f553b93 is described below commit f553b93d85dfc9cd06dc968921a2111de9000310 Author: Nicholas Nezis AuthorDate: Thu Apr 2 00:10:18 2020 -0400 Trying out newer Ubuntu version --- .travis.yml | 31 +-- 1 file changed, 13 insertions(+), 18 deletions(-) diff --git a/.travis.yml b/.travis.yml index 3767679..3b49ccd 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,35 +1,30 @@ -group: edge +--- -sudo: required +group: edge -dist: trusty +dist: bionic language: java +os: linux + jdk: - oraclejdk11 addons: apt: -sources: - - ubuntu-toolchain-r-test - packages: - - gcc-4.8 - - g++-4.8 - - gcc-4.8-multilib - - g++-4.8-multilib - - python2.7 - - libcppunit-dev + - libtool-bin + - python-dev + - python-wheel - wget - - pkg-config - zip - zlib1g-dev - google-perftools - libgoogle-perftools-dev env: - - BAZEL_VERSION=2.0.0 CC=gcc-4.8 CXX=g++-4.8 CPP=cpp-4.8 CXXCPP=cpp-4.8 ENABLE_HEAPCHECK=1 + - BAZEL_VERSION=2.2.0 ENABLE_HEAPCHECK=1 before_install: # download and install bazel @@ -37,13 +32,13 @@ before_install: - chmod +x bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh - ./bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh --user -before_script: - - which gcc-4.8 +script: + - which gcc - gcc --version - - which g++-4.8 + - which g++ - g++ --version - which python - python -V - which python2.7 - python2.7 -V - - scripts/travis/ci.sh + - scripts/travis/ci.sh \ No newline at end of file
[GitHub] [incubator-heron] dttlgotv opened a new issue #3504: terrible resource is needed when submitting to aurora
dttlgotv opened a new issue #3504: terrible resource is needed when submitting to aurora URL: https://github.com/apache/incubator-heron/issues/3504 I write a simple example to test resource requirement to submit to aurora. It is very terrible that too more resource is needed. Please check below: my example: builder.newSource(() -> ThreadLocalRandom.current().nextInt(100)) .setName("random-sentences-source"); Config config = Config.newBuilder() .setNumContainers(1) .setPerContainerCpu(2) .setPerContainerRamInGigabytes(3) .build(); When I submit this topo to aurora, 4.5 cpu and 6.3G memory are allocated to this topology. It seems that Tmaster used 2.5G cpu and 3.3G memory. My question is how to limit tmaster resource requirement. It seems that Tmaster task using more resource. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-heron] branch master updated: Travis CI update to Ubuntu 18.04 (#3497)
This is an automated email from the ASF dual-hosted git repository. nicknezis pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/master by this push: new a6a4bf8 Travis CI update to Ubuntu 18.04 (#3497) a6a4bf8 is described below commit a6a4bf88adf819bdf32c65ae5cc783ae32d8ef15 Author: Nicholas Nezis AuthorDate: Wed Apr 1 20:01:05 2020 -0400 Travis CI update to Ubuntu 18.04 (#3497) Updated Travic CI to use Ubuntu Bionic 18.04 --- .travis.yml| 31 -- .../src/python/local_test_runner/test_template.py | 3 ++- 2 files changed, 13 insertions(+), 21 deletions(-) diff --git a/.travis.yml b/.travis.yml index 08dff22..6808ddd 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,34 +1,30 @@ -group: edge +--- -sudo: required +group: edge -dist: trusty +dist: bionic language: java +os: linux + jdk: - oraclejdk11 addons: apt: -sources: - - ubuntu-toolchain-r-test - packages: - - gcc-4.8 - - g++-4.8 - - gcc-4.8-multilib - - g++-4.8-multilib - - python2.7 + - libtool-bin + - python-dev + - python-wheel - wget - - pkg-config - zip - zlib1g-dev - google-perftools - libgoogle-perftools-dev env: - - BAZEL_VERSION=0.26.0 CC=gcc-4.8 CXX=g++-4.8 CPP=cpp-4.8 CXXCPP=cpp-4.8 ENABLE_HEAPCHECK=1 + - BAZEL_VERSION=0.26.0 ENABLE_HEAPCHECK=1 before_install: # download and install bazel @@ -36,15 +32,10 @@ before_install: - chmod +x bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh - ./bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh --user -before_script: - # install python module for wheel files - # required for python packaging - - sudo pip install wheel - script: - - which gcc-4.8 + - which gcc - gcc --version - - which g++-4.8 + - which g++ - g++ --version - which python - python -V diff --git a/integration_test/src/python/local_test_runner/test_template.py b/integration_test/src/python/local_test_runner/test_template.py index 3ad5fa5..349b64f 100644 --- a/integration_test/src/python/local_test_runner/test_template.py +++ b/integration_test/src/python/local_test_runner/test_template.py @@ -27,6 +27,7 @@ import logging import os import time from urllib.request import urlopen +import shutil import signal import subprocess from collections import namedtuple @@ -152,7 +153,7 @@ class TestTemplate(object): # move to read file. This guarantees contents will be put into the file the # spout is reading from atomically # which increases the determinism -os.rename('temp.txt', self.params['readFile']) +shutil.move('temp.txt', self.params['readFile']) def _check_results(self): """ get actual and expected result.
[GitHub] [incubator-heron] nicknezis merged pull request #3497: Travis CI update to Ubuntu 18.04
nicknezis merged pull request #3497: Travis CI update to Ubuntu 18.04 URL: https://github.com/apache/incubator-heron/pull/3497 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services