[GitHub] [incubator-heron] thinker0 closed issue #3445: Too many Prometheus metric names.
thinker0 closed issue #3445: Too many Prometheus metric names. URL: https://github.com/apache/incubator-heron/issues/3445 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-heron] branch master updated: Support custom metrics rules for PrometheusSink (#3493)
This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new b069e45 Support custom metrics rules for PrometheusSink (#3493)
b069e45 is described below
commit b069e450a78c2f77ba5427c3f1aceda1e881e9f2
Author: choi se
AuthorDate: Thu Apr 2 14:14:28 2020 +0900
Support custom metrics rules for PrometheusSink (#3493)
* Support Java 11
* config travis to use oracle jdk 11
* Java 11 support (#3399)
* Support Java 11
* config travis to use oracle jdk 11
* Add check jdk version
* Fix command arguments.
Change insert gc_options
Update list
Fix gc-logging
* Add missing parameter
* typo
* Add pause time
* wip
* Support jmx_exporter format configuration.
* Fix checkstyle
* Remove unused
* Java 11 support (#3399)
* Support Java 11
* config travis to use oracle jdk 11
* Add check jdk version
* Fix command arguments.
Change insert gc_options
Update list
Fix gc-logging
* wip
* Support jmx_exporter format configuration.
* Fix checkstyle
* Remove unused
* Update kafkaOffset metrics
* Add Rules
* Make log/sink/consume Streamlet component support setName and
setNumPartitions (#3459)
* Patch to fix cppcheck with newer glibc (#3471)
* Add documents for setting up a docker based development environment
(#3475)
* Improve concurrency for needed parts. (#3107)
* Change concurrent Map
* Change concurrent Map
* HashMap changes for unneeded parts.
* HashMap changes for unneeded parts.
* Review changes
* Changes HashMap for unneeded parts.
* Improve concurrency for needed parts.
* Remove unused imports.
* Remove unused imports.
* Remove unused imports.
* Fix NPE
(cherry picked from commit 545d3814b315c29d3e396309a2ededaad193ec32)
* Fix WhitespaceAround
* Add dummy Object
* Fix ConstantName
(cherry picked from commit 8d6d5067072e92d6e276f93e18297ddedc647c6c)
* Update kafkaOffset metrics
* Add Rules
* Update line is longer than 100 characters
* Update line is longer than 100 characters
* Add attrNameSnakeCase or other metrics fix
* fix checkstyle
Co-authored-by: Ning Wang
Co-authored-by: Ning Wang
Co-authored-by: Nicholas Nezis
---
.../heron/metricsmgr/sink/PrometheusSink.java | 236 ++---
.../heron/metricsmgr/sink/PrometheusSinkTests.java | 124 ++-
2 files changed, 324 insertions(+), 36 deletions(-)
diff --git
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java
index 0db442a..56a244b 100644
---
a/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java
+++
b/heron/metricsmgr/src/java/org/apache/heron/metricsmgr/sink/PrometheusSink.java
@@ -20,10 +20,17 @@
package org.apache.heron.metricsmgr.sink;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.logging.Level;
import java.util.logging.Logger;
+import java.util.regex.Matcher;
import java.util.regex.Pattern;
import com.google.common.cache.Cache;
@@ -33,6 +40,9 @@ import org.apache.heron.spi.metricsmgr.metrics.MetricsInfo;
import org.apache.heron.spi.metricsmgr.metrics.MetricsRecord;
import org.apache.heron.spi.metricsmgr.sink.SinkContext;
+import static java.lang.String.format;
+import static
org.apache.heron.metricsmgr.sink.PrometheusSink.Prometheus.sanitizeMetricName;
+
/**
* A web sink that exposes and endpoint that Prometheus can scrape
*
@@ -57,6 +67,7 @@ public class PrometheusSink extends AbstractWebSink {
// This is the cache that is used to serve the metrics
private Cache> metricsCache;
+ private List rules = new ArrayList();
private String cluster;
private String role;
@@ -66,6 +77,26 @@ public class PrometheusSink extends AbstractWebSink {
super();
}
+ private enum Type {
+COUNTER,
+GAUGE,
+SUMMARY,
+HISTOGRAM,
+UNTYPED,
+ }
+
+ private static class Rule {
+public Pattern pattern;
+public String name;
+public String value;
+public Double valueFactor = 1.0;
+public String help;
+public boolean attrNameSnakeCase;
+public Type type =
[GitHub] [incubator-heron] nwangtw merged pull request #3493: Support custom metrics rules for PrometheusSink
nwangtw merged pull request #3493: Support custom metrics rules for PrometheusSink URL: https://github.com/apache/incubator-heron/pull/3493 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-heron] nwangtw commented on issue #3504: terrible resource is needed when submitting to aurora
nwangtw commented on issue #3504: terrible resource is needed when submitting to aurora URL: https://github.com/apache/incubator-heron/issues/3504#issuecomment-607623734 In aurora, all containers need to have the same size, and container 0 is mainly for tmaster. Therefore the key is to keep the container size under control. In your example, your cpu and memory usage could be lower if you have 2 containers and 1 core/1.5G per container. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-heron] branch nicknezis/bazel-2 updated: Removing Zookeeper autoreconf
This is an automated email from the ASF dual-hosted git repository. nicknezis pushed a commit to branch nicknezis/bazel-2 in repository https://gitbox.apache.org/repos/asf/incubator-heron.git The following commit(s) were added to refs/heads/nicknezis/bazel-2 by this push: new c0cf358 Removing Zookeeper autoreconf c0cf358 is described below commit c0cf3582c3ef54e122b9421b28142acd8e136483 Author: Nicholas Nezis AuthorDate: Thu Apr 2 00:20:56 2020 -0400 Removing Zookeeper autoreconf --- third_party/zookeeper/zookeeper.BUILD | 1 - 1 file changed, 1 deletion(-) diff --git a/third_party/zookeeper/zookeeper.BUILD b/third_party/zookeeper/zookeeper.BUILD index d45e758..008cb5b 100644 --- a/third_party/zookeeper/zookeeper.BUILD +++ b/third_party/zookeeper/zookeeper.BUILD @@ -29,7 +29,6 @@ genrule( "mkdir -p $$TMP_DIR", "cp -R $$(pwd)/external/org_apache_zookeeper/* $$TMP_DIR", "cd $$TMP_DIR/zookeeper-client/zookeeper-client-c", -"autoreconf -f -i", "./configure --prefix=$$INSTALL_DIR --enable-shared=no", "make install", "rm -rf $$TMP_DIR",
[incubator-heron] branch nicknezis/bazel-2 updated: Trying out newer Ubuntu version
This is an automated email from the ASF dual-hosted git repository.
nicknezis pushed a commit to branch nicknezis/bazel-2
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/nicknezis/bazel-2 by this push:
new f553b93 Trying out newer Ubuntu version
f553b93 is described below
commit f553b93d85dfc9cd06dc968921a2111de9000310
Author: Nicholas Nezis
AuthorDate: Thu Apr 2 00:10:18 2020 -0400
Trying out newer Ubuntu version
---
.travis.yml | 31 +--
1 file changed, 13 insertions(+), 18 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 3767679..3b49ccd 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,35 +1,30 @@
-group: edge
+---
-sudo: required
+group: edge
-dist: trusty
+dist: bionic
language: java
+os: linux
+
jdk:
- oraclejdk11
addons:
apt:
-sources:
- - ubuntu-toolchain-r-test
-
packages:
- - gcc-4.8
- - g++-4.8
- - gcc-4.8-multilib
- - g++-4.8-multilib
- - python2.7
- - libcppunit-dev
+ - libtool-bin
+ - python-dev
+ - python-wheel
- wget
- - pkg-config
- zip
- zlib1g-dev
- google-perftools
- libgoogle-perftools-dev
env:
- - BAZEL_VERSION=2.0.0 CC=gcc-4.8 CXX=g++-4.8 CPP=cpp-4.8 CXXCPP=cpp-4.8
ENABLE_HEAPCHECK=1
+ - BAZEL_VERSION=2.2.0 ENABLE_HEAPCHECK=1
before_install:
# download and install bazel
@@ -37,13 +32,13 @@ before_install:
- chmod +x bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh
- ./bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh --user
-before_script:
- - which gcc-4.8
+script:
+ - which gcc
- gcc --version
- - which g++-4.8
+ - which g++
- g++ --version
- which python
- python -V
- which python2.7
- python2.7 -V
- - scripts/travis/ci.sh
+ - scripts/travis/ci.sh
\ No newline at end of file
[GitHub] [incubator-heron] dttlgotv opened a new issue #3504: terrible resource is needed when submitting to aurora
dttlgotv opened a new issue #3504: terrible resource is needed when submitting
to aurora
URL: https://github.com/apache/incubator-heron/issues/3504
I write a simple example to test resource requirement to submit to aurora.
It is very terrible that too more resource is needed. Please check below:
my example:
builder.newSource(() -> ThreadLocalRandom.current().nextInt(100))
.setName("random-sentences-source");
Config config = Config.newBuilder()
.setNumContainers(1)
.setPerContainerCpu(2)
.setPerContainerRamInGigabytes(3)
.build();
When I submit this topo to aurora, 4.5 cpu and 6.3G memory are allocated to
this topology. It seems that Tmaster used 2.5G cpu and 3.3G memory.
My question is how to limit tmaster resource requirement. It seems that
Tmaster task using more resource.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-heron] branch master updated: Travis CI update to Ubuntu 18.04 (#3497)
This is an automated email from the ASF dual-hosted git repository.
nicknezis pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new a6a4bf8 Travis CI update to Ubuntu 18.04 (#3497)
a6a4bf8 is described below
commit a6a4bf88adf819bdf32c65ae5cc783ae32d8ef15
Author: Nicholas Nezis
AuthorDate: Wed Apr 1 20:01:05 2020 -0400
Travis CI update to Ubuntu 18.04 (#3497)
Updated Travic CI to use Ubuntu Bionic 18.04
---
.travis.yml| 31 --
.../src/python/local_test_runner/test_template.py | 3 ++-
2 files changed, 13 insertions(+), 21 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 08dff22..6808ddd 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,34 +1,30 @@
-group: edge
+---
-sudo: required
+group: edge
-dist: trusty
+dist: bionic
language: java
+os: linux
+
jdk:
- oraclejdk11
addons:
apt:
-sources:
- - ubuntu-toolchain-r-test
-
packages:
- - gcc-4.8
- - g++-4.8
- - gcc-4.8-multilib
- - g++-4.8-multilib
- - python2.7
+ - libtool-bin
+ - python-dev
+ - python-wheel
- wget
- - pkg-config
- zip
- zlib1g-dev
- google-perftools
- libgoogle-perftools-dev
env:
- - BAZEL_VERSION=0.26.0 CC=gcc-4.8 CXX=g++-4.8 CPP=cpp-4.8 CXXCPP=cpp-4.8
ENABLE_HEAPCHECK=1
+ - BAZEL_VERSION=0.26.0 ENABLE_HEAPCHECK=1
before_install:
# download and install bazel
@@ -36,15 +32,10 @@ before_install:
- chmod +x bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh
- ./bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh --user
-before_script:
- # install python module for wheel files
- # required for python packaging
- - sudo pip install wheel
-
script:
- - which gcc-4.8
+ - which gcc
- gcc --version
- - which g++-4.8
+ - which g++
- g++ --version
- which python
- python -V
diff --git a/integration_test/src/python/local_test_runner/test_template.py
b/integration_test/src/python/local_test_runner/test_template.py
index 3ad5fa5..349b64f 100644
--- a/integration_test/src/python/local_test_runner/test_template.py
+++ b/integration_test/src/python/local_test_runner/test_template.py
@@ -27,6 +27,7 @@ import logging
import os
import time
from urllib.request import urlopen
+import shutil
import signal
import subprocess
from collections import namedtuple
@@ -152,7 +153,7 @@ class TestTemplate(object):
# move to read file. This guarantees contents will be put into the file the
# spout is reading from atomically
# which increases the determinism
-os.rename('temp.txt', self.params['readFile'])
+shutil.move('temp.txt', self.params['readFile'])
def _check_results(self):
""" get actual and expected result.
[GitHub] [incubator-heron] nicknezis merged pull request #3497: Travis CI update to Ubuntu 18.04
nicknezis merged pull request #3497: Travis CI update to Ubuntu 18.04 URL: https://github.com/apache/incubator-heron/pull/3497 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
