[incubator-heron] branch nicknezis/travis-update updated (fa2147f -> 6c1e8a3)
This is an automated email from the ASF dual-hosted git repository. nicknezis pushed a change to branch nicknezis/travis-update in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. from fa2147f Adding `group: edge` back add 6c1e8a3 Adding google perftools back No new revisions were added by this update. Summary of changes: .travis.yml | 3 +++ 1 file changed, 3 insertions(+)
[GitHub] [incubator-heron] dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
URL:
https://github.com/apache/incubator-heron/issues/3502#issuecomment-605559205
@thinker0
The last modification.
My heron.aurora file is below, but error can be seen yet.
import textwrap
heron_core_release_uri = '{{CORE_PACKAGE_URI}}'
heron_topology_jar_uri = '{{TOPOLOGY_PACKAGE_URI}}'
core_release_file = "heron-centos.tar.gz"
topology_package_file = "topology.tar.gz"
# --- processes ---
fetch_heron_system = Process(
name = 'fetch_heron_system',
cmdline = 'hdfs dfs -get %s %s && tar zxf %s' % (heron_core_release_uri,
core_release_file, core_release_file)
)
fetch_user_package = Process(
name = 'fetch_user_package',
cmdline = 'hdfs dfs -get %s %s && tar zxf %s' % (heron_topology_jar_uri,
topology_package_file, topology_package_file)
)
Aurora error:
log stderr download
/bin/bash: hdfs: command not found
mesos error:
I0329 12:48:59.507381 23905 logging.cpp:201] INFO level logging started!
I0329 12:48:59.508378 23905 fetcher.cpp:562] Fetcher Info:
{"cache_directory":"/tmp/mesos/fetch/root","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"/usr/bin/thermos_executor"}}],"sandbox_directory":"/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-e50143f2-87fd-45f5-bd6f-00ab46dcf48c/runs/752698d1-c0e5-4f5f-be41-efa49aee3a12","stall_timeout":{"nanoseconds":600},"user":"root"}
I0329 12:48:59.547435 23905 fetcher.cpp:459] Fetching URI
'/usr/bin/thermos_executor'
I0329 12:48:59.547525 23905 fetcher.cpp:290] Fetching
'/usr/bin/thermos_executor' directly into the sandbox directory
I0329 12:48:59.566543 23905 fetcher.cpp:618] Fetched
'/usr/bin/thermos_executor' to
'/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-e50143f2-87fd-45f5-bd6f-00ab46dcf48c/runs/752698d1-c0e5-4f5f-be41-efa49aee3a12/thermos_executor'
I0329 12:48:59.566643 23905 fetcher.cpp:623] Successfully fetched all URIs
into
'/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-e50143f2-87fd-45f5-bd6f-00ab46dcf48c/runs/752698d1-c0e5-4f5f-be41-efa49aee3a12'
twitter.common.app debug: Initializing: twitter.common.log (Logging
subsystem.)
Writing log files to disk in
/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-e50143f2-87fd-45f5-bd6f-00ab46dcf48c/runs/752698d1-c0e5-4f5f-be41-efa49aee3a12
I0329 12:49:01.113678 23907 exec.cpp:162] Version: 1.1.0
I0329 12:49:01.123690 23914 exec.cpp:237] Executor registered on agent
e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1
Writing log files to disk in
/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-e50143f2-87fd-45f5-bd6f-00ab46dcf48c/runs/752698d1-c0e5-4f5f-be41-efa49aee3a12
ERROR] Regular plan unhealthy!
Traceback (most recent call last):
File
"/root/.pex/install/twitter.common.exceptions-0.3.7-py2-none-any.whl.f6376bcca9bfda5eba4396de2676af5dfe36237d/twitter.common.exceptions-0.3.7-py2-none-any.whl/twitter/common/exceptions/__init__.py",
line 126, in _excepting_run
self.__real_run(*args, **kw)
File
"/root/.pex/install/twitter.common.concurrent-0.3.7-py2-none-any.whl.f1ab836a5554c86d07fa3f075905c95fb20c78dd/twitter.common.concurrent-0.3.7-py2-none-any.whl/twitter/common/concurrent/deferred.py",
line 42, in run
self._closure()
File
"/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-e50143f2-87fd-45f5-bd6f-00ab46dcf48c/runs/752698d1-c0e5-4f5f-be41-efa49aee3a12/thermos_executor/apache/aurora/executor/common/announcer.py",
line 269, in stop
AttributeError: 'NoneType' object has no attribute 'stop'
twitter.common.app debug: Shutting application down.
twitter.common.app debug: Running exit function for twitter.common.log
(Logging subsystem.)
twitter.common.app debug: Finishing up module teardown.
twitter.common.app debug: Active thread: <_MainThread(MainThread, started
140524700247872)>
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon): <_DummyThread(Dummy-2,
started daemon 140524538636032)>
twitter.common.app debug: Active thread (daemon):
[GitHub] [incubator-heron] dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up. URL: https://github.com/apache/incubator-heron/issues/3502#issuecomment-605557965 > curl /heron/ your meanings is that using hdfs to get . ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-heron] dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up. URL: https://github.com/apache/incubator-heron/issues/3502#issuecomment-605557877 > curl Sorry, I can not understand what is your meanings. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-heron] thinker0 commented on issue #3502: can not submit to aurora, it seems halt up.
thinker0 commented on issue #3502: can not submit to aurora, it seems halt up. URL: https://github.com/apache/incubator-heron/issues/3502#issuecomment-605557425 ``` curl /heron/ ``` to ``` hdfs get ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-heron] dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
URL:
https://github.com/apache/incubator-heron/issues/3502#issuecomment-605556985
some my config files:
upload.yaml file:
heron.class.uploader:
"org.apache.heron.uploader.hdfs.HdfsUploader"
heron.uploader.hdfs.config.directory: "/usr/local/hadoop/etc/hadoop"
# # heron.uploader.hdfs.topologies.directory.uri:
hdfs://heron/topologies/${CLUSTER}
heron.uploader.hdfs.topologies.directory.uri: "/heron/topologies/${CLUSTER}"
client.yaml:
# location of the core package
heron.package.core.uri: "/heron/dist/heron-core.tar.gz"
# Whether role/env is required to submit a topology. Default value is False.
heron.config.is.role.required: True
heron.config.is.env.required:True
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
URL:
https://github.com/apache/incubator-heron/issues/3502#issuecomment-605556376
> ```
> import textwrap
> ```
>
> or
>
> Remove
>
> ```
> textwrap.dedent
> ```
I missed this line.
Then I add it , the topology can not be run well.
aurora error:
+ curl /heron/dist/heron-core.tar.gz -o heron-centos.tar.gz
curl: (3) malformed
mesos error:
I0329 12:12:03.846160 23213 logging.cpp:201] INFO level logging started!
I0329 12:12:03.846814 23213 fetcher.cpp:562] Fetcher Info:
{"cache_directory":"/tmp/mesos/fetch/root","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"/usr/bin/thermos_executor"}}],"sandbox_directory":"/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-cebe9de4-50dc-4790-93e2-43bcd550ce6f/runs/7bd1c130-001e-452e-b2af-dc283269288f","stall_timeout":{"nanoseconds":600},"user":"root"}
I0329 12:12:03.870548 23213 fetcher.cpp:459] Fetching URI
'/usr/bin/thermos_executor'
I0329 12:12:03.870640 23213 fetcher.cpp:290] Fetching
'/usr/bin/thermos_executor' directly into the sandbox directory
I0329 12:12:03.880676 23213 fetcher.cpp:618] Fetched
'/usr/bin/thermos_executor' to
'/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-cebe9de4-50dc-4790-93e2-43bcd550ce6f/runs/7bd1c130-001e-452e-b2af-dc283269288f/thermos_executor'
I0329 12:12:03.880781 23213 fetcher.cpp:623] Successfully fetched all URIs
into
'/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-cebe9de4-50dc-4790-93e2-43bcd550ce6f/runs/7bd1c130-001e-452e-b2af-dc283269288f'
twitter.common.app debug: Initializing: twitter.common.log (Logging
subsystem.)
Writing log files to disk in
/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-cebe9de4-50dc-4790-93e2-43bcd550ce6f/runs/7bd1c130-001e-452e-b2af-dc283269288f
I0329 12:12:05.558152 23215 exec.cpp:162] Version: 1.1.0
I0329 12:12:05.566694 23227 exec.cpp:237] Executor registered on agent
e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1
Writing log files to disk in
/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-cebe9de4-50dc-4790-93e2-43bcd550ce6f/runs/7bd1c130-001e-452e-b2af-dc283269288f
ERROR] Regular plan unhealthy!
Traceback (most recent call last):
File
"/root/.pex/install/twitter.common.exceptions-0.3.7-py2-none-any.whl.f6376bcca9bfda5eba4396de2676af5dfe36237d/twitter.common.exceptions-0.3.7-py2-none-any.whl/twitter/common/exceptions/__init__.py",
line 126, in _excepting_run
self.__real_run(*args, **kw)
File
"/root/.pex/install/twitter.common.concurrent-0.3.7-py2-none-any.whl.f1ab836a5554c86d07fa3f075905c95fb20c78dd/twitter.common.concurrent-0.3.7-py2-none-any.whl/twitter/common/concurrent/deferred.py",
line 42, in run
self._closure()
File
"/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-StreamletCloneTopology-0-cebe9de4-50dc-4790-93e2-43bcd550ce6f/runs/7bd1c130-001e-452e-b2af-dc283269288f/thermos_executor/apache/aurora/executor/common/announcer.py",
line 269, in stop
AttributeError: 'NoneType' object has no attribute 'stop'
twitter.common.app debug: Shutting application down.
twitter.common.app debug: Running exit function for twitter.common.log
(Logging subsystem.)
twitter.common.app debug: Finishing up module teardown.
twitter.common.app debug: Active thread: <_MainThread(MainThread, started
140638577588032)>
twitter.common.app debug: Active thread (daemon): <_DummyThread(Dummy-3,
started daemon 140638357231360)>
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon): <_DummyThread(Dummy-2,
started daemon 140638374016768)>
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Exiting cleanly.
[GitHub] [incubator-heron] thinker0 commented on issue #3502: can not submit to aurora, it seems halt up.
thinker0 commented on issue #3502: can not submit to aurora, it seems halt up. URL: https://github.com/apache/incubator-heron/issues/3502#issuecomment-60062 ``` import textwrap ``` or Remove ``` textwrap.dedent ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-heron] thinker0 edited a comment on issue #3502: can not submit to aurora, it seems halt up.
thinker0 edited a comment on issue #3502: can not submit to aurora, it seems
halt up.
URL:
https://github.com/apache/incubator-heron/issues/3502#issuecomment-605466291
@dttlgotv
See Mesos or Aurora stderr, stdout log.
```
hdfs dfs -get /heron/dist/heron-core.tar.gz heron-core.tar.gz && tar zxf
heron-core.tar.gz
```
work ?
Please refer to this in my aurora file.
```
import textwrap
heron_core_release_uri = '{{CORE_PACKAGE_URI}}'
heron_topology_jar_uri = '{{TOPOLOGY_PACKAGE_URI}}'
core_release_file = "heron-centos.tar.gz"
topology_package_file = "topology.tar.gz"
# --- processes ---
fetch_heron_system = Process(
name='fetch_heron_system',
cmdline=textwrap.dedent('''
set -x
curl %s -o %s && tar zxf %s && {
rm -f heron-centos.tar.gz
} && tar xvfz dist/heron-core.tar.gz && {
rm -f dist/heron-core.tar.gz
}
''') % (heron_core_release_uri, core_release_file, core_release_file)
)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
URL:
https://github.com/apache/incubator-heron/issues/3502#issuecomment-605553007
> @dttlgotv
> See Mesos or Aurora stderr, stdout log.
>
> ```
> hdfs dfs -get /heron/dist/heron-core.tar.gz heron-core.tar.gz && tar zxf
heron-core.tar.gz
> ```
>
> work ?
>
> Please refer to this in my aurora file.
>
> ```
> heron_core_release_uri = '{{CORE_PACKAGE_URI}}'
> heron_topology_jar_uri = '{{TOPOLOGY_PACKAGE_URI}}'
> core_release_file = "heron-centos.tar.gz"
> topology_package_file = "topology.tar.gz"
>
> # --- processes ---
> fetch_heron_system = Process(
> name='fetch_heron_system',
> cmdline=textwrap.dedent('''
> set -x
> curl %s -o %s && tar zxf %s && {
> rm -f heron-centos.tar.gz
> } && tar xvfz dist/heron-core.tar.gz && {
> rm -f dist/heron-core.tar.gz
> }
> ''') % (heron_core_release_uri, core_release_file, core_release_file)
> )
> ```
I just give two results (your reference and mine), please check my comment
.Thanks a lot
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
URL:
https://github.com/apache/incubator-heron/issues/3502#issuecomment-605552785
I used your reference:
heron_core_release_uri = '{{CORE_PACKAGE_URI}}'
heron_topology_jar_uri = '{{TOPOLOGY_PACKAGE_URI}}'
core_release_file = "heron-centos.tar.gz"
topology_package_file = "topology.tar.gz"
#heron_core_release_uri = '{{CORE_PACKAGE_URI}}'
#heron_topology_jar_uri = '{{TOPOLOGY_PACKAGE_URI}}'
#core_release_file = "heron-core.tar.gz"
#topology_package_file = "topology.tar.gz"
# --- processes ---
fetch_heron_system = Process(
name='fetch_heron_system',
cmdline=textwrap.dedent('''
set -x
curl %s -o %s && tar zxf %s && {
rm -f heron-centos.tar.gz
} && tar xvfz dist/heron-core.tar.gz && {
rm -f dist/heron-core.tar.gz
}
''') % (heron_core_release_uri, core_release_file, core_release_file)
)
fetch_user_package = Process(
name = 'fetch_user_package',
cmdline = 'hdfs dfs -get %s %s && tar zxf %s' % (heron_topology_jar_uri,
topology_package_file, topology_package_file)
)
result: mesos and aurora stderr can not be seen , perhaps the task is not be
scheduled. But I cmd this error can be seen below:
Error loading configuration: name 'textwrap' is not defined
[2020-03-29 11:15:44 +0800] [严重]
org.apache.heron.scheduler.aurora.AuroraCLIController: Failed to run process.
Command=[aurora, job, create, --wait-until, RUNNING, --bind,
CPUS_PER_CONTAINER=1.0, --bind,
EXECUTOR_BINARY=./heron-core/bin/heron-executor, --bind, ROLE=root, --bind,
TOPOLOGY_NAME=Test3Topology, --bind,
TOPOLOGY_PACKAGE_URI=/heron/topologies/aurora/Test3Topology-root-tag-0--688933785017338.tar.gz,
--bind, RAM_PER_CONTAINER=2357198848, --bind,
CORE_PACKAGE_URI=/heron/dist/heron-core.tar.gz, --bind, TIER=preemptible,
--bind, TOPOLOGY_ARGUMENTS=--topology-name=Test3Topology
--topology-id=Test3Topologye1439966-dc4f-47b9-9749-136375ba4528
--topology-defn-file=Test3Topology.defn
--state-manager-connection=127.0.0.1:2181 --state-manager-root=/heron
--state-manager-config-file=./heron-conf/statemgr.yaml
--tmaster-binary=./heron-core/bin/heron-tmaster
--stmgr-binary=./heron-core/bin/heron-stmgr
--metrics-manager-classpath=./heron-core/lib/metricsmgr/*
--instance-jvm-opts="" --classpath=heron-streamlet-examples.jar
--heron-internals-config-file=./heron-conf/heron_internals.yaml
--override-config-file=./heron-conf/override.yaml
--component-ram-map=random-sentences-source:209715200 --component-jvm-opts=""
--pkg-type=jar --topology-binary-file=heron-streamlet-examples.jar
--heron-java-home=/usr/lib/jvm/java-1.8.0-openjdk-amd64
--heron-shell-binary=./heron-core/bin/heron-shell --cluster=aurora --role=root
--environment=devel --instance-classpath=./heron-core/lib/instance/*
--metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
--scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
--python-instance-binary=./heron-core/bin/heron-python-instance
--cpp-instance-binary=./heron-core/bin/heron-cpp-instance
--metricscache-manager-classpath=./heron-core/lib/metricscachemgr/*
--metricscache-manager-mode=disabled --is-stateful=false
--checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
--stateful-config-file=./heron-conf/stateful.yaml
--checkpoint-manager-ram=1073741824 --health-manager-mode=disabled
--health-manager-classpath=./heron-core/lib/healthmgr/*, --bind,
NUM_CONTAINERS=2, --bind, CLUSTER=aurora, --bind, ENVIRON=devel, --bind,
DISK_PER_CONTAINER=13958643712, aurora/root/devel/Test3Topology,
/root/.heron/conf/aurora/heron.aurora, --verbose], STDOUT=null, STDERR=null
[2020-03-29 11:15:44 +0800] [严重]
org.apache.heron.scheduler.utils.LauncherUtils: Failed to invoke IScheduler as
library
[2020-03-29 11:15:44 +0800] [非常详细]
org.apache.curator.utils.DefaultTracerDriver: Trace:
DeleteBuilderImpl-Foreground - 20 ms
[2020-03-29 11:15:44 +0800] [信息]
org.apache.heron.statemgr.zookeeper.curator.CuratorStateManager: Deleted node
for path: /heron/executionstate/Test3Topology
[2020-03-29 11:15:44 +0800] [非常详细]
org.apache.curator.utils.DefaultTracerDriver: Trace:
DeleteBuilderImpl-Foreground - 18 ms
[2020-03-29 11:15:44 +0800] [信息]
org.apache.heron.statemgr.zookeeper.curator.CuratorStateManager: Deleted node
for path: /heron/packingplans/Test3Topology
[2020-03-29 11:15:44 +0800] [非常详细]
org.apache.curator.utils.DefaultTracerDriver: Trace:
DeleteBuilderImpl-Foreground - 21 ms
[2020-03-29 11:15:44 +0800] [信息]
org.apache.heron.statemgr.zookeeper.curator.CuratorStateManager: Deleted node
for path: /heron/topologies/Test3Topology
[2020-03-29 11:15:44 +0800] [详细] org.apache.heron.spi.utils.ShellUtils:
Running synced process: ``hadoop --config
[GitHub] [incubator-heron] dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
dttlgotv commented on issue #3502: can not submit to aurora, it seems halt up.
URL:
https://github.com/apache/incubator-heron/issues/3502#issuecomment-605551451
About cmd: hdfs dfs -get /heron/dist/heron-core.tar.gz heron-core.tar.gz &&
tar zxf heron-core.tar.gz
I tried this command on three cluster machine, it works well.
My heron.aurora is below:
heron_core_release_uri = '{{CORE_PACKAGE_URI}}'
heron_topology_jar_uri = '{{TOPOLOGY_PACKAGE_URI}}'
core_release_file = "heron-core.tar.gz"
topology_package_file = "topology.tar.gz"
# --- processes ---
fetch_heron_system = Process(
name = 'fetch_heron_system',
cmdline = 'hdfs dfs -get %s %s && tar zxf %s' % (heron_core_release_uri,
core_release_file, core_release_file)
)
fetch_user_package = Process(
name = 'fetch_user_package',
cmdline = 'hdfs dfs -get %s %s && tar zxf %s' % (heron_topology_jar_uri,
topology_package_file, topology_package_file)
)
result:
in aurara stderr:
log stderr download
/bin/bash: hdfs: command not found
mesos stderr:
I0329 10:58:19.809157 22467 logging.cpp:201] INFO level logging started!
I0329 10:58:19.809937 22467 fetcher.cpp:562] Fetcher Info:
{"cache_directory":"/tmp/mesos/fetch/root","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"/usr/bin/thermos_executor"}}],"sandbox_directory":"/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-Test3Topology-1-d2093f25-b263-4ada-9a33-823ae2ff5075/runs/bd2f2ba6-9be1-4243-b027-9ac8ddee447a","stall_timeout":{"nanoseconds":600},"user":"root"}
I0329 10:58:19.836925 22467 fetcher.cpp:459] Fetching URI
'/usr/bin/thermos_executor'
I0329 10:58:19.836985 22467 fetcher.cpp:290] Fetching
'/usr/bin/thermos_executor' directly into the sandbox directory
I0329 10:58:19.849915 22467 fetcher.cpp:618] Fetched
'/usr/bin/thermos_executor' to
'/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-Test3Topology-1-d2093f25-b263-4ada-9a33-823ae2ff5075/runs/bd2f2ba6-9be1-4243-b027-9ac8ddee447a/thermos_executor'
I0329 10:58:19.850029 22467 fetcher.cpp:623] Successfully fetched all URIs
into
'/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-Test3Topology-1-d2093f25-b263-4ada-9a33-823ae2ff5075/runs/bd2f2ba6-9be1-4243-b027-9ac8ddee447a'
twitter.common.app debug: Initializing: twitter.common.log (Logging
subsystem.)
Writing log files to disk in
/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-Test3Topology-1-d2093f25-b263-4ada-9a33-823ae2ff5075/runs/bd2f2ba6-9be1-4243-b027-9ac8ddee447a
I0329 10:58:21.379936 22469 exec.cpp:162] Version: 1.1.0
I0329 10:58:21.391768 22481 exec.cpp:237] Executor registered on agent
e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1
Writing log files to disk in
/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-Test3Topology-1-d2093f25-b263-4ada-9a33-823ae2ff5075/runs/bd2f2ba6-9be1-4243-b027-9ac8ddee447a
ERROR] Regular plan unhealthy!
Traceback (most recent call last):
File
"/root/.pex/install/twitter.common.exceptions-0.3.7-py2-none-any.whl.f6376bcca9bfda5eba4396de2676af5dfe36237d/twitter.common.exceptions-0.3.7-py2-none-any.whl/twitter/common/exceptions/__init__.py",
line 126, in _excepting_run
self.__real_run(*args, **kw)
File
"/root/.pex/install/twitter.common.concurrent-0.3.7-py2-none-any.whl.f1ab836a5554c86d07fa3f075905c95fb20c78dd/twitter.common.concurrent-0.3.7-py2-none-any.whl/twitter/common/concurrent/deferred.py",
line 42, in run
self._closure()
File
"/root/mesosdata/run/slaves/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-S1/frameworks/e2da2e47-de60-4a1c-a81e-4f14ac3cf16f-/executors/thermos-root-devel-Test3Topology-1-d2093f25-b263-4ada-9a33-823ae2ff5075/runs/bd2f2ba6-9be1-4243-b027-9ac8ddee447a/thermos_executor/apache/aurora/executor/common/announcer.py",
line 269, in stop
AttributeError: 'NoneType' object has no attribute 'stop'
twitter.common.app debug: Shutting application down.
twitter.common.app debug: Running exit function for twitter.common.log
(Logging subsystem.)
twitter.common.app debug: Finishing up module teardown.
twitter.common.app debug: Active thread: <_MainThread(MainThread, started
140676464191296)>
twitter.common.app debug: Active thread (daemon):
twitter.common.app debug: Active thread (daemon): <_DummyThread(Dummy-3,
started daemon 140676277581568)>
twitter.common.app debug: Active thread (daemon):
twitter.common.app
[incubator-heron] branch nicknezis/travis-update created (now fa2147f)
This is an automated email from the ASF dual-hosted git repository. nicknezis pushed a change to branch nicknezis/travis-update in repository https://gitbox.apache.org/repos/asf/incubator-heron.git. at fa2147f Adding `group: edge` back No new revisions were added by this update.
[GitHub] [incubator-heron] thinker0 opened a new pull request #3503: Fix copy Maven Artifacts permission error
thinker0 opened a new pull request #3503: Fix copy Maven Artifacts permission error URL: https://github.com/apache/incubator-heron/pull/3503 Fix cp command ``` Found Version Tag 0.20.2-SNAPSHOT Run Maven template for poms ... Removing heron-api-0.20.2-SNAPSHOT.pom Removing heron-kafka-bolt-0.20.2-SNAPSHOT.pom Removing heron-kafka-spout-0.20.2-SNAPSHOT.pom Removing heron-simulator-0.20.2-SNAPSHOT.pom Removing heron-spi-0.20.2-SNAPSHOT.pom Removing heron-storm-0.20.2-SNAPSHOT.pom Build directories for jars ... Copy heron-api artifacts ... cp: /Users/thinker0/heron-release/0.20.2-SNAPSHOT/heron-api/heron-api-0.20.2-SNAPSHOT.jar: Permission denied ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-heron] thinker0 commented on issue #3502: can not submit to aurora, it seems halt up.
thinker0 commented on issue #3502: can not submit to aurora, it seems halt up.
URL:
https://github.com/apache/incubator-heron/issues/3502#issuecomment-605466291
@dttlgotv
See Mesos or Aurora stderr, stdout log.
```
hdfs dfs -get /heron/dist/heron-core.tar.gz heron-core.tar.gz && tar zxf
heron-core.tar.gz
```
work ?
Please refer to this in my aurora file.
```
heron_core_release_uri = '{{CORE_PACKAGE_URI}}'
heron_topology_jar_uri = '{{TOPOLOGY_PACKAGE_URI}}'
core_release_file = "heron-centos.tar.gz"
topology_package_file = "topology.tar.gz"
# --- processes ---
fetch_heron_system = Process(
name='fetch_heron_system',
cmdline=textwrap.dedent('''
set -x
curl %s -o %s && tar zxf %s && {
rm -f heron-centos.tar.gz
} && tar xvfz dist/heron-core.tar.gz && {
rm -f dist/heron-core.tar.gz
}
''') % (heron_core_release_uri, core_release_file, core_release_file)
)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] joshfischer1108 merged pull request #3501: roll back docusaurus version
joshfischer1108 merged pull request #3501: roll back docusaurus version URL: https://github.com/apache/incubator-heron/pull/3501 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-heron] dttlgotv opened a new issue #3502: can not submit to aurora, it seems halt up.
dttlgotv opened a new issue #3502: can not submit to aurora, it seems halt up.
URL: https://github.com/apache/incubator-heron/issues/3502
DEBUG] Using auth module:
INFO] Creating job Test3Topology
DEBUG] Full configuration: JobConfiguration(instanceCount=2,
cronSchedule=None, cronCollisionPolicy=0, key=JobKey(environment=u'devel',
role=u'gxh', name=u'Test3Topology'), taskConfig=TaskConfig(isService=True,
contactEmail=None, taskLinks={}, tier=u'preemptible', mesosFetcherUris=None,
executorConfig=ExecutorConfig(data='{"environment": "devel",
"health_check_config": {"health_checker": {"http": {"expected_response_code":
200, "endpoint": "/health", "expected_response": "ok"}},
"min_consecutive_successes": 1, "initial_interval_secs": 30.0,
"max_consecutive_failures": 2, "timeout_secs": 5.0, "interval_secs": 10.0},
"name": "Test3Topology", "service": true, "max_task_failures": 1,
"cron_collision_policy": "KILL_EXISTING", "enable_hooks": false, "cluster":
"aurora", "task": {"processes": [{"daemon": false, "name":
"fetch_heron_system", "ephemeral": false, "max_failures": 1, "min_duration": 5,
"cmdline": "hdfs dfs -get /heron/dist/heron-core.tar.gz heron-core.tar.gz &&
tar zxf heron-core.tar.gz", "final": false}, {"daemon": false, "name":
"fetch_user_package", "ephemeral": false, "max_failures": 1, "min_duration": 5,
"cmdline": "hdfs dfs -get
/heron/topologies/aurora/Test3Topology-gxh-tag-0-922516477660846776.tar.gz
topology.tar.gz && tar zxf topology.tar.gz", "final": false}, {"daemon": false,
"name": "launch_heron_executor", "ephemeral": false, "max_failures": 1,
"min_duration": 5, "cmdline": "./heron-core/bin/heron-executor
--shard={{mesos.instance}} --master-port={{thermos.ports[port1]}}
--tmaster-controller-port={{thermos.ports[port2]}}
--tmaster-stats-port={{thermos.ports[port3]}}
--shell-port={{thermos.ports[http]}}
--metrics-manager-port={{thermos.ports[port4]}}
--scheduler-port={{thermos.ports[scheduler]}}
--metricscache-manager-master-port={{thermos.ports[metricscachemgr_masterport]}}
--metricscache-manager-stats-port={{thermos.ports[metricscachemgr_statsport]}}
--checkpoint-manager-port={{thermos.ports[ckptmgr_port]}}
--topology-name=Test3Topology
--topology-id=Test3Topology3dd4ac0f-b248-4dd9-a91d-6bc53dafb8c2
--topology-defn-file=Test3Topology.defn
--state-manager-connection=127.0.0.1:2181 --state-manager-root=/heron
--state-manager-config-file=./heron-conf/statemgr.yaml
--tmaster-binary=./heron-core/bin/heron-tmaster
--stmgr-binary=./heron-core/bin/heron-stmgr
--metrics-manager-classpath=./heron-core/lib/metricsmgr/*
--instance-jvm-opts=\\"\\" --classpath=heron-streamlet-examples.jar
--heron-internals-config-file=./heron-conf/heron_internals.yaml
--override-config-file=./heron-conf/override.yaml
--component-ram-map=random-sentences-source:209715200
--component-jvm-opts=\\"\\" --pkg-type=jar
--topology-binary-file=heron-streamlet-examples.jar
--heron-java-home=/usr/lib/jvm/java-1.8.0-openjdk-amd64
--heron-shell-binary=./heron-core/bin/heron-shell --cluster=aurora --role=gxh
--environment=devel --instance-classpath=./heron-core/lib/instance/*
--metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
--scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
--python-instance-binary=./heron-core/bin/heron-python-instance
--cpp-instance-binary=./heron-core/bin/heron-cpp-instance
--metricscache-manager-classpath=./heron-core/lib/metricscachemgr/*
--metricscache-manager-mode=disabled --is-stateful=false
--checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
--stateful-config-file=./heron-conf/stateful.yaml
--checkpoint-manager-ram=1073741824 --health-manager-mode=disabled
--health-manager-classpath=./heron-core/lib/healthmgr/*", "final": false},
{"daemon": false, "name": "discover_profiler_port", "ephemeral": false,
"max_failures": 1, "min_duration": 5, "cmdline": "echo
{{thermos.ports[yourkit]}} > yourkit.port", "final": false}], "name":
"setup_and_run", "finalization_wait": 30, "max_failures": 1, "max_concurrency":
0, "resources": {"gpu": 0, "disk": 13958643712, "ram": 2357198848, "cpu": 1.0},
"constraints": [{"order": ["fetch_heron_system", "fetch_user_package",
"launch_heron_executor", "discover_profiler_port"]}]}, "production": false,
"role": "gxh", "tier": "preemptible", "announce": {"primary_port": "http",
"portmap": {"health": "http"}}, "lifecycle": {"http":
{"graceful_shutdown_endpoint": "/quitquitquit", "port": "health",
"shutdown_endpoint": "/abortabortabort"}}, "priority": 0}',
name='AuroraExecutor'), requestedPorts=set([u'port4', u'http',
u'metricscachemgr_masterport', u'yourkit', u'metricscachemgr_statsport',
u'scheduler', u'ckptmgr_port', u'port2', u'port3', u'port1']),
maxTaskFailures=1, priority=0, ramMb=2248, job=JobKey(environment=u'devel',
role=u'gxh', name=u'Test3Topology'), production=False, diskMb=13312,
