[
https://issues.apache.org/jira/browse/IGNITE-9585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Ignatenko updated IGNITE-9585:
-----------------------------------
Description:
Teamcity build logs sometimes refer to remote node log files that aren't
present in build artifacts
([example|https://ci.ignite.apache.org/viewLog.html?buildTypeId=IgniteTests24Java8_StartNodes&buildId=1849937&branch_IgniteTests24Java8_StartNodes=%3Cdefault%3E]).
I managed to reproduce this on my machine (details below) and it looks like
typically the root cause of this is error message from
[StartNodeCallableImpl|https://github.com/apache/ignite/blob/master/modules/ssh/src/main/java/org/apache/ignite/internal/util/nodestart/StartNodeCallableImpl.java]
referring readers to file that doesn't exist (and it wasn't even created to
start with).
{code:java}
return new ClusterStartNodeResultImpl(spec.host(), false, "Remote
node could not start. " +
"See log for details: " + scriptOutputPath);
{code}
This is quite painful when one tries to investigate node launching failures
because the misleading message causes one to waste time investigating the
problem that doesn't exist (it appears as if log file was there but somehow
disappeared for some mysterious reason).
----
To reproduce the issue locally (on Ubuntu 16.04) one can do as follows: first,
modify file
[start-nodes-custom.sh|https://github.com/apache/ignite/blob/master/modules/core/src/test/bin/start-nodes-custom.sh]
by replacing {{ignite.sh}} with the name of script that doesn't exist, eg
{{noignite.sh}}. After that, execute unit test
[IgniteProjectionStartStopRestartSelfTest|https://github.com/apache/ignite/blob/master/modules/ssh/src/test/java/org/apache/ignite/internal/IgniteProjectionStartStopRestartSelfTest.java]
and study its results and logs.
You will find that {{testCustomScript}} fails - which is expected because name
of the script intended to be executed has changed to one that doesn't exist.
Also you will find that log file for respective node hasn't been created -
which is also expected because shell command fails before creating it. But in
the same time test log will refer to mentioned file as if it exists.
was:
Teamcity build logs sometimes refer to remote node log files that aren't
present in build artifacts
([example|https://ci.ignite.apache.org/viewLog.html?buildTypeId=IgniteTests24Java8_StartNodes&buildId=1849937&branch_IgniteTests24Java8_StartNodes=%3Cdefault%3E]).
I managed to reproduce this on my machine (details below) and it looks like
typically the root cause of this is error message from
[StartNodeCallableImpl|https://github.com/apache/ignite/blob/master/modules/ssh/src/main/java/org/apache/ignite/internal/util/nodestart/StartNodeCallableImpl.java]
referring readers to file that doesn't exist (and it wasn't even created to
start with).
{code:java}
return new ClusterStartNodeResultImpl(spec.host(), false, "Remote
node could not start. " +
"See log for details: " + scriptOutputPath);
{code}
This is quite painful when one tries to investigate node launching failures
because the misleading message causes one to waste time investigating the
problem that doesn't exist (it appears as if log file was there but somehow
disappeared for some mysterious reason).
----
To reproduce the issue locally one can do as follows: first, modify file
[start-nodes-custom.sh|https://github.com/apache/ignite/blob/master/modules/core/src/test/bin/start-nodes-custom.sh]
by replacing {{ignite.sh}} with the name of script that doesn't exist, eg
{{noignite.sh}}. After that, execute unit test
[IgniteProjectionStartStopRestartSelfTest|https://github.com/apache/ignite/blob/master/modules/ssh/src/test/java/org/apache/ignite/internal/IgniteProjectionStartStopRestartSelfTest.java]
and study its results and logs.
You will find that {{testCustomScript}} fails - which is expected because name
of the script intended to be executed has changed to one that doesn't exist.
Also you will find that log file for respective node hasn't been created -
which is also expected because shell command fails before creating it. But in
the same time test log will refer to mentioned file as if it exists.
> Error message sometimes refers nonexisting log file when remote node fails to
> start
> -----------------------------------------------------------------------------------
>
> Key: IGNITE-9585
> URL: https://issues.apache.org/jira/browse/IGNITE-9585
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.6
> Reporter: Oleg Ignatenko
> Assignee: Oleg Ignatenko
> Priority: Minor
> Labels: MakeTeamcityGreenAgain
> Fix For: 2.7
>
>
> Teamcity build logs sometimes refer to remote node log files that aren't
> present in build artifacts
> ([example|https://ci.ignite.apache.org/viewLog.html?buildTypeId=IgniteTests24Java8_StartNodes&buildId=1849937&branch_IgniteTests24Java8_StartNodes=%3Cdefault%3E]).
> I managed to reproduce this on my machine (details below) and it looks like
> typically the root cause of this is error message from
> [StartNodeCallableImpl|https://github.com/apache/ignite/blob/master/modules/ssh/src/main/java/org/apache/ignite/internal/util/nodestart/StartNodeCallableImpl.java]
> referring readers to file that doesn't exist (and it wasn't even created to
> start with).
> {code:java}
> return new ClusterStartNodeResultImpl(spec.host(), false, "Remote
> node could not start. " +
> "See log for details: " + scriptOutputPath);
> {code}
> This is quite painful when one tries to investigate node launching failures
> because the misleading message causes one to waste time investigating the
> problem that doesn't exist (it appears as if log file was there but somehow
> disappeared for some mysterious reason).
> ----
> To reproduce the issue locally (on Ubuntu 16.04) one can do as follows:
> first, modify file
> [start-nodes-custom.sh|https://github.com/apache/ignite/blob/master/modules/core/src/test/bin/start-nodes-custom.sh]
> by replacing {{ignite.sh}} with the name of script that doesn't exist, eg
> {{noignite.sh}}. After that, execute unit test
> [IgniteProjectionStartStopRestartSelfTest|https://github.com/apache/ignite/blob/master/modules/ssh/src/test/java/org/apache/ignite/internal/IgniteProjectionStartStopRestartSelfTest.java]
> and study its results and logs.
> You will find that {{testCustomScript}} fails - which is expected because
> name of the script intended to be executed has changed to one that doesn't
> exist. Also you will find that log file for respective node hasn't been
> created - which is also expected because shell command fails before creating
> it. But in the same time test log will refer to mentioned file as if it
> exists.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
