Arafat Khan created HDDS-14046:
----------------------------------
Summary: ReconStorageContainerManagerFacade is not getting
initialized properly after upgrade
Key: HDDS-14046
URL: https://issues.apache.org/jira/browse/HDDS-14046
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Recon
Reporter: Arafat Khan
Assignee: Arafat Khan
Fix For: 2.2.0
The issue is a race condition during the Recon startup sequence. The
*{{ReconTaskStatusTableUpgradeAction}}* (which adds the
*{{last_task_run_status}}* column) was being executed _after_ the Recon
services were started.
Specifically, *{{ReconServer.start()}}* initializes and registers
{*}{{ReconTaskStatusMetrics}}{*}. The metrics system (or other background tasks
started by {*}{{start()}}{*}) immediately attempts to query the
*{{RECON_TASK_STATUS}}* table. Since the upgrade logic
({*}{{finalizeLayoutFeatures}}{*}) was scheduled to run _after_
{*}{{start()}}{*}, the query failed with *{{SQLSyntaxErrorException}}* because
the new column did not exist yet.
*The Error we are getting now -*
{code:java}
Caused by: java.sql.SQLSyntaxErrorException: Column
'RECON_TASK_STATUS.last_task_run_status' is either not in any table in the FROM
list or appears within a join specification and is outside the scope of the
join specification or appears in a HAVING clause and is not in the GROUP BY
list. If this is a CREATE or ALTER TABLE statement then
'RECON_TASK_STATUS.last_task_run_status' is not a column in the target table.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source) at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source) at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown
Source) at
org.apache.derby.impl.jdbc.EmbedPreparedStatement42.<init>(Unknown Source) at
org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at
org.jooq.impl.ProviderEnabledConnection.prepareStatement(ProviderEnabledConnection.java:109)
at
org.jooq.impl.SettingsEnabledConnection.prepareStatement(SettingsEnabledConnection.java:73)
at org.jooq.impl.AbstractResultQuery.prepare(AbstractResultQuery.java:239)
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:322) ... 49
moreCaused by: ERROR 42X04: Column 'RECON_TASK_STATUS.last_task_run_status' is
either not in any table in the FROM list or appears within a join specification
and is outside the scope of the join specification or appears in a HAVING
clause and is not in the GROUP BY list. If this is a CREATE or ALTER TABLE
statement then 'RECON_TASK_STATUS.last_task_run_status' is not a column in the
target table. at
org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at
org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at
org.apache.derby.impl.sql.compile.ColumnReference.bindExpression(Unknown
Source) at
org.apache.derby.impl.sql.compile.ColumnReference.bindExpression(Unknown
Source) at
org.apache.derby.impl.sql.compile.ResultColumn.bindExpression(Unknown Source)
at
org.apache.derby.impl.sql.compile.ResultColumnList.bindExpressions(Unknown
Source) at
org.apache.derby.impl.sql.compile.SelectNode.bindExpressions(Unknown Source) at
org.apache.derby.impl.sql.compile.DMLStatementNode.bindExpressions(Unknown
Source) at org.apache.derby.impl.sql.compile.DMLStatementNode.bind(Unknown
Source) at
org.apache.derby.impl.sql.compile.CursorNode.bindStatement(Unknown Source) at
org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source) at
org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source) at
org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown
Source) ... 58 more2025-11-24 06:52:10,562 ERROR
[main]-org.apache.hadoop.ozone.recon.ReconServer:
ReconStorageContainerManagerFacade is not initialized properly. Setting health
status to false and adding error code.2025-11-24 06:52:10,563 INFO
[main]-org.apache.hadoop.ozone.recon.ReconContext: Update healthStatus of Recon
from true to false.2025-11-24 06:52:10,563 ERROR
[main]-org.apache.hadoop.ozone.recon.ReconServer: OzoneManagerServiceProvider
is not initialized properly. Setting health status to false and adding error
code.2025-11-24 06:52:10,563 INFO
[main]-org.apache.hadoop.ozone.recon.ReconContext: Update healthStatus of Recon
from false to false. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
