[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samarth Jain updated PHOENIX-3230:
----------------------------------
Attachment: PHOENIX-3230_nowhitespacediff.patch
Patch to make sure only client JVM is able to run the upgrade. Clients that
lose the race are thrown an UpgradeInProgressException. The
UpgradeInProgressException isn't marked as initException so that the clients
can keep retrying on their end to establish the first connection to the
cluster. I verified this by writing a simple test that keeps on requesting
connection in a loop when another client is concurrently running the upgrade.
Once the upgrade is done, the test was able to acquire connection successfully.
The UpgradeInProgressException stacktrace looks like this:
{code}
org.apache.phoenix.query.ConnectionQueryServicesImpl$UpgradeInProgressException:
Cluster is being upgraded from 4.7.x to 4.8.x. Please retry establishing
connection
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.acquireUpgradeMutex(ConnectionQueryServicesImpl.java:2768)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2341)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:1)
at
org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2278)
at
org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:232)
at
org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:147)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at
org.apache.phoenix.end2end.PhoenixRuntimeIT.testConnection(PhoenixRuntimeIT.java:151)
{code}
I also had to tweak the way we were looking up version string for system
catalog timestamps. It turns out when doing concurrent upgrades, we could be at
an intermediate timestamp. Using a navigable map to help store the
timestamp->version combo helps provide a range based lookup.
I haven't added the part to store the upgrade state in a STATUS column in
SYSTEM.CATALOG as I believe this mechanism ends up providing enough info to the
user.
[~jamestaylor] - please review.
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> ----------------------------------------------------------------------
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
> Reporter: Mujtaba Chohan
> Assignee: Samarth Jain
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3230_nowhitespacediff.patch
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:231)
> at
> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:144)
> at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
