[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475675#comment-15475675
]
Samarth Jain commented on PHOENIX-3230:
---
We could still have the initializationException code path. We could be a little
smart and set initializationException when the exception is not an instance of
IOException. If it is an instance of IOException, then set it only if it is an
instance of DoNotRetryIOException.
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3230_nowhitespacediff.patch,
> PHOENIX-3230_v2_nowhitespacediff.patch
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
> at
>
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475627#comment-15475627
]
James Taylor commented on PHOENIX-3230:
---
+1. Nice work, [~samarthjain]. Just curious on the initializationException code
path - should we not being doing that, but doing it the way you've done for
this new exception?
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3230_nowhitespacediff.patch,
> PHOENIX-3230_v2_nowhitespacediff.patch
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
> at
>
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475128#comment-15475128
]
James Taylor commented on PHOENIX-3230:
---
Looks good, [~samarthjain]. If a test is possible, that'd be nice - maybe two
connections with different ConnectionQueryServicesImpl? If it's too difficult
(but not impossible), then maybe file another JIRA?
+1 with one minor nit: can you make UpgradeInProgressException derived from
SQLException and add it to SQLExceptionCode like others so that users can react
to it more easily (as it'll have a unique error code)?
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3230_nowhitespacediff.patch
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at
>
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474964#comment-15474964
]
Lars Hofhansl commented on PHOENIX-3230:
Patch looks good. No test? (probably hard given that this is a timing issue,
still worth thinking about)
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3230_nowhitespacediff.patch
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2282)
> at
>
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472300#comment-15472300
]
James Taylor commented on PHOENIX-3230:
---
And maybe use the INDEX_STATUS column (ideally renamed to just STATUS) and set
the STATUS to a new value of UPGRADING while upgrading and clear it afterwards.
Or something along those lines.
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2282)
> at
>
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472293#comment-15472293
]
James Taylor commented on PHOENIX-3230:
---
You could either have a polling loop, or you can fail the other connections
with an "Upgrade in progress" exception. Probably the latter is fine.
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2282)
> at
>
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472277#comment-15472277
]
Samarth Jain commented on PHOENIX-3230:
---
Actually, I am not sure if the checkAndPut approach will work. We want only one
client to run the upgrade and we want the other clients to wait for the upgrade
to complete. Simple checkAndPut won't accomplish that.
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2282)
> at
>
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472181#comment-15472181
]
Samarth Jain commented on PHOENIX-3230:
---
bq.
Another approach is to use an HBase checkAndPut on a new VERSION cell on the
row representing the SYSTEM.CATALOG table header. If a client does this and the
VERSION is already equal to the current version, then you don't do the upgrade
(since another client beat you to it).
I like this approach better. Relying on ColumnAlreadyExists exception and not
having to run rest of the upgrade when it happens will make the code a bit
brittle, IMHO.
This would also need to be just a client side change so it should be OK
backward compatibility wise too. Would have to be a little careful with how to
use the checkAndPut to ensure mutex behavior. Will post a patch since
explaining it in plain English would be tricky.
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
>
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472133#comment-15472133
]
James Taylor commented on PHOENIX-3230:
---
I think a simpler solution would be to use
ConnectionQueryServicesImpl.addColumn() with an addIfNotExists as false. Only
one client (the first) will be the one to succeed. The others will get a
ColumnAlreadyExistsException which can be ignored, but we should not do the
snapshot or other upgrade in this case.
Another approach is to use an HBase checkAndPut on a new VERSION cell on the
row representing the SYSTEM.CATALOG table header. If a client does this and the
VERSION is already equal to the current version, then you don't do the upgrade
(since another client beat you to it).
I don't think we need to manage ZK distributed locks.
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
>
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472097#comment-15472097
]
Samarth Jain commented on PHOENIX-3230:
---
When upgrading with multiple clients (different JVMs), we are running into race
conditions.
Client 1 trying to execute:
{code}
metaConnection = addColumnsIfNotExists(metaConnection,
PhoenixDatabaseMetaData.SYSTEM_CATALOG,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_8_0 - 1,
PhoenixDatabaseMetaData.AUTO_PARTITION_SEQ + " "
+
PVarchar.INSTANCE.getSqlTypeName());
{code}
Client 2 trying to execute:
{code}
createSnapshot(snapshotName,
sysCatalogTableName);
{code}
Client 2 then fails with:
{code}
java.sql.SQLException: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException:
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
ss=SNAPSHOT_SYSTEM.CATALOG_null_TO_4.8.0_20160907155025-0700
table=SYSTEM.CATALOG type=FLUSH } had an error. Procedure
SNAPSHOT_SYSTEM.CATALOG_null_TO_4.8.0_20160907155025-0700 { waiting=[]
done=[localhost,58539,1473287287332] }
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:342)
at
org.apache.hadoop.hbase.master.HMaster.isSnapshotDone(HMaster.java:3237)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:43294)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2149)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
at
org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via
localhost,58539,1473287287332:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
org.apache.hadoop.hbase.NotServingRegionException:
SYSTEM.CATALOG,,1473283460590.fe40df52aa069a8d4a3ee52e4b282e5c. is closing
at
org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
at
org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:332)
... 10 more
Caused by:
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
org.apache.hadoop.hbase.NotServingRegionException:
SYSTEM.CATALOG,,1473283460590.fe40df52aa069a8d4a3ee52e4b282e5c. is closing
at
org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:338)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:138)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:157)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:186)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:53)
... 4 more
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.createSnapshot(ConnectionQueryServicesImpl.java:2597)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2337)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:1)
at
org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2272)
at
org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:232)
at
org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:147)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at
org.apache.phoenix.end2end.PhoenixRuntimeIT.testConnection(PhoenixRuntimeIT.java:150)
at
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459098#comment-15459098
]
Mujtaba Chohan commented on PHOENIX-3230:
-
Just regular index on base table in 4.7. No indexes on views/local.
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:231)
> at
>
[jira] [Commented] (PHOENIX-3230) SYSTEM.CATALOG get restored from snapshot with multi-client connection
[
https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459079#comment-15459079
]
Samarth Jain commented on PHOENIX-3230:
---
[~mujtabachohan] - did you have any view indexes or local indexes created using
Phoenix 4.7 or before?
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> --
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
> Fix For: 4.8.1
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1
> then second connection fails with the following exception. This happens even
> if second connection is couple of seconds apart but within upgrade window.
> This is likely to happen in situation where pool of client machines all get
> upgraded to latest Phoenix version. After this exception, all clients will
> cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ
> VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this
> timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA
> BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG
> using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully
> restored SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully
> restored and enabled SYSTEM.CATALOG using snapshot
> SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2282)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:231)
> at
>
