[
https://issues.apache.org/jira/browse/PHOENIX-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939128#comment-16939128
]
Chinmay Kulkarni commented on PHOENIX-5333:
-------------------------------------------
First of all, sorry for the long comment :(
[~kadir] On a closer look, this seems to be related to splittable syscat
changes for child views to inherit indexes of its parent. [~tdsilva] any ideas
what might be going on here?
I created the following using a 4.14.3 client and 4.14.3 server:
* Data table (Z_BASE_TABLE):
{noformat}
CREATE TABLE IF NOT EXISTS Z_BASE_TABLE (ID INTEGER NOT NULL PRIMARY KEY, HOST
VARCHAR(10), FLAG BOOLEAN);
{noformat}
* View (Z_VIEW1)
{noformat}
CREATE VIEW Z_VIEW1 (col1 INTEGER, col2 INTEGER, col3 INTEGER, col4 INTEGER,
col5 INTEGER) AS SELECT * FROM Z_BASE_TABLE WHERE ID>10;
{noformat}
* Global index on data table (Z_INDEX)
{noformat}
CREATE INDEX Z_INDEX ON Z_BASE_TABLE(HOST);
{noformat}
* Global Index on Z_VIEW1 (V_I)
{noformat}
CREATE INDEX V_I ON Z_VIEW1(COL4);
{noformat}
Then I stopped HBase, replaced the server jar with 4.15.0 jar, restarted HBase
and then connected using a 4.15.0 client in sqlline (code for 4.15.0 client
matches this
[commit|https://github.com/ChinmaySKulkarni/phoenix/commit/a225bf5d80ffcdd1f7f0e78bed3cd11b192c933b]
in case you want to trace it). This triggered the upgradeSystemTables code
path, and failed to get a connection, with this exception:
{noformat}
$ ./bin/sqlline.py
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix: none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:
19/09/26 21:28:14 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
19/09/26 21:28:26 WARN query.ConnectionQueryServicesImpl: Starting restore of
SYSTEM:CATALOG using snapshot
SNAPSHOT_SYSTEM.CATALOG_4.14.x_TO_4.15.0_20190926212816 because upgrade failed
19/09/26 21:28:28 WARN query.ConnectionQueryServicesImpl: Successfully restored
SYSTEM:CATALOG using snapshot
SNAPSHOT_SYSTEM.CATALOG_4.14.x_TO_4.15.0_20190926212816
19/09/26 21:28:32 WARN query.ConnectionQueryServicesImpl: Successfully restored
and enabled SYSTEM:CATALOG using snapshot
SNAPSHOT_SYSTEM.CATALOG_4.14.x_TO_4.15.0_20190926212816
Error: ERROR 504 (42703): Undefined column. columnName=V_I.0:COL1
(state=42703,code=504)
org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703): Undefined
column. columnName=V_I.0:COL1
at
org.apache.phoenix.schema.PTableImpl.getColumnForColumnName(PTableImpl.java:1076)
at
org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:528)
at
org.apache.phoenix.compile.ExpressionCompiler.resolveColumn(ExpressionCompiler.java:368)
at
org.apache.phoenix.compile.WhereCompiler$WhereExpressionCompiler.resolveColumn(WhereCompiler.java:191)
at
org.apache.phoenix.compile.WhereCompiler$WhereExpressionCompiler.visit(WhereCompiler.java:177)
at
org.apache.phoenix.compile.WhereCompiler$WhereExpressionCompiler.visit(WhereCompiler.java:164)
at
org.apache.phoenix.parse.ColumnParseNode.accept(ColumnParseNode.java:56)
at
org.apache.phoenix.parse.CompoundParseNode.acceptChildren(CompoundParseNode.java:64)
at org.apache.phoenix.parse.CastParseNode.accept(CastParseNode.java:60)
at
org.apache.phoenix.parse.CompoundParseNode.acceptChildren(CompoundParseNode.java:64)
at
org.apache.phoenix.parse.ComparisonParseNode.accept(ComparisonParseNode.java:45)
at
org.apache.phoenix.parse.CompoundParseNode.acceptChildren(CompoundParseNode.java:64)
at org.apache.phoenix.parse.AndParseNode.accept(AndParseNode.java:47)
at
org.apache.phoenix.compile.WhereCompiler.compile(WhereCompiler.java:94)
at
org.apache.phoenix.util.IndexUtil.rewriteViewStatement(IndexUtil.java:549)
at
org.apache.phoenix.util.ViewUtil.addIndexesFromParent(ViewUtil.java:283)
at
org.apache.phoenix.util.ViewUtil.addDerivedColumnsAndIndexesFromParent(ViewUtil.java:531)
at
org.apache.phoenix.schema.MetaDataClient.addColumnsAndIndexesFromAncestors(MetaDataClient.java:859)
at
org.apache.phoenix.schema.MetaDataClient.addTableToCache(MetaDataClient.java:4456)
at
org.apache.phoenix.schema.MetaDataClient.addTableToCache(MetaDataClient.java:4452)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:684)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:533)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:525)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:521)
at
org.apache.phoenix.util.PhoenixRuntime.getTable(PhoenixRuntime.java:458)
at
org.apache.phoenix.util.UpgradeUtil.syncUpdateCacheFreqAllIndexes(UpgradeUtil.java:1366)
at
org.apache.phoenix.util.UpgradeUtil.syncTableAndIndexProperties(UpgradeUtil.java:1414)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.upgradeSystemTables(ConnectionQueryServicesImpl.java:3631)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:3113)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:3026)
at
org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:3026)
at
org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
at
org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:144)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:154)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:202)
at sqlline.Commands.connect(Commands.java:1165)
at sqlline.Commands.connect(Commands.java:1055)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:643)
at sqlline.SqlLine.initArgs(SqlLine.java:385)
at sqlline.SqlLine.begin(SqlLine.java:485)
at sqlline.SqlLine.start(SqlLine.java:233)
at sqlline.SqlLine.main(SqlLine.java:175)
{noformat}
The data table name is 'Z_BASE_TABLE'. Descriptor is:
{noformat}
'Z_BASE_TABLE', {TABLE_ATTRIBUTES => {coprocessor$1 =>
'|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2
=>
'|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
coprocessor$3 =>
'|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
coprocessor$4 =>
'|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
coprocessor$5 =>
'|org.apache.phoenix.hbase.index.IndexRegionObserver|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'},
{NAME => '0', BLOOMFILTER => 'NONE', DATA_BLOCK_ENCODING => 'FAST_DIFF'}
{noformat}
The global index on this table is called 'Z_INDEX'. Descriptor is:
{noformat}
'Z_INDEX', {TABLE_ATTRIBUTES => {PRIORITY => '1000', coprocessor$1 =>
'|org.apache.phoenix.index.GlobalIndexChecker|805306365|', coprocessor$2 =>
'|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$3
=>
'|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
coprocessor$4 =>
'|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
coprocessor$5 =>
'|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
METADATA => {'DATA_TABLE_NAME' => 'Z_BASE_TABLE'}}, {NAME => '0', BLOOMFILTER
=> 'NONE', DATA_BLOCK_ENCODING => 'FAST_DIFF'}
{noformat}
The descriptor for the physical view index table '_IDX_Z_BASE_TABLE' is:
{noformat}
'_IDX_Z_BASE_TABLE', {TABLE_ATTRIBUTES => {coprocessor$1 =>
'|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2
=>
'|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
coprocessor$3 =>
'|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
coprocessor$4 =>
'|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
coprocessor$5 =>
'|org.apache.phoenix.hbase.index.IndexRegionObserver|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder',
coprocessor$6 => '|org.apache.phoenix.index.GlobalIndexChecker|805306365|',
METADATA => {'IS_VIEW_INDEX_TABLE' => '\x01'}}, {NAME => '0', BLOOMFILTER =>
'NONE', DATA_BLOCK_ENCODING => 'FAST_DIFF'}
{noformat}
Later, I used the previous 4.14.3 client and dropped the view index (V_I) and
tried reconnecting with the 4.15.0 client, which failed with the following:
{noformat}
$ ./bin/sqlline.py
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix: none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:
19/09/26 21:51:25 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
19/09/26 21:51:34 WARN query.ConnectionQueryServicesImpl: Starting restore of
SYSTEM:CATALOG using snapshot
SNAPSHOT_SYSTEM.CATALOG_4.14.x_TO_4.15.0_20190926215127 because upgrade failed
19/09/26 21:51:36 WARN query.ConnectionQueryServicesImpl: Successfully restored
SYSTEM:CATALOG using snapshot
SNAPSHOT_SYSTEM.CATALOG_4.14.x_TO_4.15.0_20190926215127
19/09/26 21:51:41 WARN query.ConnectionQueryServicesImpl: Successfully restored
and enabled SYSTEM:CATALOG using snapshot
SNAPSHOT_SYSTEM.CATALOG_4.14.x_TO_4.15.0_20190926215127
Error: ERROR 504 (42703): Undefined column. columnName=Z_VIEW1#Z_INDEX.0:COL1
(state=42703,code=504)
org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703): Undefined
column. columnName=Z_VIEW1#Z_INDEX.0:COL1
at
org.apache.phoenix.schema.PTableImpl.getColumnForColumnName(PTableImpl.java:1076)
at
org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:528)
at
org.apache.phoenix.compile.ExpressionCompiler.resolveColumn(ExpressionCompiler.java:368)
at
org.apache.phoenix.compile.WhereCompiler$WhereExpressionCompiler.resolveColumn(WhereCompiler.java:191)
at
org.apache.phoenix.compile.WhereCompiler$WhereExpressionCompiler.visit(WhereCompiler.java:177)
at
org.apache.phoenix.compile.WhereCompiler$WhereExpressionCompiler.visit(WhereCompiler.java:164)
at
org.apache.phoenix.parse.ColumnParseNode.accept(ColumnParseNode.java:56)
at
org.apache.phoenix.parse.CompoundParseNode.acceptChildren(CompoundParseNode.java:64)
at org.apache.phoenix.parse.CastParseNode.accept(CastParseNode.java:60)
at
org.apache.phoenix.parse.CompoundParseNode.acceptChildren(CompoundParseNode.java:64)
at
org.apache.phoenix.parse.ComparisonParseNode.accept(ComparisonParseNode.java:45)
at
org.apache.phoenix.parse.CompoundParseNode.acceptChildren(CompoundParseNode.java:64)
at org.apache.phoenix.parse.AndParseNode.accept(AndParseNode.java:47)
at
org.apache.phoenix.compile.WhereCompiler.compile(WhereCompiler.java:94)
at
org.apache.phoenix.util.IndexUtil.rewriteViewStatement(IndexUtil.java:549)
at
org.apache.phoenix.util.ViewUtil.addIndexesFromParent(ViewUtil.java:283)
at
org.apache.phoenix.util.ViewUtil.addDerivedColumnsAndIndexesFromParent(ViewUtil.java:531)
at
org.apache.phoenix.schema.MetaDataClient.addColumnsAndIndexesFromAncestors(MetaDataClient.java:859)
at
org.apache.phoenix.schema.MetaDataClient.addTableToCache(MetaDataClient.java:4456)
at
org.apache.phoenix.schema.MetaDataClient.addTableToCache(MetaDataClient.java:4452)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:684)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:533)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:525)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:521)
at
org.apache.phoenix.util.PhoenixRuntime.getTable(PhoenixRuntime.java:458)
at
org.apache.phoenix.util.UpgradeUtil.syncUpdateCacheFreqAllIndexes(UpgradeUtil.java:1366)
at
org.apache.phoenix.util.UpgradeUtil.syncTableAndIndexProperties(UpgradeUtil.java:1414)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.upgradeSystemTables(ConnectionQueryServicesImpl.java:3631)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:3113)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:3026)
at
org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:3026)
at
org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
at
org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:144)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:154)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:202)
at sqlline.Commands.connect(Commands.java:1165)
at sqlline.Commands.connect(Commands.java:1055)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:643)
at sqlline.SqlLine.initArgs(SqlLine.java:385)
at sqlline.SqlLine.begin(SqlLine.java:485)
at sqlline.SqlLine.start(SqlLine.java:233)
at sqlline.SqlLine.main(SqlLine.java:175)
{noformat}
FWIW, after this I also dropped the Z_INDEX via the previous 4.14.3 client. Now
that there were no more indexes or view indexes left, I could get a connection
for the 4.15.0 client, which is what drove me to believe that there is some
index related incompatibility here.
FYI [~lhofhansl]
> A tool to upgrade existing tables/indexes to use self-consistent global
> indexes design
> --------------------------------------------------------------------------------------
>
> Key: PHOENIX-5333
> URL: https://issues.apache.org/jira/browse/PHOENIX-5333
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Swaroopa Kadam
> Assignee: Swaroopa Kadam
> Priority: Major
> Fix For: 4.15.0, 5.1.0, 4.14.3
>
> Attachments: PHOENIX-5333.4.x-HBase-1.3.v1.patch,
> PHOENIX-5333.4.x-HBase-1.3.v2.patch, PHOENIX-5333.master.v1.patch,
> PHOENIX-5333.master.v2.patch, PHOENIX-5333.master.v3.patch,
> PHOENIX-5333.master.v4.patch, PHOENIX-5333.master.v5.patch,
> PHOENIX-5333.master.v6.patch
>
> Time Spent: 20h 20m
> Remaining Estimate: 0h
>
> A tool to upgrade existing tables/indexes to use self-consistent global
> indexes design in PHOENIX-5156
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
