[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549799#comment-16549799 ] Hudson commented on PHOENIX-3534: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #165 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/165/]) PHOENIX-3534 Support multi region SYSTEM.CATALOG table (Thomas D'Silva (tdsilva: rev 93fdd5bad22cde313c9f34fe7448dca44377a27c) * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/SplitSystemCatalogIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/IndexMetadataIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/MetaDataUtil.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java * (edit) phoenix-protocol/src/main/PTable.proto * (add) phoenix-core/src/test/java/org/apache/phoenix/coprocessor/MetaDataEndpointImplTest.java * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/SplitSystemCatalogTests.java * (add) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/TableInfo.java * (edit) pom.xml * (delete) phoenix-core/src/it/java/org/apache/phoenix/end2end/SaltedViewIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/compile/CreateTableCompiler.java * (edit) phoenix-protocol/src/main/MetaDataService.proto * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/generated/MetaDataProtos.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/PTable.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/rpc/UpdateCacheIT.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/execute/CorrelatePlanTest.java * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/StatsEnabledSplitSystemCatalogIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (add) phoenix-core/src/main/java/org/apache/phoenix/schema/ParentTableNotFoundException.java * (add) phoenix-core/src/main/java/org/apache/phoenix/compile/ColumnNameTrackingExpressionCompiler.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/TenantSpecificViewIndexIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilter.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/QueryDatabaseMetaDataIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/DelegateConnectionQueryServices.java * (add) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/WhereConstantParser.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/ExplainPlanWithStatsEnabledIT.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterMultiTenantTableWithViewsIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/DelegateColumn.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/PColumnImpl.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableIT.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/PhoenixDriverIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/SchemaUtil.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableWithViewsIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/PhoenixRuntime.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/coprocessor/StatisticsCollectionRunTrackerIT.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilterIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/TableProperty.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ViewIndexIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/MigrateSystemTablesToSystemNamespaceIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/generated/PTableProtos.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/parse/DropTableStatement.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataSplitPolicy.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/DropColumnIT.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogCreationOnConnectionIT.java * (edit)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549791#comment-16549791 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva closed the pull request at: https://github.com/apache/phoenix/pull/303 > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534-v2.patch, PHOENIX-3534-v3.patch, > PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544301#comment-16544301
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r202522040
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/schema/TableProperty.java ---
@@ -231,36 +231,26 @@ public Object getPTableValue(PTable table) {
private final SQLExceptionCode mutatingImmutablePropException;
private final boolean isValidOnView;
private final boolean isMutableOnView;
-private final boolean propagateToViews;
private TableProperty(String propertyName, boolean isMutable, boolean
isValidOnView, boolean isMutableOnView) {
-this(propertyName, COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY,
isMutable, CANNOT_ALTER_PROPERTY, isValidOnView, isMutableOnView, true);
-}
-
-private TableProperty(String propertyName, boolean isMutable, boolean
isValidOnView, boolean isMutableOnView, boolean propagateToViews) {
-this(propertyName, COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY,
isMutable, CANNOT_ALTER_PROPERTY, isValidOnView, isMutableOnView,
propagateToViews);
+this(propertyName, COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY,
isMutable, CANNOT_ALTER_PROPERTY, isValidOnView, isMutableOnView);
}
private TableProperty(String propertyName, SQLExceptionCode
colFamilySpecifiedException, boolean isMutable, boolean isValidOnView, boolean
isMutableOnView) {
-this(propertyName, colFamilySpecifiedException, isMutable,
CANNOT_ALTER_PROPERTY, isValidOnView, isMutableOnView, true);
+this(propertyName, colFamilySpecifiedException, isMutable,
CANNOT_ALTER_PROPERTY, isValidOnView, isMutableOnView);
}
private TableProperty(String propertyName, boolean isMutable, boolean
isValidOnView, boolean isMutableOnView, SQLExceptionCode isMutatingException) {
-this(propertyName, COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY,
isMutable, isMutatingException, isValidOnView, isMutableOnView, true);
+this(propertyName, COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY,
isMutable, isMutatingException, isValidOnView, isMutableOnView);
}
private TableProperty(String propertyName, SQLExceptionCode
colFamSpecifiedException, boolean isMutable, SQLExceptionCode
mutatingException, boolean isValidOnView, boolean isMutableOnView) {
-this(propertyName, colFamSpecifiedException, isMutable,
mutatingException, isValidOnView, isMutableOnView, true);
-}
-
-private TableProperty(String propertyName, SQLExceptionCode
colFamSpecifiedException, boolean isMutable, SQLExceptionCode
mutatingException, boolean isValidOnView, boolean isMutableOnView, boolean
propagateToViews) {
--- End diff --
I filed PHOENIX-4763 to fix this. We should be able to use the cell
timestamp to differentiate, still need to figure out how to expose this since
its the properties in PTable don't currently expose the timestamp.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534-v2.patch, PHOENIX-3534-v3.patch,
> PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542032#comment-16542032 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on the issue: https://github.com/apache/phoenix/pull/303 @JamesRTaylor Thanks for the feedback, I have updated the PR. I will get this committed shortly. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534-v2.patch, PHOENIX-3534-v3.patch, > PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541179#comment-16541179
]
Hadoop QA commented on PHOENIX-3534:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12931252/PHOENIX-3534-v3.patch
against master branch at commit aee568beb02cdf983bb10889902c338ea016e6c9.
ATTACHMENT ID: 12931252
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 29 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 release audit{color}. The applied patch generated 3 release
audit warnings (more than the master's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+conn.createStatement().execute("CREATE LOCAL INDEX " +
generateUniqueName() + " ON " + tableName + "(KV1)");
+assertEquals(4, getIndexOfPkColumn(phxConn,
IndexUtil.getIndexColumnName(null, "k2"), fullView2IndexName));
+assertEquals(5, getIndexOfPkColumn(phxConn,
IndexUtil.getIndexColumnName(null, "k3"), fullView2IndexName));
+assertEquals(4, getIndexOfPkColumn(phxConn,
IndexUtil.getIndexColumnName(null, "k2"), fullView3IndexName));
+assertEquals(5, getIndexOfPkColumn(phxConn,
IndexUtil.getIndexColumnName(null, "k3"), fullView3IndexName));
+String view1DDL = "CREATE VIEW " + view1 + " ( VIEW_COL1
DECIMAL(10,2), VIEW_COL2 CHAR(256)) AS SELECT * FROM " + baseTable;
+String divergedViewDDL = "CREATE VIEW " + divergedView + " (
VIEW_COL1 DECIMAL(10,2), VIEW_COL2 CHAR(256)) AS SELECT * FROM " + baseTable;
+String indexDDL = "CREATE INDEX " + divergedViewIndex + " ON " +
divergedView + " (V1) include (V3)";
+assertTableDefinition(tenant1Conn, view1, PTableType.VIEW,
baseTable, 0, 7, 5, "PK1", "V1", "V2", "V3", "KV", "PK2", "VIEW_COL1",
"VIEW_COL2");
+assertTableDefinition(tenant2Conn, divergedView, PTableType.VIEW,
baseTable, 1, 6, DIVERGED_VIEW_BASE_COLUMN_COUNT, "PK1", "V1", "V3", "PK2",
"VIEW_COL1", "VIEW_COL2");
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.MutableIndexIT
{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):
at
org.apache.phoenix.end2end.DeleteIT.testPointDeleteRowFromTableWithImmutableIndex(DeleteIT.java:403)
at
org.apache.phoenix.end2end.DeleteIT.testPointDeleteRowFromTableWithImmutableIndex(DeleteIT.java:376)
at
org.apache.phoenix.end2end.DefaultColumnValueIT.testDefaultColumnValue(DefaultColumnValueIT.java:66)
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1926//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1926//artifact/patchprocess/patchReleaseAuditWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1926//console
This message is automatically generated.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534-v2.patch, PHOENIX-3534-v3.patch,
> PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540649#comment-16540649
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201835494
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
---
@@ -372,6 +378,31 @@ public void testViewAndTableAndDrop() throws Exception
{
// drop table cascade should succeed
conn.createStatement().execute("DROP TABLE " + fullTableName + "
CASCADE");
+validateViewDoesNotExist(conn, fullViewName1);
+validateViewDoesNotExist(conn, fullViewName2);
+
+}
+
+@Test
+public void testRecreateDroppedTableWithChildViews() throws Exception {
--- End diff --
We write the parent->child link first, then if the table uses column
encoding we update the encoded column qualifiers on the parent table, and
finally use mutateRowsWithLocks to write the view metadata atomically.
We ignore views that can't be found (in case writing the child view
metadata fails).
If the metadata write fails and the table uses column encoding then we will
lose a few column qualifiers.
I'll add a test for this.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540504#comment-16540504
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201800952
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -1893,26 +1981,45 @@ public static void upgradeTable(PhoenixConnection
conn, String srcTable) throws
conn.commit();
}
conn.getQueryServices().clearTableFromCache(
-conn.getTenantId() == null ?
ByteUtil.EMPTY_BYTE_ARRAY : conn.getTenantId().getBytes(),
+tenantIdBytes,
index.getSchemaName().getBytes(),
index.getTableName().getBytes(),
PhoenixRuntime.getCurrentScn(readOnlyProps));
}
updateIndexesSequenceIfPresent(conn, table);
conn.commit();
-
} else {
throw new RuntimeException("Error: problem occured during
upgrade. Table is not upgraded successfully");
}
if (table.getType() == PTableType.VIEW) {
logger.info(String.format("Updating link information for
view '%s' ..", table.getTableName()));
updateLink(conn, oldPhysicalName,
newPhysicalTablename,table.getSchemaName(),table.getTableName());
conn.commit();
-
+
+// if the view is a first level child, then we need to
create the PARENT_TABLE link
+// that was overwritten by the PHYSICAL_TABLE link
--- End diff --
Yes this will make it so that the parent link row will be created correctly
when upgrading tables to be namespace mapped.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540498#comment-16540498 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r201799117 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1779,13 +2174,65 @@ public void createTable(RpcController controller, CreateTableRequest request, } } +// The mutations to create a table are written in the following order: +// 1. Write the child link as if the next two steps fail we +// ignore missing children while processing a parent +// 2. Update the encoded column qualifier for the parent table if its on a +// different region server (for tables that use column qualifier encoding) +// if the next step fails we end up wasting a few col qualifiers +// 3. Finally write the mutations to create the table + +// From 4.15 the parent->child links are stored in a separate table SYSTEM.CHILD_LINK +List childLinkMutations = MetaDataUtil.removeChildLinks(tableMetadata); --- End diff -- I filed PHOENIX-4810 and added a comment to reference this jira. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540416#comment-16540416
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201776089
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation
row, byte[][] schemaTableNames)
schemaTableNames[2] = tName;
}
}
-
+
@Override
public void createTable(RpcController controller, CreateTableRequest
request,
RpcCallback done) {
MetaDataResponse.Builder builder = MetaDataResponse.newBuilder();
byte[][] rowKeyMetaData = new byte[3][];
byte[] schemaName = null;
byte[] tableName = null;
+String fullTableName = null;
try {
int clientVersion = request.getClientVersion();
List tableMetadata =
ProtobufUtil.getMutations(request);
MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata,
rowKeyMetaData);
byte[] tenantIdBytes =
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
schemaName =
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
tableName =
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+fullTableName = SchemaUtil.getTableName(schemaName, tableName);
+// TODO before creating a table we need to see if the table
was previously created and then dropped
+// and clean up any parent->child links or child views
--- End diff --
Done.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540415#comment-16540415
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201776046
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -586,48 +590,359 @@ public void getTable(RpcController controller,
GetTableRequest request,
builder.setMutationTime(minNonZerodisableIndexTimestamp - 1);
}
}
-
-if (table.getTimeStamp() != tableTimeStamp) {
+// the PTable of views and indexes on views might get updated
because a column is added to one of
+// their parents (this won't change the timestamp)
+if (table.getType()!=PTableType.TABLE || table.getTimeStamp()
!= tableTimeStamp) {
builder.setTable(PTableImpl.toProto(table));
}
done.run(builder.build());
-return;
} catch (Throwable t) {
logger.error("getTable failed", t);
ProtobufUtil.setControllerException(controller,
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName),
t));
}
}
+/**
+ * Used to add the columns present the ancestor hierarchy to the
PTable of the given view or
+ * view index
+ * @param table PTable of the view or view index
+ * @param skipAddingIndexes if true the returned PTable won't include
indexes
+ * @param skipAddingParentColumns if true the returned PTable won't
include columns derived from
+ *ancestor tables
+ * @param lockedAncestorTable ancestor table table that is being
mutated (as we won't be able to
+ *resolve this table as its locked)
+ */
+private Pair
combineColumns(PTable table, long timestamp,
+int clientVersion, boolean skipAddingIndexes, boolean
skipAddingParentColumns,
+PTable lockedAncestorTable) throws SQLException, IOException {
+boolean hasIndexId = table.getViewIndexId() != null;
+if (table.getType() != PTableType.VIEW && !hasIndexId) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+}
+if (!skipAddingParentColumns) {
+table =
+addDerivedColumnsFromAncestors(table, timestamp,
clientVersion,
+lockedAncestorTable);
+if (table==null) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_NOT_FOUND);
+}
+// we need to resolve the indexes of views (to get ensure they
also have all the columns
+// derived from their ancestors)
+if (!skipAddingIndexes && !table.getIndexes().isEmpty()) {
+List indexes =
Lists.newArrayListWithExpectedSize(table.getIndexes().size());
+for (PTable index : table.getIndexes()) {
+byte[] tenantIdBytes =
+index.getTenantId() == null ?
ByteUtil.EMPTY_BYTE_ARRAY
+: index.getTenantId().getBytes();
+PTable latestIndex =
+doGetTable(tenantIdBytes,
index.getSchemaName().getBytes(),
+index.getTableName().getBytes(),
timestamp, null, clientVersion, true,
+false, lockedAncestorTable);
+if (latestIndex == null) {
+throw new TableNotFoundException(
+"Could not find index table while
combining columns "
++ index.getTableName().getString()
+ " with tenant id "
++ index.getTenantId());
+}
+indexes.add(latestIndex);
+}
+table = PTableImpl.makePTable(table, table.getTimeStamp(),
indexes);
+}
+}
+
+MetaDataProtos.MutationCode mutationCode =
+table != null ?
MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS
+: MetaDataProtos.MutationCode.TABLE_NOT_FOUND;
+return new Pair(table,
mutationCode);
+}
+
+
+private PTable addDerivedColumnsFromAncestors(PTable table, long
timestamp,
+int clientVersion, PTable
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540419#comment-16540419
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201776204
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1809,180 +2256,97 @@ public void createTable(RpcController controller,
CreateTableRequest request,
} catch (Throwable t) {
logger.error("createTable failed", t);
ProtobufUtil.setControllerException(controller,
-
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName),
t));
+ServerUtil.createIOException(fullTableName, t));
}
}
+ private void dropChildMetadata(byte[] schemaName, byte[] tableName,
byte[] tenantIdBytes)
+ throws IOException, SQLException,
ClassNotFoundException {
+ TableViewFinderResult childViewsResult = new
TableViewFinderResult();
+ findAllChildViews(tenantIdBytes, schemaName, tableName,
childViewsResult);
+ if (childViewsResult.hasViews()) {
+ for (TableInfo viewInfo :
childViewsResult.getResults()) {
+ byte[] viewTenantId = viewInfo.getTenantId();
+ byte[] viewSchemaName =
viewInfo.getSchemaName();
+ byte[] viewName = viewInfo.getTableName();
+ Properties props = new Properties();
+ if (viewTenantId != null && viewTenantId.length
!= 0)
+
props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB,
Bytes.toString(viewTenantId));
+ try (PhoenixConnection connection =
QueryUtil.getConnectionOnServer(env.getConfiguration())
+
.unwrap(PhoenixConnection.class)) {
+ MetaDataClient client = new
MetaDataClient(connection);
+ org.apache.phoenix.parse.TableName
viewTableName = org.apache.phoenix.parse.TableName
+
.create(Bytes.toString(viewSchemaName), Bytes.toString(viewName));
+ client.dropTable(
+ new
DropTableStatement(viewTableName, PTableType.VIEW, false, true, true));
+ }
+ }
+ }
+ }
+
private boolean execeededIndexQuota(PTableType tableType, PTable
parentTable) {
return PTableType.INDEX == tableType &&
parentTable.getIndexes().size() >= maxIndexesPerTable;
}
-
-private void findAllChildViews(Region region, byte[] tenantId, PTable
table,
-TableViewFinder result, long clientTimeStamp, int
clientVersion) throws IOException, SQLException {
-TableViewFinder currResult = findChildViews(region, tenantId,
table, clientVersion, false);
-result.addResult(currResult);
-for (ViewInfo viewInfo : currResult.getViewInfoList()) {
-byte[] viewtenantId = viewInfo.getTenantId();
-byte[] viewSchema = viewInfo.getSchemaName();
-byte[] viewTable = viewInfo.getViewName();
-byte[] tableKey = SchemaUtil.getTableKey(viewtenantId,
viewSchema, viewTable);
-ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
-PTable view = loadTable(env, tableKey, cacheKey,
clientTimeStamp, clientTimeStamp, clientVersion);
-if (view == null) {
-logger.warn("Found orphan tenant view row in
SYSTEM.CATALOG with tenantId:"
-+ Bytes.toString(tenantId) + ", schema:"
-+ Bytes.toString(viewSchema) + ", table:"
-+ Bytes.toString(viewTable));
-continue;
-}
-findAllChildViews(region, viewtenantId, view, result,
clientTimeStamp, clientVersion);
-}
-}
-
-// TODO use child link instead once splittable system catalog
(PHOENIX-3534) is implemented
-// and we have a separate table for links.
-private TableViewFinder findChildViews_deprecated(Region region,
byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst)
throws IOException {
-byte[] schemaName = table.getSchemaName().getBytes();
-byte[] tableName = table.getTableName().getBytes();
-boolean isMultiTenant = table.isMultiTenant();
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540420#comment-16540420
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201776288
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -447,7 +447,7 @@
static {
Collections.sort(FUNCTION_KV_COLUMNS, KeyValue.COMPARATOR);
}
-
+
--- End diff --
I modified the class level comment.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540417#comment-16540417
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201776136
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation
row, byte[][] schemaTableNames)
schemaTableNames[2] = tName;
}
}
-
+
@Override
public void createTable(RpcController controller, CreateTableRequest
request,
RpcCallback done) {
MetaDataResponse.Builder builder = MetaDataResponse.newBuilder();
byte[][] rowKeyMetaData = new byte[3][];
byte[] schemaName = null;
byte[] tableName = null;
+String fullTableName = null;
try {
int clientVersion = request.getClientVersion();
List tableMetadata =
ProtobufUtil.getMutations(request);
MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata,
rowKeyMetaData);
byte[] tenantIdBytes =
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
schemaName =
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
tableName =
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+fullTableName = SchemaUtil.getTableName(schemaName, tableName);
+// TODO before creating a table we need to see if the table
was previously created and then dropped
+// and clean up any parent->child links or child views
boolean isNamespaceMapped =
MetaDataUtil.isNameSpaceMapped(tableMetadata, GenericKeyValueBuilder.INSTANCE,
new ImmutableBytesWritable());
final IndexType indexType =
MetaDataUtil.getIndexType(tableMetadata, GenericKeyValueBuilder.INSTANCE,
new ImmutableBytesWritable());
+byte[] parentTenantId = null;
byte[] parentSchemaName = null;
byte[] parentTableName = null;
PTableType tableType =
MetaDataUtil.getTableType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new
ImmutableBytesWritable());
+ViewType viewType = MetaDataUtil.getViewType(tableMetadata,
GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable());
+
+// Load table to see if it already exists
+byte[] tableKey = SchemaUtil.getTableKey(tenantIdBytes,
schemaName, tableName);
+ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
+long clientTimeStamp =
MetaDataUtil.getClientTimeStamp(tableMetadata);
+PTable table = null;
+ try {
+ // Get as of latest timestamp so we can detect
if we have a newer table that already
+ // exists without making an additional query
+ table = loadTable(env, tableKey, cacheKey,
clientTimeStamp, HConstants.LATEST_TIMESTAMP,
+ clientVersion);
+ } catch (ParentTableNotFoundException e) {
+ dropChildMetadata(e.getParentSchemaName(),
e.getParentTableName(), e.getParentTenantId());
+ }
+if (table != null) {
+if (table.getTimeStamp() < clientTimeStamp) {
+// If the table is older than the client time stamp
and it's deleted,
+// continue
+if (!isTableDeleted(table)) {
+
builder.setReturnCode(MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+
builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis());
+builder.setTable(PTableImpl.toProto(table));
+done.run(builder.build());
+return;
+}
+} else {
+
builder.setReturnCode(MetaDataProtos.MutationCode.NEWER_TABLE_FOUND);
+
builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis());
+builder.setTable(PTableImpl.toProto(table));
+done.run(builder.build());
+return;
+}
+}
+
+ // check if the table was dropped, but had child views
that were have not yet
+ // been cleaned up by compaction
+ if
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540418#comment-16540418
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201776174
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1488,20 +1904,19 @@ public void createTable(RpcController controller,
CreateTableRequest request,
if (tableType == PTableType.VIEW) {
byte[][] parentSchemaTableNames = new byte[3][];
byte[][] parentPhysicalSchemaTableNames = new byte[3][];
-/*
- * For a view, we lock the base physical table row. For a
mapped view, there is
- * no link present to the physical table. So the
viewPhysicalTableRow is null
- * in that case.
- */
+ /*
+* For a mapped view, there is no link present
to the physical table. So the
+* viewPhysicalTableRow is null in that case.
+*/
--- End diff --
Fixed.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540414#comment-16540414
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201776009
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -586,48 +590,359 @@ public void getTable(RpcController controller,
GetTableRequest request,
builder.setMutationTime(minNonZerodisableIndexTimestamp - 1);
}
}
-
-if (table.getTimeStamp() != tableTimeStamp) {
+// the PTable of views and indexes on views might get updated
because a column is added to one of
+// their parents (this won't change the timestamp)
+if (table.getType()!=PTableType.TABLE || table.getTimeStamp()
!= tableTimeStamp) {
builder.setTable(PTableImpl.toProto(table));
}
done.run(builder.build());
-return;
} catch (Throwable t) {
logger.error("getTable failed", t);
ProtobufUtil.setControllerException(controller,
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName),
t));
}
}
+/**
+ * Used to add the columns present the ancestor hierarchy to the
PTable of the given view or
+ * view index
+ * @param table PTable of the view or view index
+ * @param skipAddingIndexes if true the returned PTable won't include
indexes
+ * @param skipAddingParentColumns if true the returned PTable won't
include columns derived from
+ *ancestor tables
+ * @param lockedAncestorTable ancestor table table that is being
mutated (as we won't be able to
+ *resolve this table as its locked)
+ */
+private Pair
combineColumns(PTable table, long timestamp,
+int clientVersion, boolean skipAddingIndexes, boolean
skipAddingParentColumns,
+PTable lockedAncestorTable) throws SQLException, IOException {
+boolean hasIndexId = table.getViewIndexId() != null;
+if (table.getType() != PTableType.VIEW && !hasIndexId) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+}
+if (!skipAddingParentColumns) {
+table =
+addDerivedColumnsFromAncestors(table, timestamp,
clientVersion,
+lockedAncestorTable);
+if (table==null) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_NOT_FOUND);
+}
+// we need to resolve the indexes of views (to get ensure they
also have all the columns
+// derived from their ancestors)
+if (!skipAddingIndexes && !table.getIndexes().isEmpty()) {
+List indexes =
Lists.newArrayListWithExpectedSize(table.getIndexes().size());
+for (PTable index : table.getIndexes()) {
+byte[] tenantIdBytes =
+index.getTenantId() == null ?
ByteUtil.EMPTY_BYTE_ARRAY
+: index.getTenantId().getBytes();
+PTable latestIndex =
+doGetTable(tenantIdBytes,
index.getSchemaName().getBytes(),
+index.getTableName().getBytes(),
timestamp, null, clientVersion, true,
+false, lockedAncestorTable);
+if (latestIndex == null) {
+throw new TableNotFoundException(
+"Could not find index table while
combining columns "
++ index.getTableName().getString()
+ " with tenant id "
++ index.getTenantId());
+}
+indexes.add(latestIndex);
+}
+table = PTableImpl.makePTable(table, table.getTimeStamp(),
indexes);
+}
+}
+
+MetaDataProtos.MutationCode mutationCode =
+table != null ?
MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS
+: MetaDataProtos.MutationCode.TABLE_NOT_FOUND;
+return new Pair(table,
mutationCode);
+}
+
+
+private PTable addDerivedColumnsFromAncestors(PTable table, long
timestamp,
+int clientVersion, PTable
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540284#comment-16540284
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201748315
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java ---
@@ -1957,6 +1968,17 @@ private PTable
createTableInternal(CreateTableStatement statement, byte[][] spli
linkStatement.setLong(6, parent.getSequenceNumber());
linkStatement.setString(7,
PTableType.INDEX.getSerializedValue());
linkStatement.execute();
+
+// Add row linking index table to parent table for indexes
on views
+if (parent.getType() == PTableType.VIEW) {
+ linkStatement =
connection.prepareStatement(CREATE_VIEW_INDEX_PARENT_LINK);
+ linkStatement.setString(1, tenantIdStr);
+ linkStatement.setString(2, schemaName);
+ linkStatement.setString(3, tableName);
+ linkStatement.setString(4,
parent.getName().getString());
+ linkStatement.setByte(5,
LinkType.VIEW_INDEX_PARENT_TABLE.getSerializedValue());
+ linkStatement.execute();
+}
--- End diff --
I created PHOENIX-4766 for this, I will add a comment referencing this JIRA
in createTableInternal().
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540272#comment-16540272
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201744675
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -2573,307 +2897,139 @@ else if (pkCount <= COLUMN_NAME_INDEX
return new
MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION,
EnvironmentEdgeManager.currentTimeMillis(), basePhysicalTable);
}
-ColumnOrdinalPositionUpdateList ordinalPositionList = new
ColumnOrdinalPositionUpdateList();
+//add the new columns to the child view
List viewPkCols = new
ArrayList<>(view.getPKColumns());
boolean addingExistingPkCol = false;
-int numCols = view.getColumns().size();
-// add the new columns to the child view
-for (PutWithOrdinalPosition p : columnPutsForBaseTable) {
-Put baseTableColumnPut = p.put;
+for (Put columnToBeAdded : columnPutsForBaseTable) {
PColumn existingViewColumn = null;
byte[][] rkmd = new byte[5][];
-getVarChars(baseTableColumnPut.getRow(), rkmd);
+getVarChars(columnToBeAdded.getRow(), rkmd);
String columnName =
Bytes.toString(rkmd[COLUMN_NAME_INDEX]);
-String columnFamily = rkmd[FAMILY_NAME_INDEX] == null ?
null : Bytes.toString(rkmd[FAMILY_NAME_INDEX]);
+String columnFamily =
+rkmd[FAMILY_NAME_INDEX] == null ? null
+: Bytes.toString(rkmd[FAMILY_NAME_INDEX]);
try {
-existingViewColumn = columnFamily == null ?
view.getColumnForColumnName(columnName) : view.getColumnFamily(
-
columnFamily).getPColumnForColumnName(columnName);
+existingViewColumn =
+columnFamily == null ?
view.getColumnForColumnName(columnName)
+: view.getColumnFamily(columnFamily)
+
.getPColumnForColumnName(columnName);
} catch (ColumnFamilyNotFoundException e) {
-// ignore since it means that the column family is not
present for the column to be added.
+// ignore since it means that the column family is not
present for the column to
+// be added.
} catch (ColumnNotFoundException e) {
// ignore since it means the column is not present in
the view
}
-
-boolean isPkCol = columnFamily == null;
-byte[] columnKey = getColumnKey(viewKey, columnName,
columnFamily);
+
+boolean isColumnToBeAddPkCol = columnFamily == null;
if (existingViewColumn != null) {
-MetaDataMutationResult result =
validateColumnForAddToBaseTable(existingViewColumn, baseTableColumnPut,
basePhysicalTable, isPkCol, view);
-if (result != null) {
-return result;
+if
(EncodedColumnsUtil.usesEncodedColumnNames(basePhysicalTable)
+&& !SchemaUtil.isPKColumn(existingViewColumn))
{
--- End diff --
The race condition with adding the same column to the base table and a
child view will be covered in PHOENIX-4799.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540261#comment-16540261
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201741221
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -2147,46 +2566,29 @@ private MetaDataMutationResult doDropTable(byte[]
key, byte[] tenantId, byte[] s
}
if (tableType == PTableType.TABLE || tableType ==
PTableType.SYSTEM) {
-// Handle any child views that exist
-TableViewFinder tableViewFinderResult =
findChildViews(region, tenantId, table, clientVersion, !isCascade);
-if (tableViewFinderResult.hasViews()) {
-if (isCascade) {
-if
(tableViewFinderResult.allViewsInMultipleRegions()) {
-// We don't yet support deleting a table with
views where SYSTEM.CATALOG has split and the
-// view metadata spans multiple regions
-return new
MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION,
-
EnvironmentEdgeManager.currentTimeMillis(), null);
-} else if
(tableViewFinderResult.allViewsInSingleRegion()) {
-// Recursively delete views - safe as all the
views as all in the same region
-for (ViewInfo viewInfo :
tableViewFinderResult.getViewInfoList()) {
-byte[] viewTenantId =
viewInfo.getTenantId();
-byte[] viewSchemaName =
viewInfo.getSchemaName();
-byte[] viewName = viewInfo.getViewName();
-byte[] viewKey =
SchemaUtil.getTableKey(viewTenantId, viewSchemaName, viewName);
-Delete delete = new Delete(viewKey,
clientTimeStamp);
-rowsToDelete.add(delete);
-acquireLock(region, viewKey, locks);
-MetaDataMutationResult result =
doDropTable(viewKey, viewTenantId, viewSchemaName,
-viewName, null, PTableType.VIEW,
rowsToDelete, invalidateList, locks,
-tableNamesToDelete,
sharedTablesToDelete, false, clientVersion);
-if (result.getMutationCode() !=
MutationCode.TABLE_ALREADY_EXISTS) { return result; }
-}
-}
-} else {
+// check to see if the table has any child views
+try (Table hTable =
+env.getTable(SchemaUtil.getPhysicalTableName(
+
PhoenixDatabaseMetaData.SYSTEM_CHILD_LINK_NAME_BYTES,
+env.getConfiguration( {
+boolean hasChildViews =
+ViewFinder.hasChildViews(hTable, tenantId,
schemaName, tableName,
+clientTimeStamp);
+if (hasChildViews && !isCascade) {
--- End diff --
I think we can handle the race condition in a similar way to how we handle
conflicting columns using checkAndMutate. I have updated PHOENIX-4799 to
include this case.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540256#comment-16540256
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r201739480
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1809,180 +2256,97 @@ public void createTable(RpcController controller,
CreateTableRequest request,
} catch (Throwable t) {
logger.error("createTable failed", t);
ProtobufUtil.setControllerException(controller,
-
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName),
t));
+ServerUtil.createIOException(fullTableName, t));
}
}
+ private void dropChildMetadata(byte[] schemaName, byte[] tableName,
byte[] tenantIdBytes)
+ throws IOException, SQLException,
ClassNotFoundException {
+ TableViewFinderResult childViewsResult = new
TableViewFinderResult();
+ findAllChildViews(tenantIdBytes, schemaName, tableName,
childViewsResult);
+ if (childViewsResult.hasViews()) {
+ for (TableInfo viewInfo :
childViewsResult.getResults()) {
+ byte[] viewTenantId = viewInfo.getTenantId();
+ byte[] viewSchemaName =
viewInfo.getSchemaName();
+ byte[] viewName = viewInfo.getTableName();
+ Properties props = new Properties();
+ if (viewTenantId != null && viewTenantId.length
!= 0)
+
props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB,
Bytes.toString(viewTenantId));
+ try (PhoenixConnection connection =
QueryUtil.getConnectionOnServer(env.getConfiguration())
+
.unwrap(PhoenixConnection.class)) {
+ MetaDataClient client = new
MetaDataClient(connection);
+ org.apache.phoenix.parse.TableName
viewTableName = org.apache.phoenix.parse.TableName
+
.create(Bytes.toString(viewSchemaName), Bytes.toString(viewName));
+ client.dropTable(
+ new
DropTableStatement(viewTableName, PTableType.VIEW, false, true, true));
+ }
+ }
+ }
+ }
+
private boolean execeededIndexQuota(PTableType tableType, PTable
parentTable) {
return PTableType.INDEX == tableType &&
parentTable.getIndexes().size() >= maxIndexesPerTable;
}
-
-private void findAllChildViews(Region region, byte[] tenantId, PTable
table,
-TableViewFinder result, long clientTimeStamp, int
clientVersion) throws IOException, SQLException {
-TableViewFinder currResult = findChildViews(region, tenantId,
table, clientVersion, false);
-result.addResult(currResult);
-for (ViewInfo viewInfo : currResult.getViewInfoList()) {
-byte[] viewtenantId = viewInfo.getTenantId();
-byte[] viewSchema = viewInfo.getSchemaName();
-byte[] viewTable = viewInfo.getViewName();
-byte[] tableKey = SchemaUtil.getTableKey(viewtenantId,
viewSchema, viewTable);
-ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
-PTable view = loadTable(env, tableKey, cacheKey,
clientTimeStamp, clientTimeStamp, clientVersion);
-if (view == null) {
-logger.warn("Found orphan tenant view row in
SYSTEM.CATALOG with tenantId:"
-+ Bytes.toString(tenantId) + ", schema:"
-+ Bytes.toString(viewSchema) + ", table:"
-+ Bytes.toString(viewTable));
-continue;
-}
-findAllChildViews(region, viewtenantId, view, result,
clientTimeStamp, clientVersion);
-}
-}
-
-// TODO use child link instead once splittable system catalog
(PHOENIX-3534) is implemented
-// and we have a separate table for links.
-private TableViewFinder findChildViews_deprecated(Region region,
byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst)
throws IOException {
-byte[] schemaName = table.getSchemaName().getBytes();
-byte[] tableName = table.getTableName().getBytes();
-boolean isMultiTenant = table.isMultiTenant();
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533119#comment-16533119
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200207759
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java ---
@@ -1957,6 +1968,17 @@ private PTable
createTableInternal(CreateTableStatement statement, byte[][] spli
linkStatement.setLong(6, parent.getSequenceNumber());
linkStatement.setString(7,
PTableType.INDEX.getSerializedValue());
linkStatement.execute();
+
+// Add row linking index table to parent table for indexes
on views
+if (parent.getType() == PTableType.VIEW) {
+ linkStatement =
connection.prepareStatement(CREATE_VIEW_INDEX_PARENT_LINK);
+ linkStatement.setString(1, tenantIdStr);
+ linkStatement.setString(2, schemaName);
+ linkStatement.setString(3, tableName);
+ linkStatement.setString(4,
parent.getName().getString());
+ linkStatement.setByte(5,
LinkType.VIEW_INDEX_PARENT_TABLE.getSerializedValue());
+ linkStatement.execute();
+}
--- End diff --
We need to update MetaDataClient.createTableInternal() to not include the
columns from the parent table in 4.15 so that we can remove the code in
MetaDataEndPointImpl that filters the columns. It's fine to do this in a follow
up JIRA, but we should remember to do it.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533121#comment-16533121
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200208028
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -447,7 +447,7 @@
static {
Collections.sort(FUNCTION_KV_COLUMNS, KeyValue.COMPARATOR);
}
-
+
--- End diff --
Might be good to include a class level comment that explains the overall
approach at a high level.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533123#comment-16533123
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200208444
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -1893,26 +1981,45 @@ public static void upgradeTable(PhoenixConnection
conn, String srcTable) throws
conn.commit();
}
conn.getQueryServices().clearTableFromCache(
-conn.getTenantId() == null ?
ByteUtil.EMPTY_BYTE_ARRAY : conn.getTenantId().getBytes(),
+tenantIdBytes,
index.getSchemaName().getBytes(),
index.getTableName().getBytes(),
PhoenixRuntime.getCurrentScn(readOnlyProps));
}
updateIndexesSequenceIfPresent(conn, table);
conn.commit();
-
} else {
throw new RuntimeException("Error: problem occured during
upgrade. Table is not upgraded successfully");
}
if (table.getType() == PTableType.VIEW) {
logger.info(String.format("Updating link information for
view '%s' ..", table.getTableName()));
updateLink(conn, oldPhysicalName,
newPhysicalTablename,table.getSchemaName(),table.getTableName());
conn.commit();
-
+
+// if the view is a first level child, then we need to
create the PARENT_TABLE link
+// that was overwritten by the PHYSICAL_TABLE link
--- End diff --
Ah, good. So we'll be consistent with the parent link now, right?
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533116#comment-16533116
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200207100
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -2147,46 +2566,29 @@ private MetaDataMutationResult doDropTable(byte[]
key, byte[] tenantId, byte[] s
}
if (tableType == PTableType.TABLE || tableType ==
PTableType.SYSTEM) {
-// Handle any child views that exist
-TableViewFinder tableViewFinderResult =
findChildViews(region, tenantId, table, clientVersion, !isCascade);
-if (tableViewFinderResult.hasViews()) {
-if (isCascade) {
-if
(tableViewFinderResult.allViewsInMultipleRegions()) {
-// We don't yet support deleting a table with
views where SYSTEM.CATALOG has split and the
-// view metadata spans multiple regions
-return new
MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION,
-
EnvironmentEdgeManager.currentTimeMillis(), null);
-} else if
(tableViewFinderResult.allViewsInSingleRegion()) {
-// Recursively delete views - safe as all the
views as all in the same region
-for (ViewInfo viewInfo :
tableViewFinderResult.getViewInfoList()) {
-byte[] viewTenantId =
viewInfo.getTenantId();
-byte[] viewSchemaName =
viewInfo.getSchemaName();
-byte[] viewName = viewInfo.getViewName();
-byte[] viewKey =
SchemaUtil.getTableKey(viewTenantId, viewSchemaName, viewName);
-Delete delete = new Delete(viewKey,
clientTimeStamp);
-rowsToDelete.add(delete);
-acquireLock(region, viewKey, locks);
-MetaDataMutationResult result =
doDropTable(viewKey, viewTenantId, viewSchemaName,
-viewName, null, PTableType.VIEW,
rowsToDelete, invalidateList, locks,
-tableNamesToDelete,
sharedTablesToDelete, false, clientVersion);
-if (result.getMutationCode() !=
MutationCode.TABLE_ALREADY_EXISTS) { return result; }
-}
-}
-} else {
+// check to see if the table has any child views
+try (Table hTable =
+env.getTable(SchemaUtil.getPhysicalTableName(
+
PhoenixDatabaseMetaData.SYSTEM_CHILD_LINK_NAME_BYTES,
+env.getConfiguration( {
+boolean hasChildViews =
+ViewFinder.hasChildViews(hTable, tenantId,
schemaName, tableName,
+clientTimeStamp);
+if (hasChildViews && !isCascade) {
--- End diff --
Isn't there a race condition with this check?
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533114#comment-16533114
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200206862
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1809,180 +2256,97 @@ public void createTable(RpcController controller,
CreateTableRequest request,
} catch (Throwable t) {
logger.error("createTable failed", t);
ProtobufUtil.setControllerException(controller,
-
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName),
t));
+ServerUtil.createIOException(fullTableName, t));
}
}
+ private void dropChildMetadata(byte[] schemaName, byte[] tableName,
byte[] tenantIdBytes)
+ throws IOException, SQLException,
ClassNotFoundException {
+ TableViewFinderResult childViewsResult = new
TableViewFinderResult();
+ findAllChildViews(tenantIdBytes, schemaName, tableName,
childViewsResult);
+ if (childViewsResult.hasViews()) {
+ for (TableInfo viewInfo :
childViewsResult.getResults()) {
+ byte[] viewTenantId = viewInfo.getTenantId();
+ byte[] viewSchemaName =
viewInfo.getSchemaName();
+ byte[] viewName = viewInfo.getTableName();
+ Properties props = new Properties();
+ if (viewTenantId != null && viewTenantId.length
!= 0)
+
props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB,
Bytes.toString(viewTenantId));
+ try (PhoenixConnection connection =
QueryUtil.getConnectionOnServer(env.getConfiguration())
+
.unwrap(PhoenixConnection.class)) {
+ MetaDataClient client = new
MetaDataClient(connection);
+ org.apache.phoenix.parse.TableName
viewTableName = org.apache.phoenix.parse.TableName
+
.create(Bytes.toString(viewSchemaName), Bytes.toString(viewName));
+ client.dropTable(
+ new
DropTableStatement(viewTableName, PTableType.VIEW, false, true, true));
+ }
+ }
+ }
+ }
+
private boolean execeededIndexQuota(PTableType tableType, PTable
parentTable) {
return PTableType.INDEX == tableType &&
parentTable.getIndexes().size() >= maxIndexesPerTable;
}
-
-private void findAllChildViews(Region region, byte[] tenantId, PTable
table,
-TableViewFinder result, long clientTimeStamp, int
clientVersion) throws IOException, SQLException {
-TableViewFinder currResult = findChildViews(region, tenantId,
table, clientVersion, false);
-result.addResult(currResult);
-for (ViewInfo viewInfo : currResult.getViewInfoList()) {
-byte[] viewtenantId = viewInfo.getTenantId();
-byte[] viewSchema = viewInfo.getSchemaName();
-byte[] viewTable = viewInfo.getViewName();
-byte[] tableKey = SchemaUtil.getTableKey(viewtenantId,
viewSchema, viewTable);
-ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
-PTable view = loadTable(env, tableKey, cacheKey,
clientTimeStamp, clientTimeStamp, clientVersion);
-if (view == null) {
-logger.warn("Found orphan tenant view row in
SYSTEM.CATALOG with tenantId:"
-+ Bytes.toString(tenantId) + ", schema:"
-+ Bytes.toString(viewSchema) + ", table:"
-+ Bytes.toString(viewTable));
-continue;
-}
-findAllChildViews(region, viewtenantId, view, result,
clientTimeStamp, clientVersion);
-}
-}
-
-// TODO use child link instead once splittable system catalog
(PHOENIX-3534) is implemented
-// and we have a separate table for links.
-private TableViewFinder findChildViews_deprecated(Region region,
byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst)
throws IOException {
-byte[] schemaName = table.getSchemaName().getBytes();
-byte[] tableName = table.getTableName().getBytes();
-boolean isMultiTenant = table.isMultiTenant();
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533120#comment-16533120
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200208160
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/schema/TableProperty.java ---
@@ -231,36 +231,26 @@ public Object getPTableValue(PTable table) {
private final SQLExceptionCode mutatingImmutablePropException;
private final boolean isValidOnView;
private final boolean isMutableOnView;
-private final boolean propagateToViews;
private TableProperty(String propertyName, boolean isMutable, boolean
isValidOnView, boolean isMutableOnView) {
-this(propertyName, COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY,
isMutable, CANNOT_ALTER_PROPERTY, isValidOnView, isMutableOnView, true);
-}
-
-private TableProperty(String propertyName, boolean isMutable, boolean
isValidOnView, boolean isMutableOnView, boolean propagateToViews) {
-this(propertyName, COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY,
isMutable, CANNOT_ALTER_PROPERTY, isValidOnView, isMutableOnView,
propagateToViews);
+this(propertyName, COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY,
isMutable, CANNOT_ALTER_PROPERTY, isValidOnView, isMutableOnView);
}
private TableProperty(String propertyName, SQLExceptionCode
colFamilySpecifiedException, boolean isMutable, boolean isValidOnView, boolean
isMutableOnView) {
-this(propertyName, colFamilySpecifiedException, isMutable,
CANNOT_ALTER_PROPERTY, isValidOnView, isMutableOnView, true);
+this(propertyName, colFamilySpecifiedException, isMutable,
CANNOT_ALTER_PROPERTY, isValidOnView, isMutableOnView);
}
private TableProperty(String propertyName, boolean isMutable, boolean
isValidOnView, boolean isMutableOnView, SQLExceptionCode isMutatingException) {
-this(propertyName, COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY,
isMutable, isMutatingException, isValidOnView, isMutableOnView, true);
+this(propertyName, COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY,
isMutable, isMutatingException, isValidOnView, isMutableOnView);
}
private TableProperty(String propertyName, SQLExceptionCode
colFamSpecifiedException, boolean isMutable, SQLExceptionCode
mutatingException, boolean isValidOnView, boolean isMutableOnView) {
-this(propertyName, colFamSpecifiedException, isMutable,
mutatingException, isValidOnView, isMutableOnView, true);
-}
-
-private TableProperty(String propertyName, SQLExceptionCode
colFamSpecifiedException, boolean isMutable, SQLExceptionCode
mutatingException, boolean isValidOnView, boolean isMutableOnView, boolean
propagateToViews) {
--- End diff --
How did you end up dealing with table property conflicts between parent and
children? Is there follow up work required? Can we use the timestamp of the
Cell storing the property to differentiate similar to the logic for columns?
It's fine to do this work in a follow up JIRA.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533115#comment-16533115
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200206792
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1809,180 +2256,97 @@ public void createTable(RpcController controller,
CreateTableRequest request,
} catch (Throwable t) {
logger.error("createTable failed", t);
ProtobufUtil.setControllerException(controller,
-
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName),
t));
+ServerUtil.createIOException(fullTableName, t));
}
}
+ private void dropChildMetadata(byte[] schemaName, byte[] tableName,
byte[] tenantIdBytes)
+ throws IOException, SQLException,
ClassNotFoundException {
+ TableViewFinderResult childViewsResult = new
TableViewFinderResult();
+ findAllChildViews(tenantIdBytes, schemaName, tableName,
childViewsResult);
+ if (childViewsResult.hasViews()) {
+ for (TableInfo viewInfo :
childViewsResult.getResults()) {
+ byte[] viewTenantId = viewInfo.getTenantId();
+ byte[] viewSchemaName =
viewInfo.getSchemaName();
+ byte[] viewName = viewInfo.getTableName();
+ Properties props = new Properties();
+ if (viewTenantId != null && viewTenantId.length
!= 0)
+
props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB,
Bytes.toString(viewTenantId));
+ try (PhoenixConnection connection =
QueryUtil.getConnectionOnServer(env.getConfiguration())
+
.unwrap(PhoenixConnection.class)) {
+ MetaDataClient client = new
MetaDataClient(connection);
+ org.apache.phoenix.parse.TableName
viewTableName = org.apache.phoenix.parse.TableName
+
.create(Bytes.toString(viewSchemaName), Bytes.toString(viewName));
+ client.dropTable(
+ new
DropTableStatement(viewTableName, PTableType.VIEW, false, true, true));
+ }
+ }
+ }
+ }
+
private boolean execeededIndexQuota(PTableType tableType, PTable
parentTable) {
return PTableType.INDEX == tableType &&
parentTable.getIndexes().size() >= maxIndexesPerTable;
}
-
-private void findAllChildViews(Region region, byte[] tenantId, PTable
table,
-TableViewFinder result, long clientTimeStamp, int
clientVersion) throws IOException, SQLException {
-TableViewFinder currResult = findChildViews(region, tenantId,
table, clientVersion, false);
-result.addResult(currResult);
-for (ViewInfo viewInfo : currResult.getViewInfoList()) {
-byte[] viewtenantId = viewInfo.getTenantId();
-byte[] viewSchema = viewInfo.getSchemaName();
-byte[] viewTable = viewInfo.getViewName();
-byte[] tableKey = SchemaUtil.getTableKey(viewtenantId,
viewSchema, viewTable);
-ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
-PTable view = loadTable(env, tableKey, cacheKey,
clientTimeStamp, clientTimeStamp, clientVersion);
-if (view == null) {
-logger.warn("Found orphan tenant view row in
SYSTEM.CATALOG with tenantId:"
-+ Bytes.toString(tenantId) + ", schema:"
-+ Bytes.toString(viewSchema) + ", table:"
-+ Bytes.toString(viewTable));
-continue;
-}
-findAllChildViews(region, viewtenantId, view, result,
clientTimeStamp, clientVersion);
-}
-}
-
-// TODO use child link instead once splittable system catalog
(PHOENIX-3534) is implemented
-// and we have a separate table for links.
-private TableViewFinder findChildViews_deprecated(Region region,
byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst)
throws IOException {
-byte[] schemaName = table.getSchemaName().getBytes();
-byte[] tableName = table.getTableName().getBytes();
-boolean isMultiTenant = table.isMultiTenant();
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533109#comment-16533109
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200206293
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation
row, byte[][] schemaTableNames)
schemaTableNames[2] = tName;
}
}
-
+
@Override
public void createTable(RpcController controller, CreateTableRequest
request,
RpcCallback done) {
MetaDataResponse.Builder builder = MetaDataResponse.newBuilder();
byte[][] rowKeyMetaData = new byte[3][];
byte[] schemaName = null;
byte[] tableName = null;
+String fullTableName = null;
try {
int clientVersion = request.getClientVersion();
List tableMetadata =
ProtobufUtil.getMutations(request);
MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata,
rowKeyMetaData);
byte[] tenantIdBytes =
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
schemaName =
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
tableName =
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+fullTableName = SchemaUtil.getTableName(schemaName, tableName);
+// TODO before creating a table we need to see if the table
was previously created and then dropped
+// and clean up any parent->child links or child views
--- End diff --
Remove TODO as isn't this done now?
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533118#comment-16533118
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200207388
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -2573,307 +2897,139 @@ else if (pkCount <= COLUMN_NAME_INDEX
return new
MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION,
EnvironmentEdgeManager.currentTimeMillis(), basePhysicalTable);
}
-ColumnOrdinalPositionUpdateList ordinalPositionList = new
ColumnOrdinalPositionUpdateList();
+//add the new columns to the child view
List viewPkCols = new
ArrayList<>(view.getPKColumns());
boolean addingExistingPkCol = false;
-int numCols = view.getColumns().size();
-// add the new columns to the child view
-for (PutWithOrdinalPosition p : columnPutsForBaseTable) {
-Put baseTableColumnPut = p.put;
+for (Put columnToBeAdded : columnPutsForBaseTable) {
PColumn existingViewColumn = null;
byte[][] rkmd = new byte[5][];
-getVarChars(baseTableColumnPut.getRow(), rkmd);
+getVarChars(columnToBeAdded.getRow(), rkmd);
String columnName =
Bytes.toString(rkmd[COLUMN_NAME_INDEX]);
-String columnFamily = rkmd[FAMILY_NAME_INDEX] == null ?
null : Bytes.toString(rkmd[FAMILY_NAME_INDEX]);
+String columnFamily =
+rkmd[FAMILY_NAME_INDEX] == null ? null
+: Bytes.toString(rkmd[FAMILY_NAME_INDEX]);
try {
-existingViewColumn = columnFamily == null ?
view.getColumnForColumnName(columnName) : view.getColumnFamily(
-
columnFamily).getPColumnForColumnName(columnName);
+existingViewColumn =
+columnFamily == null ?
view.getColumnForColumnName(columnName)
+: view.getColumnFamily(columnFamily)
+
.getPColumnForColumnName(columnName);
} catch (ColumnFamilyNotFoundException e) {
-// ignore since it means that the column family is not
present for the column to be added.
+// ignore since it means that the column family is not
present for the column to
+// be added.
} catch (ColumnNotFoundException e) {
// ignore since it means the column is not present in
the view
}
-
-boolean isPkCol = columnFamily == null;
-byte[] columnKey = getColumnKey(viewKey, columnName,
columnFamily);
+
+boolean isColumnToBeAddPkCol = columnFamily == null;
if (existingViewColumn != null) {
-MetaDataMutationResult result =
validateColumnForAddToBaseTable(existingViewColumn, baseTableColumnPut,
basePhysicalTable, isPkCol, view);
-if (result != null) {
-return result;
+if
(EncodedColumnsUtil.usesEncodedColumnNames(basePhysicalTable)
+&& !SchemaUtil.isPKColumn(existingViewColumn))
{
--- End diff --
Is there a race condition with this check and would the be covered by one
of the future JIRAs you mentioned?
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533113#comment-16533113
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200206428
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1488,20 +1904,19 @@ public void createTable(RpcController controller,
CreateTableRequest request,
if (tableType == PTableType.VIEW) {
byte[][] parentSchemaTableNames = new byte[3][];
byte[][] parentPhysicalSchemaTableNames = new byte[3][];
-/*
- * For a view, we lock the base physical table row. For a
mapped view, there is
- * no link present to the physical table. So the
viewPhysicalTableRow is null
- * in that case.
- */
+ /*
+* For a mapped view, there is no link present
to the physical table. So the
+* viewPhysicalTableRow is null in that case.
+*/
--- End diff --
Fix indentation
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533124#comment-16533124
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200209126
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
---
@@ -372,6 +378,31 @@ public void testViewAndTableAndDrop() throws Exception
{
// drop table cascade should succeed
conn.createStatement().execute("DROP TABLE " + fullTableName + "
CASCADE");
+validateViewDoesNotExist(conn, fullViewName1);
+validateViewDoesNotExist(conn, fullViewName2);
+
+}
+
+@Test
+public void testRecreateDroppedTableWithChildViews() throws Exception {
--- End diff --
These new tests are good. These are testing that the left over metadata
doesn't impact the re-creation of a table since we don't make the RPC to delete
views when a base table is dropped, right? Do you think there'd be any issues
if part of the rows for a view were there (i.e. say that the create view
failed, but some of the rows were written)? Might be good to have a test like
this - you could set it up by using HBase APIs to manually delete some rows of
a view.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533117#comment-16533117
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200207594
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -3642,30 +3596,62 @@ private void dropIndexes(PTable table, Region
region, List in
boolean isCoveredColumn =
indexMaintainer.getCoveredColumns().contains(colDropRef);
// If index requires this column for its pk, then drop it
if (isColumnIndexed) {
-// Since we're dropping the index, lock it to ensure
-// that a change in index state doesn't
-// occur while we're dropping it.
-acquireLock(region, indexKey, locks);
// Drop the index table. The doDropTable will expand
// this to all of the table rows and invalidate the
// index table
-additionalTableMetaData.add(new Delete(indexKey,
clientTimeStamp));
+Delete delete = new Delete(indexKey, clientTimeStamp);
byte[] linkKey =
MetaDataUtil.getParentLinkKey(tenantId,
schemaName, tableName, index
.getTableName().getBytes());
-// Drop the link between the data table and the
+// Drop the link between the parent table and the
// index table
-additionalTableMetaData.add(new Delete(linkKey,
clientTimeStamp));
-doDropTable(indexKey, tenantId,
index.getSchemaName().getBytes(), index
-.getTableName().getBytes(), tableName,
index.getType(),
-additionalTableMetaData, invalidateList, locks,
tableNamesToDelete, sharedTablesToDelete, false, clientVersion);
-invalidateList.add(new ImmutableBytesPtr(indexKey));
+Delete linkDelete = new Delete(linkKey, clientTimeStamp);
+List tableMetaData =
Lists.newArrayListWithExpectedSize(2);
+Delete tableDelete = delete;
+tableMetaData.add(tableDelete);
+tableMetaData.add(linkDelete);
+// if the index is not present on the current region make
an rpc to drop it
--- End diff --
Is this ever the case since the index should be in the same schema as it's
table? Or is there a corner case with indexes on views?
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533122#comment-16533122 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user JamesRTaylor commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r200208319 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java --- @@ -1779,13 +2174,65 @@ public void createTable(RpcController controller, CreateTableRequest request, } } +// The mutations to create a table are written in the following order: +// 1. Write the child link as if the next two steps fail we +// ignore missing children while processing a parent +// 2. Update the encoded column qualifier for the parent table if its on a +// different region server (for tables that use column qualifier encoding) +// if the next step fails we end up wasting a few col qualifiers +// 3. Finally write the mutations to create the table + +// From 4.15 the parent->child links are stored in a separate table SYSTEM.CHILD_LINK +List childLinkMutations = MetaDataUtil.removeChildLinks(tableMetadata); --- End diff -- TODO to remove this code in 4.16. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533110#comment-16533110
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200206109
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -586,48 +590,359 @@ public void getTable(RpcController controller,
GetTableRequest request,
builder.setMutationTime(minNonZerodisableIndexTimestamp - 1);
}
}
-
-if (table.getTimeStamp() != tableTimeStamp) {
+// the PTable of views and indexes on views might get updated
because a column is added to one of
+// their parents (this won't change the timestamp)
+if (table.getType()!=PTableType.TABLE || table.getTimeStamp()
!= tableTimeStamp) {
builder.setTable(PTableImpl.toProto(table));
}
done.run(builder.build());
-return;
} catch (Throwable t) {
logger.error("getTable failed", t);
ProtobufUtil.setControllerException(controller,
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName),
t));
}
}
+/**
+ * Used to add the columns present the ancestor hierarchy to the
PTable of the given view or
+ * view index
+ * @param table PTable of the view or view index
+ * @param skipAddingIndexes if true the returned PTable won't include
indexes
+ * @param skipAddingParentColumns if true the returned PTable won't
include columns derived from
+ *ancestor tables
+ * @param lockedAncestorTable ancestor table table that is being
mutated (as we won't be able to
+ *resolve this table as its locked)
+ */
+private Pair
combineColumns(PTable table, long timestamp,
+int clientVersion, boolean skipAddingIndexes, boolean
skipAddingParentColumns,
+PTable lockedAncestorTable) throws SQLException, IOException {
+boolean hasIndexId = table.getViewIndexId() != null;
+if (table.getType() != PTableType.VIEW && !hasIndexId) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+}
+if (!skipAddingParentColumns) {
+table =
+addDerivedColumnsFromAncestors(table, timestamp,
clientVersion,
+lockedAncestorTable);
+if (table==null) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_NOT_FOUND);
+}
+// we need to resolve the indexes of views (to get ensure they
also have all the columns
+// derived from their ancestors)
+if (!skipAddingIndexes && !table.getIndexes().isEmpty()) {
+List indexes =
Lists.newArrayListWithExpectedSize(table.getIndexes().size());
+for (PTable index : table.getIndexes()) {
+byte[] tenantIdBytes =
+index.getTenantId() == null ?
ByteUtil.EMPTY_BYTE_ARRAY
+: index.getTenantId().getBytes();
+PTable latestIndex =
+doGetTable(tenantIdBytes,
index.getSchemaName().getBytes(),
+index.getTableName().getBytes(),
timestamp, null, clientVersion, true,
+false, lockedAncestorTable);
+if (latestIndex == null) {
+throw new TableNotFoundException(
+"Could not find index table while
combining columns "
++ index.getTableName().getString()
+ " with tenant id "
++ index.getTenantId());
+}
+indexes.add(latestIndex);
+}
+table = PTableImpl.makePTable(table, table.getTimeStamp(),
indexes);
+}
+}
+
+MetaDataProtos.MutationCode mutationCode =
+table != null ?
MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS
+: MetaDataProtos.MutationCode.TABLE_NOT_FOUND;
+return new Pair(table,
mutationCode);
+}
+
+
+private PTable addDerivedColumnsFromAncestors(PTable table, long
timestamp,
+int clientVersion, PTable
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533111#comment-16533111
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200205809
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -586,48 +590,359 @@ public void getTable(RpcController controller,
GetTableRequest request,
builder.setMutationTime(minNonZerodisableIndexTimestamp - 1);
}
}
-
-if (table.getTimeStamp() != tableTimeStamp) {
+// the PTable of views and indexes on views might get updated
because a column is added to one of
+// their parents (this won't change the timestamp)
+if (table.getType()!=PTableType.TABLE || table.getTimeStamp()
!= tableTimeStamp) {
builder.setTable(PTableImpl.toProto(table));
}
done.run(builder.build());
-return;
} catch (Throwable t) {
logger.error("getTable failed", t);
ProtobufUtil.setControllerException(controller,
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName),
t));
}
}
+/**
+ * Used to add the columns present the ancestor hierarchy to the
PTable of the given view or
+ * view index
+ * @param table PTable of the view or view index
+ * @param skipAddingIndexes if true the returned PTable won't include
indexes
+ * @param skipAddingParentColumns if true the returned PTable won't
include columns derived from
+ *ancestor tables
+ * @param lockedAncestorTable ancestor table table that is being
mutated (as we won't be able to
+ *resolve this table as its locked)
+ */
+private Pair
combineColumns(PTable table, long timestamp,
+int clientVersion, boolean skipAddingIndexes, boolean
skipAddingParentColumns,
+PTable lockedAncestorTable) throws SQLException, IOException {
+boolean hasIndexId = table.getViewIndexId() != null;
+if (table.getType() != PTableType.VIEW && !hasIndexId) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+}
+if (!skipAddingParentColumns) {
+table =
+addDerivedColumnsFromAncestors(table, timestamp,
clientVersion,
+lockedAncestorTable);
+if (table==null) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_NOT_FOUND);
+}
+// we need to resolve the indexes of views (to get ensure they
also have all the columns
+// derived from their ancestors)
+if (!skipAddingIndexes && !table.getIndexes().isEmpty()) {
+List indexes =
Lists.newArrayListWithExpectedSize(table.getIndexes().size());
+for (PTable index : table.getIndexes()) {
+byte[] tenantIdBytes =
+index.getTenantId() == null ?
ByteUtil.EMPTY_BYTE_ARRAY
+: index.getTenantId().getBytes();
+PTable latestIndex =
+doGetTable(tenantIdBytes,
index.getSchemaName().getBytes(),
+index.getTableName().getBytes(),
timestamp, null, clientVersion, true,
+false, lockedAncestorTable);
+if (latestIndex == null) {
+throw new TableNotFoundException(
+"Could not find index table while
combining columns "
++ index.getTableName().getString()
+ " with tenant id "
++ index.getTenantId());
+}
+indexes.add(latestIndex);
+}
+table = PTableImpl.makePTable(table, table.getTimeStamp(),
indexes);
+}
+}
+
+MetaDataProtos.MutationCode mutationCode =
+table != null ?
MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS
+: MetaDataProtos.MutationCode.TABLE_NOT_FOUND;
+return new Pair(table,
mutationCode);
+}
+
+
+private PTable addDerivedColumnsFromAncestors(PTable table, long
timestamp,
+int clientVersion, PTable
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533107#comment-16533107 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user JamesRTaylor commented on the issue: https://github.com/apache/phoenix/pull/303 +1 to the patch. Great work @twdsilva and @churrodog! I made some minor comments for some potential follow up work and had a few questions, but let's get this committed first. I'd recommend the following priority for the next JIRA as: 1. Move views to their own table 2. Get rid of client side code that is sending the base columns 3. Fix corner case/race condition issues 4. Add code that doesn't write orphaned metadata on major compaction > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533112#comment-16533112
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r200206389
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation
row, byte[][] schemaTableNames)
schemaTableNames[2] = tName;
}
}
-
+
@Override
public void createTable(RpcController controller, CreateTableRequest
request,
RpcCallback done) {
MetaDataResponse.Builder builder = MetaDataResponse.newBuilder();
byte[][] rowKeyMetaData = new byte[3][];
byte[] schemaName = null;
byte[] tableName = null;
+String fullTableName = null;
try {
int clientVersion = request.getClientVersion();
List tableMetadata =
ProtobufUtil.getMutations(request);
MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata,
rowKeyMetaData);
byte[] tenantIdBytes =
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
schemaName =
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
tableName =
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+fullTableName = SchemaUtil.getTableName(schemaName, tableName);
+// TODO before creating a table we need to see if the table
was previously created and then dropped
+// and clean up any parent->child links or child views
boolean isNamespaceMapped =
MetaDataUtil.isNameSpaceMapped(tableMetadata, GenericKeyValueBuilder.INSTANCE,
new ImmutableBytesWritable());
final IndexType indexType =
MetaDataUtil.getIndexType(tableMetadata, GenericKeyValueBuilder.INSTANCE,
new ImmutableBytesWritable());
+byte[] parentTenantId = null;
byte[] parentSchemaName = null;
byte[] parentTableName = null;
PTableType tableType =
MetaDataUtil.getTableType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new
ImmutableBytesWritable());
+ViewType viewType = MetaDataUtil.getViewType(tableMetadata,
GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable());
+
+// Load table to see if it already exists
+byte[] tableKey = SchemaUtil.getTableKey(tenantIdBytes,
schemaName, tableName);
+ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
+long clientTimeStamp =
MetaDataUtil.getClientTimeStamp(tableMetadata);
+PTable table = null;
+ try {
+ // Get as of latest timestamp so we can detect
if we have a newer table that already
+ // exists without making an additional query
+ table = loadTable(env, tableKey, cacheKey,
clientTimeStamp, HConstants.LATEST_TIMESTAMP,
+ clientVersion);
+ } catch (ParentTableNotFoundException e) {
+ dropChildMetadata(e.getParentSchemaName(),
e.getParentTableName(), e.getParentTenantId());
+ }
+if (table != null) {
+if (table.getTimeStamp() < clientTimeStamp) {
+// If the table is older than the client time stamp
and it's deleted,
+// continue
+if (!isTableDeleted(table)) {
+
builder.setReturnCode(MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+
builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis());
+builder.setTable(PTableImpl.toProto(table));
+done.run(builder.build());
+return;
+}
+} else {
+
builder.setReturnCode(MetaDataProtos.MutationCode.NEWER_TABLE_FOUND);
+
builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis());
+builder.setTable(PTableImpl.toProto(table));
+done.run(builder.build());
+return;
+}
+}
+
+ // check if the table was dropped, but had child views
that were have not yet
+ // been cleaned up by compaction
+ if
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528754#comment-16528754 ] Thomas D'Silva commented on PHOENIX-3534: - The tests are passing with this patch https://builds.apache.org/job/PreCommit-PHOENIX-Build/1900/consoleFull > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Fix For: 5.0.0, 4.15.0 > > Attachments: PHOENIX-3534.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527578#comment-16527578
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r199147322
--- Diff:
phoenix-core/src/test/java/org/apache/phoenix/coprocessor/MetaDataEndpointImplTest.java
---
@@ -0,0 +1,299 @@
+package org.apache.phoenix.coprocessor;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.HTable;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.junit.Test;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+public class MetaDataEndpointImplTest extends ParallelStatsDisabledIT {
--- End diff --
@JamesRTaylor Can you please review? I modified the PR to drop child
metadata when we try to reuse a table name of a table that was dropped. It also
handles reusing a view name / view index name.
I also added a config that determines whether or not SYSTEM.CATALOG can
split. I merged the latest changes from master as well.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522396#comment-16522396
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r197829637
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilter.java
---
@@ -35,20 +35,18 @@
* during cluster upgrades. However, tenant-owned data such as
tenant-owned views need to
* be copied. This WALEntryFilter will only allow tenant-owned rows in
SYSTEM.CATALOG to
* be replicated. Data from all other tables is automatically passed. It
will also copy
- * child links in SYSTEM.CATALOG that are globally-owned but point to
tenant-owned views.
+ * child links in SYSTEM.CHILD_LINK that are globally-owned but point to
tenant-owned views.
*
*/
public class SystemCatalogWALEntryFilter implements WALEntryFilter {
- private static byte[] CHILD_TABLE_BYTES =
- new byte[]{PTable.LinkType.CHILD_TABLE.getSerializedValue()};
-
@Override
public WAL.Entry filter(WAL.Entry entry) {
-//if the WAL.Entry's table isn't System.Catalog, it auto-passes this
filter
+//if the WAL.Entry's table isn't System.Catalog or System.Child_Link,
it auto-passes this filter
//TODO: when Phoenix drops support for pre-1.3 versions of HBase, redo
as a WALCellFilter
-if (!SchemaUtil.isMetaTable(entry.getKey().getTablename().getName())){
+byte[] tableName = entry.getKey().getTablename().getName();
+ if (!SchemaUtil.isMetaTable(tableName) &&
!SchemaUtil.isChildLinkTable(tableName)){
--- End diff --
SYSTEM.CHILD_LINK contains the parent->child linking rows and cells we use
to detect race conditions (eg a column of conflicting type being added at the
same time to a parent and child).
The latter cells are written with a short TTL.
I think we can use HBase replication for SYSTEM.CHILD_LINK. All the tenant
specific view metadata rows in SYSTEM.CATALOG start with tenant id.
I will modify this filter to how it was before PHOENIX-4229.
@gjacoby126 Thanks for the suggestion.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522356#comment-16522356
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user gjacoby126 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r197821955
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilter.java
---
@@ -35,20 +35,18 @@
* during cluster upgrades. However, tenant-owned data such as
tenant-owned views need to
* be copied. This WALEntryFilter will only allow tenant-owned rows in
SYSTEM.CATALOG to
* be replicated. Data from all other tables is automatically passed. It
will also copy
- * child links in SYSTEM.CATALOG that are globally-owned but point to
tenant-owned views.
+ * child links in SYSTEM.CHILD_LINK that are globally-owned but point to
tenant-owned views.
*
*/
public class SystemCatalogWALEntryFilter implements WALEntryFilter {
- private static byte[] CHILD_TABLE_BYTES =
- new byte[]{PTable.LinkType.CHILD_TABLE.getSerializedValue()};
-
@Override
public WAL.Entry filter(WAL.Entry entry) {
-//if the WAL.Entry's table isn't System.Catalog, it auto-passes this
filter
+//if the WAL.Entry's table isn't System.Catalog or System.Child_Link,
it auto-passes this filter
//TODO: when Phoenix drops support for pre-1.3 versions of HBase, redo
as a WALCellFilter
-if (!SchemaUtil.isMetaTable(entry.getKey().getTablename().getName())){
+byte[] tableName = entry.getKey().getTablename().getName();
+ if (!SchemaUtil.isMetaTable(tableName) &&
!SchemaUtil.isChildLinkTable(tableName)){
--- End diff --
Would it be safe to turn on normal HBase replication on the new
System.CHILD_LINK? (That is, is there any unwanted data in System.CHILD_LINK
that this WALFilter wouldn't copy that normal HBase replication would?)
If normal HBase replication works for System.CHILD_LINK, and all view data
left in System.Catalog starts with tenant_id, then the logic here can be
greatly simplified, similar to how it was before PHOENIX-4229
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499193#comment-16499193
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192570609
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
---
@@ -388,51 +435,65 @@ public void
testViewAndTableInDifferentSchemas(boolean isNamespaceMapped) throws
} catch (TableNotFoundException ignore) {
}
ddl = "DROP TABLE " + fullTableName1;
-validateCannotDropTableWithChildViewsWithoutCascade(conn,
fullTableName1);
ddl = "DROP VIEW " + fullViewName2;
conn.createStatement().execute(ddl);
ddl = "DROP TABLE " + fullTableName1;
conn.createStatement().execute(ddl);
}
-
+
@Test
-public void testDisallowDropOfColumnOnParentTable() throws Exception {
+public void testDropOfColumnOnParentTableInvalidatesView() throws
Exception {
Connection conn = DriverManager.getConnection(getUrl());
+String fullTableName = generateUniqueTableName();
+String viewName = generateUniqueViewName();
+splitSystemCatalog(Lists.newArrayList(fullTableName, viewName));
+
String ddl = "CREATE TABLE " + fullTableName + " (k1 INTEGER NOT
NULL, k2 INTEGER NOT NULL, v1 DECIMAL, CONSTRAINT pk PRIMARY KEY (k1, k2))" +
tableDDLOptions;
conn.createStatement().execute(ddl);
-String viewName = "V_" + generateUniqueName();
ddl = "CREATE VIEW " + viewName + "(v2 VARCHAR, v3 VARCHAR) AS
SELECT * FROM " + fullTableName + " WHERE v1 = 1.0";
conn.createStatement().execute(ddl);
-try {
-conn.createStatement().execute("ALTER TABLE " + fullTableName
+ " DROP COLUMN v1");
-fail();
-} catch (SQLException e) {
-
assertEquals(SQLExceptionCode.CANNOT_MUTATE_TABLE.getErrorCode(),
e.getErrorCode());
+conn.createStatement().execute("ALTER TABLE " + fullTableName + "
DROP COLUMN v1");
+// TODO see if its possibel to prevent the dropping of a column
thats required by a child view (for its view where clause)
+// the view should be invalid
--- End diff --
Ok, this sounds like it'll work fine.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499176#comment-16499176
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192569643
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
---
@@ -388,51 +435,65 @@ public void
testViewAndTableInDifferentSchemas(boolean isNamespaceMapped) throws
} catch (TableNotFoundException ignore) {
}
ddl = "DROP TABLE " + fullTableName1;
-validateCannotDropTableWithChildViewsWithoutCascade(conn,
fullTableName1);
ddl = "DROP VIEW " + fullViewName2;
conn.createStatement().execute(ddl);
ddl = "DROP TABLE " + fullTableName1;
conn.createStatement().execute(ddl);
}
-
+
@Test
-public void testDisallowDropOfColumnOnParentTable() throws Exception {
+public void testDropOfColumnOnParentTableInvalidatesView() throws
Exception {
Connection conn = DriverManager.getConnection(getUrl());
+String fullTableName = generateUniqueTableName();
+String viewName = generateUniqueViewName();
+splitSystemCatalog(Lists.newArrayList(fullTableName, viewName));
+
String ddl = "CREATE TABLE " + fullTableName + " (k1 INTEGER NOT
NULL, k2 INTEGER NOT NULL, v1 DECIMAL, CONSTRAINT pk PRIMARY KEY (k1, k2))" +
tableDDLOptions;
conn.createStatement().execute(ddl);
-String viewName = "V_" + generateUniqueName();
ddl = "CREATE VIEW " + viewName + "(v2 VARCHAR, v3 VARCHAR) AS
SELECT * FROM " + fullTableName + " WHERE v1 = 1.0";
conn.createStatement().execute(ddl);
-try {
-conn.createStatement().execute("ALTER TABLE " + fullTableName
+ " DROP COLUMN v1");
-fail();
-} catch (SQLException e) {
-
assertEquals(SQLExceptionCode.CANNOT_MUTATE_TABLE.getErrorCode(),
e.getErrorCode());
+conn.createStatement().execute("ALTER TABLE " + fullTableName + "
DROP COLUMN v1");
+// TODO see if its possibel to prevent the dropping of a column
thats required by a child view (for its view where clause)
+// the view should be invalid
--- End diff --
@JamesRTaylor
We don't need to do the checkAndPut on the header row of the base table as
this will block clients making changes to different columns. We only need to
ensure that if multiple clients are making changes to the same column only one
of them is allowed to make the change. So we can use the rowkey of the column
row. We could also do the checkAndPut on the new SYSTEM.CHILD_LINK table
instead of SYSTEM.CATALOG. This would be done from MetadataClient to ensure
that no other client can make a conflicting change before we call addColumn or
dropColumn.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498272#comment-16498272
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192460500
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
---
@@ -388,51 +435,65 @@ public void
testViewAndTableInDifferentSchemas(boolean isNamespaceMapped) throws
} catch (TableNotFoundException ignore) {
}
ddl = "DROP TABLE " + fullTableName1;
-validateCannotDropTableWithChildViewsWithoutCascade(conn,
fullTableName1);
ddl = "DROP VIEW " + fullViewName2;
conn.createStatement().execute(ddl);
ddl = "DROP TABLE " + fullTableName1;
conn.createStatement().execute(ddl);
}
-
+
@Test
-public void testDisallowDropOfColumnOnParentTable() throws Exception {
+public void testDropOfColumnOnParentTableInvalidatesView() throws
Exception {
Connection conn = DriverManager.getConnection(getUrl());
+String fullTableName = generateUniqueTableName();
+String viewName = generateUniqueViewName();
+splitSystemCatalog(Lists.newArrayList(fullTableName, viewName));
+
String ddl = "CREATE TABLE " + fullTableName + " (k1 INTEGER NOT
NULL, k2 INTEGER NOT NULL, v1 DECIMAL, CONSTRAINT pk PRIMARY KEY (k1, k2))" +
tableDDLOptions;
conn.createStatement().execute(ddl);
-String viewName = "V_" + generateUniqueName();
ddl = "CREATE VIEW " + viewName + "(v2 VARCHAR, v3 VARCHAR) AS
SELECT * FROM " + fullTableName + " WHERE v1 = 1.0";
conn.createStatement().execute(ddl);
-try {
-conn.createStatement().execute("ALTER TABLE " + fullTableName
+ " DROP COLUMN v1");
-fail();
-} catch (SQLException e) {
-
assertEquals(SQLExceptionCode.CANNOT_MUTATE_TABLE.getErrorCode(),
e.getErrorCode());
+conn.createStatement().execute("ALTER TABLE " + fullTableName + "
DROP COLUMN v1");
+// TODO see if its possibel to prevent the dropping of a column
thats required by a child view (for its view where clause)
+// the view should be invalid
--- End diff --
@JamesRTaylor
I think we can avoid this while dropping a column. However while adding a
column as the view and base table could be on different region servers we need
to detect and prevent a conflicting add column change happening at the same
time. I couldn't think of a better way to prevent this other than doing a
checkAndPut.
If we do the checkAndPut in MetadataClient to ensure that only one client
is able to add the same column at the same time and then make the rpc to the
server to add the column, do you think think that would scale?
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497689#comment-16497689
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192318659
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/schema/PColumnImpl.java ---
@@ -156,14 +209,14 @@ public String toString() {
return (familyName == null ? "" : familyName.toString() +
QueryConstants.NAME_SEPARATOR) + name.toString();
}
-@Override
-public int hashCode() {
-final int prime = 31;
-int result = 1;
-result = prime * result + ((familyName == null) ? 0 :
familyName.hashCode());
-result = prime * result + ((name == null) ? 0 : name.hashCode());
-return result;
-}
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result + ((familyName == null) ? 0 :
familyName.hashCode());
+ result = prime * result + ((name == null) ? 0 : name.hashCode());
+ return result;
+ }
--- End diff --
Fixed
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497681#comment-16497681
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192318056
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/compile/ColumnNameTrackingExpressionCompiler.java
---
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile;
+
+import java.sql.SQLException;
+import java.util.List;
+
+import org.apache.phoenix.parse.ColumnParseNode;
+import org.apache.phoenix.parse.StatelessTraverseAllParseNodeVisitor;
+
+import com.google.common.collect.Lists;
+
+public class ColumnNameTrackingExpressionCompiler extends
StatelessTraverseAllParseNodeVisitor {
+
+ private List dataColumnNames =
Lists.newArrayListWithExpectedSize(10);
+
+public void reset() {
+this.getDataColumnNames().clear();
+}
+
+ @Override
+public Void visit(ColumnParseNode node) throws SQLException {
+ getDataColumnNames().add(node.getName());
+return null;
+}
+
+ public List getDataColumnNames() {
+ return dataColumnNames;
+ }
+
+}
--- End diff --
Fixed.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497683#comment-16497683
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192318109
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/WhereConstantParser.java
---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import static org.apache.phoenix.util.PhoenixRuntime.CONNECTIONLESS;
+import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
+import static
org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
+
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.BitSet;
+import java.util.List;
+
+import org.apache.phoenix.compile.ColumnResolver;
+import org.apache.phoenix.compile.CreateTableCompiler;
+import org.apache.phoenix.compile.ExpressionCompiler;
+import org.apache.phoenix.compile.FromCompiler;
+import org.apache.phoenix.compile.StatementContext;
+import org.apache.phoenix.compile.WhereCompiler;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixStatement;
+import org.apache.phoenix.parse.ParseNode;
+import org.apache.phoenix.parse.SQLParser;
+import org.apache.phoenix.parse.SelectStatement;
+import org.apache.phoenix.schema.ColumnNotFoundException;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PColumnImpl;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableImpl;
+import org.apache.phoenix.schema.TableRef;
+import org.apache.phoenix.util.MetaDataUtil;
+
+import com.google.common.collect.Lists;
+
+
+public class WhereConstantParser {
+
+static PTable addViewInfoToPColumnsIfNeeded(PTable view) throws
SQLException {
+ boolean[] viewColumnConstantsMatched = new
boolean[view.getColumns().size()];
+byte[][] viewColumnConstantsToBe = new
byte[view.getColumns().size()][];
+if (view.getViewStatement() == null) {
+ return view;
+}
+SelectStatement select = new
SQLParser(view.getViewStatement()).parseQuery();
+ParseNode whereNode = select.getWhere();
+ColumnResolver resolver = FromCompiler.getResolver(new
TableRef(view));
+StatementContext context = new StatementContext(new
PhoenixStatement(getConnectionlessConnection()), resolver);
+Expression expression = null;
+try {
+ expression = WhereCompiler.compile(context, whereNode);
+}
+catch (ColumnNotFoundException e) {
+ // if we could not find a column used in the view statement
(which means its was dropped)
+ // this view is not valid any more
+ return null;
+}
+CreateTableCompiler.ViewWhereExpressionVisitor visitor =
+new CreateTableCompiler.ViewWhereExpressionVisitor(view,
viewColumnConstantsToBe);
+expression.accept(visitor);
+
+BitSet isViewColumnReferencedToBe = new
BitSet(view.getColumns().size());
+// Used to track column references in a view
+ExpressionCompiler expressionCompiler = new
CreateTableCompiler.ColumnTrackingExpressionCompiler(context,
isViewColumnReferencedToBe);
+whereNode.accept(expressionCompiler);
+
+List result = Lists.newArrayList();
+for (PColumn column : PTableImpl.getColumnsToClone(view)) {
+ boolean isViewReferenced =
isViewColumnReferencedToBe.get(column.getPosition());
+ if ( (visitor.isUpdatable() ||
view.getPKColumns().get(MetaDataUtil.getAutoPartitionColIndex(view)).equals(column))
+
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497678#comment-16497678 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r192317572 --- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/index/DropColumnIT.java --- @@ -522,7 +523,8 @@ public void helpTestDroppingIndexedColDropsViewIndex(boolean isMultiTenant) thro byte[] cq = column.getColumnQualifierBytes(); // there should be a single row belonging to VIEWINDEX2 assertNotNull(viewIndex2 + " row is missing", result.getValue(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, cq)); -assertNull(results.next()); +// TODO enable this after we drop view indexes than need a dropped column +//assertNull(results.next()); --- End diff -- This test now passes without commenting the assertNull. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497676#comment-16497676
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192317466
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1457,28 +1761,69 @@ private static void getSchemaTableNames(Mutation
row, byte[][] schemaTableNames)
schemaTableNames[2] = tName;
}
}
-
+
@Override
public void createTable(RpcController controller, CreateTableRequest
request,
RpcCallback done) {
MetaDataResponse.Builder builder = MetaDataResponse.newBuilder();
byte[][] rowKeyMetaData = new byte[3][];
byte[] schemaName = null;
byte[] tableName = null;
+String fullTableName = SchemaUtil.getTableName(schemaName,
tableName);
try {
int clientVersion = request.getClientVersion();
List tableMetadata =
ProtobufUtil.getMutations(request);
MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata,
rowKeyMetaData);
byte[] tenantIdBytes =
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
schemaName =
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
tableName =
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+// TODO before creating a table we need to see if the table
was previously created and then dropped
+// and clean up any parent->child links or child views
boolean isNamespaceMapped =
MetaDataUtil.isNameSpaceMapped(tableMetadata, GenericKeyValueBuilder.INSTANCE,
new ImmutableBytesWritable());
final IndexType indexType =
MetaDataUtil.getIndexType(tableMetadata, GenericKeyValueBuilder.INSTANCE,
new ImmutableBytesWritable());
+byte[] parentTenantId = null;
byte[] parentSchemaName = null;
byte[] parentTableName = null;
PTableType tableType =
MetaDataUtil.getTableType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new
ImmutableBytesWritable());
+ViewType viewType = MetaDataUtil.getViewType(tableMetadata,
GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable());
+
+// Here we are passed the parent's columns to add to a view,
PHOENIX-3534 allows for a splittable
+// System.Catalog thus we only store the columns that are new
to the view, not the parents columns,
+// thus here we remove everything that is ORDINAL.POSITION <=
baseColumnCount and update the
+// ORDINAL.POSITIONS to be shifted accordingly.
--- End diff --
I filed PHOENIX-4767 to remove the dedup code. We can stop sending the
parent column metadata in the same release.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497675#comment-16497675
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192317262
--- Diff:
phoenix-core/src/test/java/org/apache/phoenix/compile/QueryCompilerTest.java ---
@@ -1253,7 +1253,9 @@ public void testUnknownColumnInPKConstraint() throws
Exception {
}
}
-
+
+// see PHOENIX-3534, now tables can have duplicate columns and they
are removed implicitly
--- End diff --
This test passes, I have remove the ignore annotation.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497667#comment-16497667
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192316080
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDatabaseMetaData.java
---
@@ -479,179 +498,355 @@ private void addTenantIdFilter(StringBuilder buf,
String tenantIdPattern) {
private static void appendConjunction(StringBuilder buf) {
buf.append(buf.length() == 0 ? "" : " and ");
}
-
-@Override
+
+private static final PColumnImpl TENANT_ID_COLUMN = new
PColumnImpl(PNameFactory.newName(TENANT_ID),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl TABLE_SCHEM_COLUMN = new
PColumnImpl(PNameFactory.newName(TABLE_SCHEM),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl TABLE_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(TABLE_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl COLUMN_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(COLUMN_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl DATA_TYPE_COLUMN = new
PColumnImpl(PNameFactory.newName(DATA_TYPE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl TYPE_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(TYPE_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(TYPE_NAME), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl COLUMN_SIZE_COLUMN = new
PColumnImpl(PNameFactory.newName(COLUMN_SIZE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, COLUMN_SIZE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl BUFFER_LENGTH_COLUMN = new
PColumnImpl(PNameFactory.newName(BUFFER_LENGTH),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(BUFFER_LENGTH), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl DECIMAL_DIGITS_COLUMN = new
PColumnImpl(PNameFactory.newName(DECIMAL_DIGITS),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
DECIMAL_DIGITS_BYTES, HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl NUM_PREC_RADIX_COLUMN = new
PColumnImpl(PNameFactory.newName(NUM_PREC_RADIX),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(NUM_PREC_RADIX), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl NULLABLE_COLUMN = new
PColumnImpl(PNameFactory.newName(NULLABLE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, NULLABLE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl REMARKS_COLUMN = new
PColumnImpl(PNameFactory.newName(REMARKS),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497668#comment-16497668
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192316117
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -2967,6 +2982,11 @@ protected PhoenixConnection
upgradeSystemCatalogIfRequired(PhoenixConnection met
HTableDescriptor.SPLIT_POLICY + "='" +
SystemStatsSplitPolicy.class.getName() +"'"
);
}
+// TODO set the version for which the following upgrade code runs
correct
+if (currentServerSideTableTimeStamp <
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_14_0) {
--- End diff --
Done
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497666#comment-16497666
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192316026
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/exception/SQLExceptionCode.java
---
@@ -293,6 +293,7 @@ public SQLException newException(SQLExceptionInfo info)
{
SEQUENCE_NOT_CASTABLE_TO_AUTO_PARTITION_ID_COLUMN(1086, "44A17",
"Sequence Value not castable to auto-partition id column"),
CANNOT_COERCE_AUTO_PARTITION_ID(1087, "44A18", "Auto-partition id
cannot be coerced"),
+
--- End diff --
Done
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497658#comment-16497658
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192313898
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java ---
@@ -787,7 +787,7 @@ public MutationPlan compile(UpsertStatement upsert)
throws SQLException {
LinkedHashSet updateColumns =
Sets.newLinkedHashSetWithExpectedSize(nColumns + 1);
updateColumns.add(new PColumnImpl(
table.getPKColumns().get(position).getName(), //
Use first PK column name as we know it won't conflict with others
-null, PVarbinary.INSTANCE, null, null, false,
position, SortOrder.getDefault(), 0, null, false, null, false, false, null));
+null, PVarbinary.INSTANCE, null, null, false,
position, SortOrder.getDefault(), 0, null, false, null, false, false, null,
table.getPKColumns().get(0).getTimestamp()));
--- End diff --
We are using the first timestamp of the first pk column, so its guaranteed
to be non null.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497643#comment-16497643
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192309043
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
---
@@ -388,51 +435,65 @@ public void
testViewAndTableInDifferentSchemas(boolean isNamespaceMapped) throws
} catch (TableNotFoundException ignore) {
}
ddl = "DROP TABLE " + fullTableName1;
-validateCannotDropTableWithChildViewsWithoutCascade(conn,
fullTableName1);
ddl = "DROP VIEW " + fullViewName2;
conn.createStatement().execute(ddl);
ddl = "DROP TABLE " + fullTableName1;
conn.createStatement().execute(ddl);
}
-
+
@Test
-public void testDisallowDropOfColumnOnParentTable() throws Exception {
+public void testDropOfColumnOnParentTableInvalidatesView() throws
Exception {
Connection conn = DriverManager.getConnection(getUrl());
+String fullTableName = generateUniqueTableName();
+String viewName = generateUniqueViewName();
+splitSystemCatalog(Lists.newArrayList(fullTableName, viewName));
+
String ddl = "CREATE TABLE " + fullTableName + " (k1 INTEGER NOT
NULL, k2 INTEGER NOT NULL, v1 DECIMAL, CONSTRAINT pk PRIMARY KEY (k1, k2))" +
tableDDLOptions;
conn.createStatement().execute(ddl);
-String viewName = "V_" + generateUniqueName();
ddl = "CREATE VIEW " + viewName + "(v2 VARCHAR, v3 VARCHAR) AS
SELECT * FROM " + fullTableName + " WHERE v1 = 1.0";
conn.createStatement().execute(ddl);
-try {
-conn.createStatement().execute("ALTER TABLE " + fullTableName
+ " DROP COLUMN v1");
-fail();
-} catch (SQLException e) {
-
assertEquals(SQLExceptionCode.CANNOT_MUTATE_TABLE.getErrorCode(),
e.getErrorCode());
+conn.createStatement().execute("ALTER TABLE " + fullTableName + "
DROP COLUMN v1");
+// TODO see if its possibel to prevent the dropping of a column
thats required by a child view (for its view where clause)
+// the view should be invalid
--- End diff --
I think we should avoid needing any kind of locking (including a
checkAndPut). The scaling issues we ran into were caused by contention on a
lock for the parent. The same thing could happen again with a checkAndPut
(which is just locking the row during the check). I'd err on the side of
scalability and have a view be invalidated if its parent is deleted. I think
that's a more scalable solution.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497642#comment-16497642
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192308674
--- Diff:
phoenix-core/src/test/java/org/apache/phoenix/coprocessor/MetaDataEndpointImplTest.java
---
@@ -0,0 +1,299 @@
+package org.apache.phoenix.coprocessor;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.HTable;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.junit.Test;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+public class MetaDataEndpointImplTest extends ParallelStatsDisabledIT {
--- End diff --
I think we should keep the OrphanCleaner code. The idea is that failed
deletions of metadata are transparent. They shouldn't block creation of new
tables. We should also cleanup on compaction.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497640#comment-16497640
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192308276
--- Diff:
phoenix-core/src/test/java/org/apache/phoenix/coprocessor/MetaDataEndpointImplTest.java
---
@@ -0,0 +1,299 @@
+package org.apache.phoenix.coprocessor;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.HTable;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.junit.Test;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+public class MetaDataEndpointImplTest extends ParallelStatsDisabledIT {
--- End diff --
I removed the OrphanCleaner code that was there previously and I create a
PHOENIX-3534 to handle cleanup of child view metadata for a parent that was
dropped during compaction. Instead of having additional clean up code, I think
we should just detect that we are trying to re-create a table that was dropped
whose child view metadata wasn't cleaned up and then throw an exception.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497628#comment-16497628
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192305732
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDatabaseMetaData.java
---
@@ -479,179 +498,355 @@ private void addTenantIdFilter(StringBuilder buf,
String tenantIdPattern) {
private static void appendConjunction(StringBuilder buf) {
buf.append(buf.length() == 0 ? "" : " and ");
}
-
-@Override
+
+private static final PColumnImpl TENANT_ID_COLUMN = new
PColumnImpl(PNameFactory.newName(TENANT_ID),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl TABLE_SCHEM_COLUMN = new
PColumnImpl(PNameFactory.newName(TABLE_SCHEM),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl TABLE_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(TABLE_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl COLUMN_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(COLUMN_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl DATA_TYPE_COLUMN = new
PColumnImpl(PNameFactory.newName(DATA_TYPE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl TYPE_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(TYPE_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(TYPE_NAME), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl COLUMN_SIZE_COLUMN = new
PColumnImpl(PNameFactory.newName(COLUMN_SIZE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, COLUMN_SIZE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl BUFFER_LENGTH_COLUMN = new
PColumnImpl(PNameFactory.newName(BUFFER_LENGTH),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(BUFFER_LENGTH), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl DECIMAL_DIGITS_COLUMN = new
PColumnImpl(PNameFactory.newName(DECIMAL_DIGITS),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
DECIMAL_DIGITS_BYTES, HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl NUM_PREC_RADIX_COLUMN = new
PColumnImpl(PNameFactory.newName(NUM_PREC_RADIX),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(NUM_PREC_RADIX), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl NULLABLE_COLUMN = new
PColumnImpl(PNameFactory.newName(NULLABLE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, NULLABLE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl REMARKS_COLUMN = new
PColumnImpl(PNameFactory.newName(REMARKS),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497627#comment-16497627
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r192305114
--- Diff:
phoenix-core/src/it/java/org/apache/phoenix/end2end/ExplainPlanWithStatsEnabledIT.java
---
@@ -1202,33 +1202,44 @@ private void
testUseStatsForParallelizationOnSaltedTable(boolean useStatsFlag, b
assertEquals("B", rs.getString(1));
}
- @Test
- public void testUseStatsForParallelizationProperyOnViewIndex() throws
SQLException {
- String tableName = generateUniqueName();
- String viewName = generateUniqueName();
- String tenantViewName = generateUniqueName();
- String viewIndexName = generateUniqueName();
- boolean useStats = !DEFAULT_USE_STATS_FOR_PARALLELIZATION;
- try (Connection conn = DriverManager.getConnection(getUrl())) {
- conn.createStatement()
- .execute("create table " + tableName
- + "(tenantId CHAR(15)
NOT NULL, pk1 integer NOT NULL, v varchar CONSTRAINT PK PRIMARY KEY "
- + "(tenantId, pk1))
MULTI_TENANT=true");
- try (Connection tenantConn =
getTenantConnection("tenant1")) {
- conn.createStatement().execute("CREATE VIEW " +
viewName + " AS SELECT * FROM " + tableName);
- conn.createStatement().execute("CREATE INDEX "
+ viewIndexName + " on " + viewName + " (v) ");
- tenantConn.createStatement().execute("CREATE
VIEW " + tenantViewName + " AS SELECT * FROM " + viewName);
- conn.createStatement()
- .execute("ALTER TABLE " +
tableName + " set USE_STATS_FOR_PARALLELIZATION=" + useStats);
- // fetch the latest view ptable
- PhoenixRuntime.getTableNoCache(tenantConn,
viewName);
- PhoenixConnection phxConn =
conn.unwrap(PhoenixConnection.class);
- PTable viewIndex = phxConn.getTable(new
PTableKey(phxConn.getTenantId(), viewIndexName));
- assertEquals("USE_STATS_FOR_PARALLELIZATION
property set incorrectly", useStats,
- PhoenixConfigurationUtil
-
.getStatsForParallelizationProp(tenantConn.unwrap(PhoenixConnection.class),
viewIndex));
- }
- }
- }
+@Test
+public void testUseStatsForParallelizationProperyOnViewIndex() throws
SQLException {
+String tableName = generateUniqueName();
+String viewName = generateUniqueName();
+String tenantViewName = generateUniqueName();
+String viewIndexName = generateUniqueName();
+boolean useStats = !DEFAULT_USE_STATS_FOR_PARALLELIZATION;
+try (Connection conn = DriverManager.getConnection(getUrl())) {
+conn.createStatement()
+.execute("create table " + tableName
++ "(tenantId CHAR(15) NOT NULL, pk1 integer
NOT NULL, v varchar CONSTRAINT PK PRIMARY KEY "
++ "(tenantId, pk1)) MULTI_TENANT=true");
+try (Connection tenantConn = getTenantConnection("tenant1")) {
+conn.createStatement().execute("CREATE VIEW " + viewName +
" AS SELECT * FROM " + tableName);
+conn.createStatement().execute("CREATE INDEX " +
viewIndexName + " on " + viewName + " (v) ");
+tenantConn.createStatement().execute("CREATE VIEW " +
tenantViewName + " AS SELECT * FROM " + viewName);
+conn.createStatement()
+.execute("ALTER TABLE " + tableName + " set
USE_STATS_FOR_PARALLELIZATION=" + useStats);
+// changing a property on a base table does not change the
property on a view
--- End diff --
In PTable we don't have access to when a table property was updated. In
combineColumns when creating the PTable of the view if a table property is not
mutable on a view , the value is set to the same as the base physical table.
If the table property is mutable on a view we don't know if the value was
modified in the view or not so its we set the table property value to the one
in the view. This is different from current behavior where if we change a
table property on a base table and a
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495850#comment-16495850
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user churrodog commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191948951
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/WhereConstantParser.java
---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import static org.apache.phoenix.util.PhoenixRuntime.CONNECTIONLESS;
+import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
+import static
org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
+
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.BitSet;
+import java.util.List;
+
+import org.apache.phoenix.compile.ColumnResolver;
+import org.apache.phoenix.compile.CreateTableCompiler;
+import org.apache.phoenix.compile.ExpressionCompiler;
+import org.apache.phoenix.compile.FromCompiler;
+import org.apache.phoenix.compile.StatementContext;
+import org.apache.phoenix.compile.WhereCompiler;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixStatement;
+import org.apache.phoenix.parse.ParseNode;
+import org.apache.phoenix.parse.SQLParser;
+import org.apache.phoenix.parse.SelectStatement;
+import org.apache.phoenix.schema.ColumnNotFoundException;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PColumnImpl;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableImpl;
+import org.apache.phoenix.schema.TableRef;
+import org.apache.phoenix.util.MetaDataUtil;
+
+import com.google.common.collect.Lists;
+
+
+public class WhereConstantParser {
+
+static PTable addViewInfoToPColumnsIfNeeded(PTable view) throws
SQLException {
+ boolean[] viewColumnConstantsMatched = new
boolean[view.getColumns().size()];
+byte[][] viewColumnConstantsToBe = new
byte[view.getColumns().size()][];
+if (view.getViewStatement() == null) {
+ return view;
+}
+SelectStatement select = new
SQLParser(view.getViewStatement()).parseQuery();
+ParseNode whereNode = select.getWhere();
+ColumnResolver resolver = FromCompiler.getResolver(new
TableRef(view));
+StatementContext context = new StatementContext(new
PhoenixStatement(getConnectionlessConnection()), resolver);
+Expression expression = null;
+try {
+ expression = WhereCompiler.compile(context, whereNode);
+}
+catch (ColumnNotFoundException e) {
+ // if we could not find a column used in the view statement
(which means its was dropped)
+ // this view is not valid any more
+ return null;
+}
+CreateTableCompiler.ViewWhereExpressionVisitor visitor =
+new CreateTableCompiler.ViewWhereExpressionVisitor(view,
viewColumnConstantsToBe);
+expression.accept(visitor);
+
+BitSet isViewColumnReferencedToBe = new
BitSet(view.getColumns().size());
+// Used to track column references in a view
+ExpressionCompiler expressionCompiler = new
CreateTableCompiler.ColumnTrackingExpressionCompiler(context,
isViewColumnReferencedToBe);
+whereNode.accept(expressionCompiler);
+
+List result = Lists.newArrayList();
+for (PColumn column : PTableImpl.getColumnsToClone(view)) {
+ boolean isViewReferenced =
isViewColumnReferencedToBe.get(column.getPosition());
+ if ( (visitor.isUpdatable() ||
view.getPKColumns().get(MetaDataUtil.getAutoPartitionColIndex(view)).equals(column))
+
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495848#comment-16495848
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user churrodog commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191948812
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java ---
@@ -787,7 +787,7 @@ public MutationPlan compile(UpsertStatement upsert)
throws SQLException {
LinkedHashSet updateColumns =
Sets.newLinkedHashSetWithExpectedSize(nColumns + 1);
updateColumns.add(new PColumnImpl(
table.getPKColumns().get(position).getName(), //
Use first PK column name as we know it won't conflict with others
-null, PVarbinary.INSTANCE, null, null, false,
position, SortOrder.getDefault(), 0, null, false, null, false, false, null));
+null, PVarbinary.INSTANCE, null, null, false,
position, SortOrder.getDefault(), 0, null, false, null, false, false, null,
table.getPKColumns().get(0).getTimestamp()));
--- End diff --
are we guaranteed to get a non null column back?
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495849#comment-16495849
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191948821
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
---
@@ -388,51 +435,65 @@ public void
testViewAndTableInDifferentSchemas(boolean isNamespaceMapped) throws
} catch (TableNotFoundException ignore) {
}
ddl = "DROP TABLE " + fullTableName1;
-validateCannotDropTableWithChildViewsWithoutCascade(conn,
fullTableName1);
ddl = "DROP VIEW " + fullViewName2;
conn.createStatement().execute(ddl);
ddl = "DROP TABLE " + fullTableName1;
conn.createStatement().execute(ddl);
}
-
+
@Test
-public void testDisallowDropOfColumnOnParentTable() throws Exception {
+public void testDropOfColumnOnParentTableInvalidatesView() throws
Exception {
Connection conn = DriverManager.getConnection(getUrl());
+String fullTableName = generateUniqueTableName();
+String viewName = generateUniqueViewName();
+splitSystemCatalog(Lists.newArrayList(fullTableName, viewName));
+
String ddl = "CREATE TABLE " + fullTableName + " (k1 INTEGER NOT
NULL, k2 INTEGER NOT NULL, v1 DECIMAL, CONSTRAINT pk PRIMARY KEY (k1, k2))" +
tableDDLOptions;
conn.createStatement().execute(ddl);
-String viewName = "V_" + generateUniqueName();
ddl = "CREATE VIEW " + viewName + "(v2 VARCHAR, v3 VARCHAR) AS
SELECT * FROM " + fullTableName + " WHERE v1 = 1.0";
conn.createStatement().execute(ddl);
-try {
-conn.createStatement().execute("ALTER TABLE " + fullTableName
+ " DROP COLUMN v1");
-fail();
-} catch (SQLException e) {
-
assertEquals(SQLExceptionCode.CANNOT_MUTATE_TABLE.getErrorCode(),
e.getErrorCode());
+conn.createStatement().execute("ALTER TABLE " + fullTableName + "
DROP COLUMN v1");
+// TODO see if its possibel to prevent the dropping of a column
thats required by a child view (for its view where clause)
+// the view should be invalid
--- End diff --
I was thinking of handling the race condition by writing a cell using
checkAndPut with the column name being mutated (added or dropped) to the header
row of the base table. When a view is create the columns used in the view where
clause will also do a checkAndPut on this cell in the base table. If this is
happening concurrently only of the clients will succeed (either the drop
column, or the view creation).
If a DROP TABLE cascade is issued then the base table (with child views) is
dropped. When any child views are resolved and we see the base table does not
exists we throw a TableNotFoundException. If the DROP VIEW is issued without
CASCADE and there are child views the statement fails with a not allowed to
mutate exception. If a drop table without cascade and a create view happens
concurrently we could create the view even though the base table would have
been dropped, but the next time the view is resolved you would get a
TableNotFoundException.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495844#comment-16495844
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user churrodog commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191948510
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/compile/ColumnNameTrackingExpressionCompiler.java
---
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile;
+
+import java.sql.SQLException;
+import java.util.List;
+
+import org.apache.phoenix.parse.ColumnParseNode;
+import org.apache.phoenix.parse.StatelessTraverseAllParseNodeVisitor;
+
+import com.google.common.collect.Lists;
+
+public class ColumnNameTrackingExpressionCompiler extends
StatelessTraverseAllParseNodeVisitor {
+
+ private List dataColumnNames =
Lists.newArrayListWithExpectedSize(10);
+
+public void reset() {
+this.getDataColumnNames().clear();
+}
+
+ @Override
+public Void visit(ColumnParseNode node) throws SQLException {
+ getDataColumnNames().add(node.getName());
+return null;
+}
+
+ public List getDataColumnNames() {
+ return dataColumnNames;
+ }
+
+}
--- End diff --
Nit, but indentation seems off for this class.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495842#comment-16495842 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user churrodog commented on a diff in the pull request: https://github.com/apache/phoenix/pull/303#discussion_r191948329 --- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/index/DropColumnIT.java --- @@ -522,7 +523,8 @@ public void helpTestDroppingIndexedColDropsViewIndex(boolean isMultiTenant) thro byte[] cq = column.getColumnQualifierBytes(); // there should be a single row belonging to VIEWINDEX2 assertNotNull(viewIndex2 + " row is missing", result.getValue(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, cq)); -assertNull(results.next()); +// TODO enable this after we drop view indexes than need a dropped column +//assertNull(results.next()); --- End diff -- what about this TODO? > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495819#comment-16495819
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191944884
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
---
@@ -388,51 +435,65 @@ public void
testViewAndTableInDifferentSchemas(boolean isNamespaceMapped) throws
} catch (TableNotFoundException ignore) {
}
ddl = "DROP TABLE " + fullTableName1;
-validateCannotDropTableWithChildViewsWithoutCascade(conn,
fullTableName1);
ddl = "DROP VIEW " + fullViewName2;
conn.createStatement().execute(ddl);
ddl = "DROP TABLE " + fullTableName1;
conn.createStatement().execute(ddl);
}
-
+
@Test
-public void testDisallowDropOfColumnOnParentTable() throws Exception {
+public void testDropOfColumnOnParentTableInvalidatesView() throws
Exception {
Connection conn = DriverManager.getConnection(getUrl());
+String fullTableName = generateUniqueTableName();
+String viewName = generateUniqueViewName();
+splitSystemCatalog(Lists.newArrayList(fullTableName, viewName));
+
String ddl = "CREATE TABLE " + fullTableName + " (k1 INTEGER NOT
NULL, k2 INTEGER NOT NULL, v1 DECIMAL, CONSTRAINT pk PRIMARY KEY (k1, k2))" +
tableDDLOptions;
conn.createStatement().execute(ddl);
-String viewName = "V_" + generateUniqueName();
ddl = "CREATE VIEW " + viewName + "(v2 VARCHAR, v3 VARCHAR) AS
SELECT * FROM " + fullTableName + " WHERE v1 = 1.0";
conn.createStatement().execute(ddl);
-try {
-conn.createStatement().execute("ALTER TABLE " + fullTableName
+ " DROP COLUMN v1");
-fail();
-} catch (SQLException e) {
-
assertEquals(SQLExceptionCode.CANNOT_MUTATE_TABLE.getErrorCode(),
e.getErrorCode());
+conn.createStatement().execute("ALTER TABLE " + fullTableName + "
DROP COLUMN v1");
+// TODO see if its possibel to prevent the dropping of a column
thats required by a child view (for its view where clause)
+// the view should be invalid
--- End diff --
Seems like there'd be an inherent race condition, but if not, I suppose
keeping the same behavior is fine. What about preventing the drop of a table
with child views?
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495816#comment-16495816
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191944295
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -2227,35 +2551,36 @@ private MetaDataMutationResult doDropTable(byte[]
key, byte[] tenantId, byte[] s
// in 0.94.4, thus if we try to use it here we can no longer
use the 0.94.2 version
// of the client.
Delete delete = new Delete(indexKey, clientTimeStamp);
-rowsToDelete.add(delete);
-acquireLock(region, indexKey, locks);
+catalogMutations.add(delete);
MetaDataMutationResult result =
doDropTable(indexKey, tenantId, schemaName, indexName,
tableName, PTableType.INDEX,
-rowsToDelete, invalidateList, locks,
tableNamesToDelete, sharedTablesToDelete, false, clientVersion);
+catalogMutations, childLinkMutations,
invalidateList, tableNamesToDelete, sharedTablesToDelete, false, clientVersion);
if (result.getMutationCode() !=
MutationCode.TABLE_ALREADY_EXISTS) {
return result;
}
}
+// no need to pass sharedTablesToDelete back to the client as they
deletion of these tables
+// is already handled in MetadataClient.dropTable
--- End diff --
We still pass sharedTablesToDelete back to the client when calling
dropColumn, so that we drop the local or view index data I think. We don't need
to pass sharedTablesToDelete back to the client when dropping a table.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495797#comment-16495797
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191942250
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -2227,35 +2551,36 @@ private MetaDataMutationResult doDropTable(byte[]
key, byte[] tenantId, byte[] s
// in 0.94.4, thus if we try to use it here we can no longer
use the 0.94.2 version
// of the client.
Delete delete = new Delete(indexKey, clientTimeStamp);
-rowsToDelete.add(delete);
-acquireLock(region, indexKey, locks);
+catalogMutations.add(delete);
MetaDataMutationResult result =
doDropTable(indexKey, tenantId, schemaName, indexName,
tableName, PTableType.INDEX,
-rowsToDelete, invalidateList, locks,
tableNamesToDelete, sharedTablesToDelete, false, clientVersion);
+catalogMutations, childLinkMutations,
invalidateList, tableNamesToDelete, sharedTablesToDelete, false, clientVersion);
if (result.getMutationCode() !=
MutationCode.TABLE_ALREADY_EXISTS) {
return result;
}
}
+// no need to pass sharedTablesToDelete back to the client as they
deletion of these tables
+// is already handled in MetadataClient.dropTable
--- End diff --
Not sure if this is handled differently now, but we passed this back
because I believe we don't know on the client all of the physical index tables
to delete. I think we have a test for this.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495794#comment-16495794
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191941040
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1995,36 +2287,46 @@ public void dropTable(RpcController controller,
DropTableRequest request,
try {
List tableMetadata =
ProtobufUtil.getMutations(request);
+List childLinkMutations = Lists.newArrayList();
MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata,
rowKeyMetaData);
byte[] tenantIdBytes =
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
schemaName =
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
tableName =
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+PTableType
pTableType=PTableType.fromSerializedValue(tableType);
// Disallow deletion of a system table
-if (tableType.equals(PTableType.SYSTEM.getSerializedValue())) {
+if (pTableType == PTableType.SYSTEM) {
builder.setReturnCode(MetaDataProtos.MutationCode.UNALLOWED_TABLE_MUTATION);
builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis());
done.run(builder.build());
return;
}
+
List tableNamesToDelete = Lists.newArrayList();
List sharedTablesToDelete =
Lists.newArrayList();
-// No need to lock parent table for views
-byte[] parentTableName =
MetaDataUtil.getParentTableName(tableMetadata);
-byte[] lockTableName = parentTableName == null ||
tableType.equals(PTableType.VIEW.getSerializedValue()) ? tableName :
parentTableName;
-byte[] lockKey = SchemaUtil.getTableKey(tenantIdBytes,
schemaName, lockTableName);
-byte[] key =
-parentTableName == null ? lockKey :
SchemaUtil.getTableKey(tenantIdBytes,
-schemaName, tableName);
+
+byte[] lockKey = SchemaUtil.getTableKey(tenantIdBytes,
schemaName, tableName);
Region region = env.getRegion();
-MetaDataMutationResult result = checkTableKeyInRegion(key,
region);
+MetaDataMutationResult result = checkTableKeyInRegion(lockKey,
region);
if (result != null) {
done.run(MetaDataMutationResult.toProto(result));
return;
}
-PTableType
ptableType=PTableType.fromSerializedValue(tableType);
+
+byte[] parentTableName =
MetaDataUtil.getParentTableName(tableMetadata);
+byte[] parentLockKey = null;
+// No need to lock parent table for views
+if (parentTableName != null && pTableType != PTableType.VIEW) {
+parentLockKey = SchemaUtil.getTableKey(tenantIdBytes,
schemaName, parentTableName);
--- End diff --
We only lock the parent for indexes.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495792#comment-16495792
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191940463
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1995,36 +2287,46 @@ public void dropTable(RpcController controller,
DropTableRequest request,
try {
List tableMetadata =
ProtobufUtil.getMutations(request);
+List childLinkMutations = Lists.newArrayList();
MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata,
rowKeyMetaData);
byte[] tenantIdBytes =
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
schemaName =
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
tableName =
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+PTableType
pTableType=PTableType.fromSerializedValue(tableType);
// Disallow deletion of a system table
-if (tableType.equals(PTableType.SYSTEM.getSerializedValue())) {
+if (pTableType == PTableType.SYSTEM) {
builder.setReturnCode(MetaDataProtos.MutationCode.UNALLOWED_TABLE_MUTATION);
builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis());
done.run(builder.build());
return;
}
+
List tableNamesToDelete = Lists.newArrayList();
List sharedTablesToDelete =
Lists.newArrayList();
-// No need to lock parent table for views
-byte[] parentTableName =
MetaDataUtil.getParentTableName(tableMetadata);
-byte[] lockTableName = parentTableName == null ||
tableType.equals(PTableType.VIEW.getSerializedValue()) ? tableName :
parentTableName;
-byte[] lockKey = SchemaUtil.getTableKey(tenantIdBytes,
schemaName, lockTableName);
-byte[] key =
-parentTableName == null ? lockKey :
SchemaUtil.getTableKey(tenantIdBytes,
-schemaName, tableName);
+
+byte[] lockKey = SchemaUtil.getTableKey(tenantIdBytes,
schemaName, tableName);
Region region = env.getRegion();
-MetaDataMutationResult result = checkTableKeyInRegion(key,
region);
+MetaDataMutationResult result = checkTableKeyInRegion(lockKey,
region);
if (result != null) {
done.run(MetaDataMutationResult.toProto(result));
return;
}
-PTableType
ptableType=PTableType.fromSerializedValue(tableType);
+
+byte[] parentTableName =
MetaDataUtil.getParentTableName(tableMetadata);
+byte[] parentLockKey = null;
+// No need to lock parent table for views
+if (parentTableName != null && pTableType != PTableType.VIEW) {
+parentLockKey = SchemaUtil.getTableKey(tenantIdBytes,
schemaName, parentTableName);
--- End diff --
Shouldn't need this parentLockKey any longer, yes? Or is this only for
indexes?
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495787#comment-16495787
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191939788
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -1457,28 +1761,69 @@ private static void getSchemaTableNames(Mutation
row, byte[][] schemaTableNames)
schemaTableNames[2] = tName;
}
}
-
+
@Override
public void createTable(RpcController controller, CreateTableRequest
request,
RpcCallback done) {
MetaDataResponse.Builder builder = MetaDataResponse.newBuilder();
byte[][] rowKeyMetaData = new byte[3][];
byte[] schemaName = null;
byte[] tableName = null;
+String fullTableName = SchemaUtil.getTableName(schemaName,
tableName);
try {
int clientVersion = request.getClientVersion();
List tableMetadata =
ProtobufUtil.getMutations(request);
MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata,
rowKeyMetaData);
byte[] tenantIdBytes =
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
schemaName =
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
tableName =
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+// TODO before creating a table we need to see if the table
was previously created and then dropped
+// and clean up any parent->child links or child views
boolean isNamespaceMapped =
MetaDataUtil.isNameSpaceMapped(tableMetadata, GenericKeyValueBuilder.INSTANCE,
new ImmutableBytesWritable());
final IndexType indexType =
MetaDataUtil.getIndexType(tableMetadata, GenericKeyValueBuilder.INSTANCE,
new ImmutableBytesWritable());
+byte[] parentTenantId = null;
byte[] parentSchemaName = null;
byte[] parentTableName = null;
PTableType tableType =
MetaDataUtil.getTableType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new
ImmutableBytesWritable());
+ViewType viewType = MetaDataUtil.getViewType(tableMetadata,
GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable());
+
+// Here we are passed the parent's columns to add to a view,
PHOENIX-3534 allows for a splittable
+// System.Catalog thus we only store the columns that are new
to the view, not the parents columns,
+// thus here we remove everything that is ORDINAL.POSITION <=
baseColumnCount and update the
+// ORDINAL.POSITIONS to be shifted accordingly.
--- End diff --
Important to file and reference a JIRA here to remove the dedup code once
clients have been upgraded to the release in which we no longer send the
duplicate information. Can we stop sending the duplicate info in the same
release that SYSTEM.CATALOG becomes splittable? Seems like yes.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495778#comment-16495778
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191939222
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
---
@@ -388,51 +435,65 @@ public void
testViewAndTableInDifferentSchemas(boolean isNamespaceMapped) throws
} catch (TableNotFoundException ignore) {
}
ddl = "DROP TABLE " + fullTableName1;
-validateCannotDropTableWithChildViewsWithoutCascade(conn,
fullTableName1);
ddl = "DROP VIEW " + fullViewName2;
conn.createStatement().execute(ddl);
ddl = "DROP TABLE " + fullTableName1;
conn.createStatement().execute(ddl);
}
-
+
@Test
-public void testDisallowDropOfColumnOnParentTable() throws Exception {
+public void testDropOfColumnOnParentTableInvalidatesView() throws
Exception {
Connection conn = DriverManager.getConnection(getUrl());
+String fullTableName = generateUniqueTableName();
+String viewName = generateUniqueViewName();
+splitSystemCatalog(Lists.newArrayList(fullTableName, viewName));
+
String ddl = "CREATE TABLE " + fullTableName + " (k1 INTEGER NOT
NULL, k2 INTEGER NOT NULL, v1 DECIMAL, CONSTRAINT pk PRIMARY KEY (k1, k2))" +
tableDDLOptions;
conn.createStatement().execute(ddl);
-String viewName = "V_" + generateUniqueName();
ddl = "CREATE VIEW " + viewName + "(v2 VARCHAR, v3 VARCHAR) AS
SELECT * FROM " + fullTableName + " WHERE v1 = 1.0";
conn.createStatement().execute(ddl);
-try {
-conn.createStatement().execute("ALTER TABLE " + fullTableName
+ " DROP COLUMN v1");
-fail();
-} catch (SQLException e) {
-
assertEquals(SQLExceptionCode.CANNOT_MUTATE_TABLE.getErrorCode(),
e.getErrorCode());
+conn.createStatement().execute("ALTER TABLE " + fullTableName + "
DROP COLUMN v1");
+// TODO see if its possibel to prevent the dropping of a column
thats required by a child view (for its view where clause)
+// the view should be invalid
--- End diff --
Previously we used to prevent a base table column from being dropped if the
column was used in a child view index.
It's possible to detect this and prevent the column drop even with
splittable system.catalog if you think we should maintain the existing behavior.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495775#comment-16495775
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191938764
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -586,48 +573,336 @@ public void getTable(RpcController controller,
GetTableRequest request,
builder.setMutationTime(minNonZerodisableIndexTimestamp - 1);
}
}
-
-if (table.getTimeStamp() != tableTimeStamp) {
+// the PTable of views and indexes on views might get updated
because a column is added to one of
+// their parents (this won't change the timestamp)
+if (table.getType()!=PTableType.TABLE || table.getTimeStamp()
!= tableTimeStamp) {
builder.setTable(PTableImpl.toProto(table));
}
done.run(builder.build());
-return;
} catch (Throwable t) {
logger.error("getTable failed", t);
ProtobufUtil.setControllerException(controller,
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName),
t));
}
}
+/**
+ * Used to add the columns present the ancestor hierarchy to the
PTable of the given view or
+ * view index
+ * @param table PTable of the view or view index
+ * @param skipAddingIndexes if true the returned PTable won't include
indexes
+ * @param skipAddingParentColumns if true the returned PTable won't
include columns derived from ancestor tables
+ */
+private Pair
combineColumns(PTable table, long timestamp,
+int clientVersion, boolean skipAddingIndexes, boolean
skipAddingParentColumns) throws SQLException, IOException {
+boolean hasIndexId = table.getViewIndexId() != null;
+if (table.getType() != PTableType.VIEW && !hasIndexId) {
--- End diff --
We only need to combine columns for a VIEW or an index on a VIEW. I will
add a comment for this.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495764#comment-16495764
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191938147
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
@@ -586,48 +573,336 @@ public void getTable(RpcController controller,
GetTableRequest request,
builder.setMutationTime(minNonZerodisableIndexTimestamp - 1);
}
}
-
-if (table.getTimeStamp() != tableTimeStamp) {
+// the PTable of views and indexes on views might get updated
because a column is added to one of
+// their parents (this won't change the timestamp)
+if (table.getType()!=PTableType.TABLE || table.getTimeStamp()
!= tableTimeStamp) {
builder.setTable(PTableImpl.toProto(table));
}
done.run(builder.build());
-return;
} catch (Throwable t) {
logger.error("getTable failed", t);
ProtobufUtil.setControllerException(controller,
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName),
t));
}
}
+/**
+ * Used to add the columns present the ancestor hierarchy to the
PTable of the given view or
+ * view index
+ * @param table PTable of the view or view index
+ * @param skipAddingIndexes if true the returned PTable won't include
indexes
+ * @param skipAddingParentColumns if true the returned PTable won't
include columns derived from ancestor tables
+ */
+private Pair
combineColumns(PTable table, long timestamp,
+int clientVersion, boolean skipAddingIndexes, boolean
skipAddingParentColumns) throws SQLException, IOException {
+boolean hasIndexId = table.getViewIndexId() != null;
+if (table.getType() != PTableType.VIEW && !hasIndexId) {
--- End diff --
Just curious - why does the viewIndexId determine whether or not the table
already exists?
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495751#comment-16495751
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191936918
--- Diff:
phoenix-core/src/test/java/org/apache/phoenix/coprocessor/MetaDataEndpointImplTest.java
---
@@ -0,0 +1,299 @@
+package org.apache.phoenix.coprocessor;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.HTable;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.junit.Test;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+public class MetaDataEndpointImplTest extends ParallelStatsDisabledIT {
--- End diff --
One test that would be useful (and maybe you already have it?) is to create
a table, create a view (maybe on a different RS). Then drop the table and
recreate it and the view with the same name but different columns. Make sure
that the lazy cleanup code cleaned up the left over state correctly.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495744#comment-16495744
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191936296
--- Diff:
phoenix-core/src/test/java/org/apache/phoenix/compile/QueryCompilerTest.java ---
@@ -1253,7 +1253,9 @@ public void testUnknownColumnInPKConstraint() throws
Exception {
}
}
-
+
+// see PHOENIX-3534, now tables can have duplicate columns and they
are removed implicitly
--- End diff --
This should still be detected with an exception thrown.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495739#comment-16495739
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191935338
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/schema/PColumnImpl.java ---
@@ -156,14 +209,14 @@ public String toString() {
return (familyName == null ? "" : familyName.toString() +
QueryConstants.NAME_SEPARATOR) + name.toString();
}
-@Override
-public int hashCode() {
-final int prime = 31;
-int result = 1;
-result = prime * result + ((familyName == null) ? 0 :
familyName.hashCode());
-result = prime * result + ((name == null) ? 0 : name.hashCode());
-return result;
-}
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result + ((familyName == null) ? 0 :
familyName.hashCode());
+ result = prime * result + ((name == null) ? 0 : name.hashCode());
+ return result;
+ }
--- End diff --
Minor nit - various formatting issues. Not sure if there are tabs now or if
the indenting was wrong before.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495725#comment-16495725
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191933107
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
---
@@ -2967,6 +2982,11 @@ protected PhoenixConnection
upgradeSystemCatalogIfRequired(PhoenixConnection met
HTableDescriptor.SPLIT_POLICY + "='" +
SystemStatsSplitPolicy.class.getName() +"'"
);
}
+// TODO set the version for which the following upgrade code runs
correct
+if (currentServerSideTableTimeStamp <
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_14_0) {
--- End diff --
Yes - just add a MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_15_0
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495720#comment-16495720
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191932416
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDatabaseMetaData.java
---
@@ -479,179 +498,355 @@ private void addTenantIdFilter(StringBuilder buf,
String tenantIdPattern) {
private static void appendConjunction(StringBuilder buf) {
buf.append(buf.length() == 0 ? "" : " and ");
}
-
-@Override
+
+private static final PColumnImpl TENANT_ID_COLUMN = new
PColumnImpl(PNameFactory.newName(TENANT_ID),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl TABLE_SCHEM_COLUMN = new
PColumnImpl(PNameFactory.newName(TABLE_SCHEM),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl TABLE_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(TABLE_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl COLUMN_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(COLUMN_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl DATA_TYPE_COLUMN = new
PColumnImpl(PNameFactory.newName(DATA_TYPE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl TYPE_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(TYPE_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(TYPE_NAME), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl COLUMN_SIZE_COLUMN = new
PColumnImpl(PNameFactory.newName(COLUMN_SIZE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, COLUMN_SIZE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl BUFFER_LENGTH_COLUMN = new
PColumnImpl(PNameFactory.newName(BUFFER_LENGTH),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(BUFFER_LENGTH), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl DECIMAL_DIGITS_COLUMN = new
PColumnImpl(PNameFactory.newName(DECIMAL_DIGITS),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
DECIMAL_DIGITS_BYTES, HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl NUM_PREC_RADIX_COLUMN = new
PColumnImpl(PNameFactory.newName(NUM_PREC_RADIX),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(NUM_PREC_RADIX), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl NULLABLE_COLUMN = new
PColumnImpl(PNameFactory.newName(NULLABLE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, NULLABLE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl REMARKS_COLUMN = new
PColumnImpl(PNameFactory.newName(REMARKS),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495717#comment-16495717
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191931341
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDatabaseMetaData.java
---
@@ -479,179 +498,355 @@ private void addTenantIdFilter(StringBuilder buf,
String tenantIdPattern) {
private static void appendConjunction(StringBuilder buf) {
buf.append(buf.length() == 0 ? "" : " and ");
}
-
-@Override
+
+private static final PColumnImpl TENANT_ID_COLUMN = new
PColumnImpl(PNameFactory.newName(TENANT_ID),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl TABLE_SCHEM_COLUMN = new
PColumnImpl(PNameFactory.newName(TABLE_SCHEM),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl TABLE_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(TABLE_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+private static final PColumnImpl COLUMN_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(COLUMN_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl DATA_TYPE_COLUMN = new
PColumnImpl(PNameFactory.newName(DATA_TYPE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, DATA_TYPE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl TYPE_NAME_COLUMN = new
PColumnImpl(PNameFactory.newName(TYPE_NAME),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(TYPE_NAME), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl COLUMN_SIZE_COLUMN = new
PColumnImpl(PNameFactory.newName(COLUMN_SIZE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, COLUMN_SIZE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl BUFFER_LENGTH_COLUMN = new
PColumnImpl(PNameFactory.newName(BUFFER_LENGTH),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(BUFFER_LENGTH), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl DECIMAL_DIGITS_COLUMN = new
PColumnImpl(PNameFactory.newName(DECIMAL_DIGITS),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
DECIMAL_DIGITS_BYTES, HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl NUM_PREC_RADIX_COLUMN = new
PColumnImpl(PNameFactory.newName(NUM_PREC_RADIX),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
Bytes.toBytes(NUM_PREC_RADIX), HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl NULLABLE_COLUMN = new
PColumnImpl(PNameFactory.newName(NULLABLE),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PInteger.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false, NULLABLE_BYTES,
HConstants.LATEST_TIMESTAMP);
+ private static final PColumnImpl REMARKS_COLUMN = new
PColumnImpl(PNameFactory.newName(REMARKS),
+ PNameFactory.newName(TABLE_FAMILY_BYTES),
PVarchar.INSTANCE, null, null, false, 1, SortOrder.getDefault(),
+ 0, null, false, null, false, false,
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495702#comment-16495702
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191927956
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/exception/SQLExceptionCode.java
---
@@ -293,6 +293,7 @@ public SQLException newException(SQLExceptionInfo info)
{
SEQUENCE_NOT_CASTABLE_TO_AUTO_PARTITION_ID_COLUMN(1086, "44A17",
"Sequence Value not castable to auto-partition id column"),
CANNOT_COERCE_AUTO_PARTITION_ID(1087, "44A18", "Auto-partition id
cannot be coerced"),
+
--- End diff --
Revert please.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495696#comment-16495696
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191926624
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java
---
@@ -388,51 +435,65 @@ public void
testViewAndTableInDifferentSchemas(boolean isNamespaceMapped) throws
} catch (TableNotFoundException ignore) {
}
ddl = "DROP TABLE " + fullTableName1;
-validateCannotDropTableWithChildViewsWithoutCascade(conn,
fullTableName1);
ddl = "DROP VIEW " + fullViewName2;
conn.createStatement().execute(ddl);
ddl = "DROP TABLE " + fullTableName1;
conn.createStatement().execute(ddl);
}
-
+
@Test
-public void testDisallowDropOfColumnOnParentTable() throws Exception {
+public void testDropOfColumnOnParentTableInvalidatesView() throws
Exception {
Connection conn = DriverManager.getConnection(getUrl());
+String fullTableName = generateUniqueTableName();
+String viewName = generateUniqueViewName();
+splitSystemCatalog(Lists.newArrayList(fullTableName, viewName));
+
String ddl = "CREATE TABLE " + fullTableName + " (k1 INTEGER NOT
NULL, k2 INTEGER NOT NULL, v1 DECIMAL, CONSTRAINT pk PRIMARY KEY (k1, k2))" +
tableDDLOptions;
conn.createStatement().execute(ddl);
-String viewName = "V_" + generateUniqueName();
ddl = "CREATE VIEW " + viewName + "(v2 VARCHAR, v3 VARCHAR) AS
SELECT * FROM " + fullTableName + " WHERE v1 = 1.0";
conn.createStatement().execute(ddl);
-try {
-conn.createStatement().execute("ALTER TABLE " + fullTableName
+ " DROP COLUMN v1");
-fail();
-} catch (SQLException e) {
-
assertEquals(SQLExceptionCode.CANNOT_MUTATE_TABLE.getErrorCode(),
e.getErrorCode());
+conn.createStatement().execute("ALTER TABLE " + fullTableName + "
DROP COLUMN v1");
+// TODO see if its possibel to prevent the dropping of a column
thats required by a child view (for its view where clause)
+// the view should be invalid
--- End diff --
I think it's fine to consider the view invalid (i.e. fail any query that
attempts to use it) if all it's columns can no longer be found. This is pretty
typical in RDBMS.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495690#comment-16495690
]
ASF GitHub Bot commented on PHOENIX-3534:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/303#discussion_r191925360
--- Diff:
phoenix-core/src/it/java/org/apache/phoenix/end2end/ExplainPlanWithStatsEnabledIT.java
---
@@ -1202,33 +1202,44 @@ private void
testUseStatsForParallelizationOnSaltedTable(boolean useStatsFlag, b
assertEquals("B", rs.getString(1));
}
- @Test
- public void testUseStatsForParallelizationProperyOnViewIndex() throws
SQLException {
- String tableName = generateUniqueName();
- String viewName = generateUniqueName();
- String tenantViewName = generateUniqueName();
- String viewIndexName = generateUniqueName();
- boolean useStats = !DEFAULT_USE_STATS_FOR_PARALLELIZATION;
- try (Connection conn = DriverManager.getConnection(getUrl())) {
- conn.createStatement()
- .execute("create table " + tableName
- + "(tenantId CHAR(15)
NOT NULL, pk1 integer NOT NULL, v varchar CONSTRAINT PK PRIMARY KEY "
- + "(tenantId, pk1))
MULTI_TENANT=true");
- try (Connection tenantConn =
getTenantConnection("tenant1")) {
- conn.createStatement().execute("CREATE VIEW " +
viewName + " AS SELECT * FROM " + tableName);
- conn.createStatement().execute("CREATE INDEX "
+ viewIndexName + " on " + viewName + " (v) ");
- tenantConn.createStatement().execute("CREATE
VIEW " + tenantViewName + " AS SELECT * FROM " + viewName);
- conn.createStatement()
- .execute("ALTER TABLE " +
tableName + " set USE_STATS_FOR_PARALLELIZATION=" + useStats);
- // fetch the latest view ptable
- PhoenixRuntime.getTableNoCache(tenantConn,
viewName);
- PhoenixConnection phxConn =
conn.unwrap(PhoenixConnection.class);
- PTable viewIndex = phxConn.getTable(new
PTableKey(phxConn.getTenantId(), viewIndexName));
- assertEquals("USE_STATS_FOR_PARALLELIZATION
property set incorrectly", useStats,
- PhoenixConfigurationUtil
-
.getStatsForParallelizationProp(tenantConn.unwrap(PhoenixConnection.class),
viewIndex));
- }
- }
- }
+@Test
+public void testUseStatsForParallelizationProperyOnViewIndex() throws
SQLException {
+String tableName = generateUniqueName();
+String viewName = generateUniqueName();
+String tenantViewName = generateUniqueName();
+String viewIndexName = generateUniqueName();
+boolean useStats = !DEFAULT_USE_STATS_FOR_PARALLELIZATION;
+try (Connection conn = DriverManager.getConnection(getUrl())) {
+conn.createStatement()
+.execute("create table " + tableName
++ "(tenantId CHAR(15) NOT NULL, pk1 integer
NOT NULL, v varchar CONSTRAINT PK PRIMARY KEY "
++ "(tenantId, pk1)) MULTI_TENANT=true");
+try (Connection tenantConn = getTenantConnection("tenant1")) {
+conn.createStatement().execute("CREATE VIEW " + viewName +
" AS SELECT * FROM " + tableName);
+conn.createStatement().execute("CREATE INDEX " +
viewIndexName + " on " + viewName + " (v) ");
+tenantConn.createStatement().execute("CREATE VIEW " +
tenantViewName + " AS SELECT * FROM " + viewName);
+conn.createStatement()
+.execute("ALTER TABLE " + tableName + " set
USE_STATS_FOR_PARALLELIZATION=" + useStats);
+// changing a property on a base table does not change the
property on a view
--- End diff --
Shouldn't setting a property on the base table impact the view as well? In
this case, USE_STATS_FOR_PARALLELIZATION only makes sense to set on a physical
table. I think we only look it up from a physical table as well, so this is
somewhat moot, but in general, I'd think that setting a property on a base
table should be seen by it's views if the property has not been set there.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key:
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493974#comment-16493974 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user twdsilva commented on the issue: https://github.com/apache/phoenix/pull/303 @vincentpoon @gjacoby126 This is the PR to support splittable system.catalog. Please review if you have some time. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493874#comment-16493874 ] ASF GitHub Bot commented on PHOENIX-3534: - GitHub user twdsilva opened a pull request: https://github.com/apache/phoenix/pull/303 PHOENIX-3534 Support multi region SYSTEM.CATALOG table This patch adds two new LinkTypes EXCLUDED_COLUMNS (used to represent a column that has been dropped) and VIEW_INDEX_PARENT_TABLE (used to link an index on a view to its parent). Views and view indexes no longer store columns derived from their ancestors in their metadata. When they are resolved the ancestors are looked up and added to the PTable that is returned to the client (see combineColumns in MetadataEndpointImpl). The PTable in the server side metadata cache only stores the columns created in the view/view index and not derived columns. We do not propagate metadata changes made to a parent to all its children. While adding columns to a base table, we no longer lock all the children in the view hierarchy, we only validate that the columns being added does not conflict with an existing base table column. We also don't lock children while dropping a parent table column. When dropping a parent column we also drop any view indexes that need the column. This patch does not handle the case when there are concurrent changes (eg. adding a conflicting column or creating a new view index that requires a parent column that is being dropped). That will be handled in a follow-up patch. While dropping a parent table, we don't drop all the child views metadata. This metadata needs to be cleaned-up (maybe at compaction time?) which will be handled in a follow-up patch. There are a few test failures I am working through, which I will fix soon and update the PR. @JamesRTaylor can you please review? FYI @karanmehta93 @ChinmaySKulkarni You can merge this pull request into a Git repository by running: $ git pull https://github.com/twdsilva/phoenix PHOENIX-3534 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/phoenix/pull/303.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #303 commit a7712e3977830ffe715a2caaa577eb5cdb071e90 Author: rgidwani Date: 2017-03-17T19:21:56Z Starting work on splittable System.Catalog commit 220849fff34211a4b6356200365afe81d639cfc1 Author: rgidwani Date: 2017-03-20T20:44:54Z Removed all references to multi-region System.Catalog for TableViewFinderResult commit 356cd43e20be0d01c7ef5f97a9b9d26e213a140d Author: rgidwani Date: 2017-03-31T23:11:59Z Create table work, still trying to get rid of all columns commit e01adb5f1c45d43a684371dcbf39eadbc381f9d2 Author: rgidwani Date: 2017-04-04T20:33:07Z Create table and read views work now commit 19c7ce54dcd1616ca8b8a7078fa8fc738e70f4f4 Author: rgidwani Date: 2017-04-04T20:35:30Z Fixed the test - moving on to add drop columns commit ab20f8da00145cc527c6f78dc7d493df7ac2f1b0 Author: rgidwani Date: 2017-04-04T22:59:10Z getting tests and add column to work commit ec3574453e8fae662a271f4340f82eaf90f52ce2 Author: rgidwani Date: 2017-04-05T23:02:29Z Figuring out the delete logic and refactoring the old tests commit 7d4133034b7167a7919eb9dd4ab19533ae9300ea Author: rgidwani Date: 2017-04-11T22:25:32Z Added proto timestamp and exluded values to pcolumn also took care of additive case where we take lower timestamp commit adfc5ce7f9e7f55f801b5b8af5f414fd7ff96c23 Author: rgidwani Date: 2017-04-28T23:02:27Z Drop Column Work in Progress, need to figure out how to resolve the combine logic for excluded columns commit 24414bd942055769b68943d93e19013777ff7299 Author: rgidwani Date: 2017-05-01T22:32:35Z Alter view drop column works! commit 13b6e520e3f7a81bb257c7fe68bae81ededa6c99 Author: rgidwani Date: 2017-05-12T22:55:49Z Drop Cascade and create check completed, need to test commit 0313ceee0b0a7515323dc0fc402a6eac76786155 Author: rgidwani Date: 2017-05-15T20:40:14Z Drop cascade seems to work commit 590689f3243a388ae528850c3240427ff49a640c Author: rgidwani Date: 2017-05-15T21:13:30Z Phoenix 3534" to "PHOENIX-3534 Support multi region SYSTEM.CATALOG table commit 96c0570bfcd767ad2342fb2fa22142eceed2b4eb Author: rgidwani Date: 2017-05-24T19:32:05Z Fixing up a few things, resolving read columns for child views doesn't always seem to work commit ca64a0b36f5bf33c8ed3169096fce08ac768d695 Author: rgidwani Date: 2017-05-31T23:00:31Z Adding in the child view constants commit ac59c72784aff0478d9a481db5729827e0ba3cce Author: rgidwani Date: 2017-05-31T23:00:31Z Adding in the child view constants commit
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470759#comment-16470759 ] James Taylor commented on PHOENIX-3534: --- For mutable properties, can we use the timestamp to differentiate? So the parent value would override the child value if it's timestamp is later (and visa versa). > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462940#comment-16462940 ] Thomas D'Silva commented on PHOENIX-3534: - [~jamestaylor] Currently if you change a table property on a base table that is valid on a view we propagate it to all child views. If the table property is not mutable on the child view we set it to the parent view value. If it is mutable and the view didn't change the property we set it to the parent view value. After splittable system catalog since we resolve the parent hierarchy of a view while combining columns we can also inherit the table properties from the base table. If a table property is valid on a view but not mutable on a view we can just use the base table value. If it is mutable on a view, we don't know if the property was changed on the view on not, so we have to use the value that is set on the view. This is different from the existing behavior, where if you change a table property that is valid on a view and mutable on view and if it wasn't changed on the view, it would get propagated to all the child views. Is this ok? > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459452#comment-16459452 ] Thomas D'Silva commented on PHOENIX-3534: - I have created a doc with the major work that is remaining and would appreciate any feedback. https://docs.google.com/document/d/1g39s-9JfTZUtW5TpUjuh9HJhPrIoING1GFHKg2HYcYM/edit?usp=sharing > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451478#comment-16451478 ] Thomas D'Silva commented on PHOENIX-3534: - I think that approach works best. We would not need to support the old behavior and we would not lose any new views that were created if for some reason we needed to rollback the release. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450431#comment-16450431 ] James Taylor commented on PHOENIX-3534: --- IMHO, I don't think the propagation of properties from base table to child views will scale. We'd have to update ~2 million rows, trying to get the locks for all of them. Instead, I think we should disallow changes to a base table that need to be propagated to child views until after the splittable system catalog has been rolled out to both server and client (when we'd basically be saying that we won't be rolling back any longer). > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450284#comment-16450284 ] Andrew Purtell commented on PHOENIX-3534: - [~lhofhansl] WDYT ^^^ > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450263#comment-16450263 ] Thomas D'Silva commented on PHOENIX-3534: - As long as we are fine with losing views that were created between the upgrade and if/when we decided to rollback. This would be a lot simpler since we wouldn't have to support the old behavior. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450233#comment-16450233 ] Andrew Purtell commented on PHOENIX-3534: - bq. Do we want to be able to rollback a release that contains this feature? If so we will still have to propagate metadata changes from the base table to all the child views and also ensure SYSTEM.CATALOG does not split. Or is it possible to make an HBase level snapshot of the whole SYSTEM schema, and then if there must be a rollback, we drop all of the upgraded SYSTEM tables and restore from the snapshot? Of course views created inbetween snapshot and rollback will be lost, but I think that is an acceptable tradeoff. Does this then give you the freedom you need to make all of the necessary changes? > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450225#comment-16450225 ] Thomas D'Silva commented on PHOENIX-3534: - [~jamestaylor] , [~lhofhansl] Do we want to be able to rollback a release that contains this feature? If so we will still have to propagate metadata changes from the base table to all the child views and also ensure SYSTEM.CATALOG does not split. Once we stop propagating metadata changes to child views we won't be able to rollback, since the metadata of the view will only contain columns that were created in the view (and will not contain any parent table columns). So should we continue propagating metadata changes to child views for one more release? > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334603#comment-16334603 ] James Taylor commented on PHOENIX-3534: --- FYI, the patch is now on the system-catalog feature branch. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva >Priority: Major > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206681#comment-16206681 ] James Taylor commented on PHOENIX-3534: --- I think the child links are going to be a scaling problem. We have a split policy to keep all meta data in the same schema together, but imagine if there are 10M or even 100M child linking rows from a base table or global view to all of the children views. Let's look at how the child links are used and either: - put them in a separate table. - only store the child count with the parent table (so we know whether it has children). > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva > Attachments: PHOENIX-3534-wip.patch > > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052510#comment-16052510 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user JamesRTaylor commented on the issue: https://github.com/apache/phoenix/pull/248 Looks like there are still conflicts, @churrodog. Would you mind amending your commit message to "PHOENIX-3534 Support multi region SYSTEM.CATALOG table" so that the JIRA is linked to the PR? > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: churro morales > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011349#comment-16011349 ] ASF GitHub Bot commented on PHOENIX-3534: - Github user JamesRTaylor commented on the issue: https://github.com/apache/phoenix/pull/248 I'll check it out, but in the meantime would you mind amending your commit message from "Phoenix 3534" to "PHOENIX-3534 Support multi region SYSTEM.CATALOG table" - that way comments on the pull will become JIRA comments. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: churro morales > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983232#comment-15983232 ] churro morales commented on PHOENIX-3534: - that is fine, we resolve columns on read so they can be excluded quite easily. I believe we walk from child to parent always as well so we can just plug that logic into our getTable code. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: churro morales > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983192#comment-15983192
]
James Taylor commented on PHOENIX-3534:
---
One more corner case: once a view has deleted columns from its parent table(s),
it's considered as "diverged". I think we can detect this if the view has any
excluded columns. In this case, we should no longer add columns from the parent
tables created after this. I think we can base this on the earliest timestamp
of the excluded column key value. This is conceptually equivalent to creating
the view without doing a SELECT *, but instead only selecting a subset of
columns, like this:
{code}
CREATE VIEW v AS SELECT a,c FROM t;
{code}
In this case, columns added to {{t}} should not impact the view {{v}} at all.
> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: churro morales
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963590#comment-15963590 ] churro morales commented on PHOENIX-3534: - For the above approach, it seems like we will need to add (timestamp) to PColumn to get this information, otherwise we will end up doing lots of extra point gets and it looks like it wont be cached either. [~jamestaylor] what do you think? > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: churro morales > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
[ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963585#comment-15963585 ] churro morales commented on PHOENIX-3534: - Discussed with [~jamestaylor] today. Right now we can only update per (table/view) so we know nothing about relations for add / drop columns or create view. So the logic can be the following: For all reads we do this: As we walk up the tree, we create a list of PColumns. If the column is contained in the parent as well, we see which one is older and keep that one (we take the min timestamp). For dropped columns as described above, we decide whether to drop the column based on the timestamp again but this time the decision is made based on the max timestamp. If the dropped column timestamp is newer than the columns timestamp, we exclude it, if it is older than the column timestamp we include it in the result. > Support multi region SYSTEM.CATALOG table > - > > Key: PHOENIX-3534 > URL: https://issues.apache.org/jira/browse/PHOENIX-3534 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: churro morales > > Currently Phoenix requires that the SYSTEM.CATALOG table is single region > based on the server-side row locks being held for operations that impact a > table and all of it's views. For example, adding/removing a column from a > base table pushes this change to all views. > As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), > when a new table is created we can do a lazy cleanup of any rows that may be > left over from a failed DDL call (kudos to [~lhofhansl] for coming up with > this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 > so that we can efficiently find derived views. > The implementation would rely on an optimistic concurrency model based on > checking our sequence numbers for each table/view before/after updating. Each > table/view row would be individually locked for their change (metadata for a > view or table cannot span regions due to our split policy), with the sequence > number being incremented under lock and then returned to the client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
