[jira] [Commented] (PHOENIX-6373) Schema changes that require table re-writes can be supported (Online data format changes)
[ https://issues.apache.org/jira/browse/PHOENIX-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463519#comment-17463519 ] ASF GitHub Bot commented on PHOENIX-6373: - gokceni commented on pull request #1368: URL: https://github.com/apache/phoenix/pull/1368#issuecomment-999170830 @gjacoby126 @virajjasani -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Schema changes that require table re-writes can be supported (Online data > format changes) > - > > Key: PHOENIX-6373 > URL: https://issues.apache.org/jira/browse/PHOENIX-6373 > Project: Phoenix > Issue Type: Improvement >Reporter: Gokcen Iskender >Assignee: Gokcen Iskender >Priority: Major > > Today, using ALTER TABLE or ALTER INDEX commands, the user can make certain > changes to the schema. For example, changing certain table properties like > TTL and immutability, adding nullable columns/dropping non-pk columns are > allowed as well as certain index state changes. All of the allowed changes > don’t require the table to be re-written. As soon as the ALTER command > returns, most of the changes became available immediately (eg. index disable, > TTL) but some of them might take some time and the syntax lets you to specify > async (eg. Rebuild index) and depending on the client cache settings, some > changes never make it to the client (eg. select * that run from a client > never seeing the new column since its schema cache is not updated). > If the user wants to change the schema properties that require table > re-writes, it is blocked and ALTER fails. Phoenix lacks of the ability to > change some of the table schemas and attributes, such changing the row key > (primary keys), the type of a column, the table storage format, the column > encoding, etc. There is no way to make this changes with no or very minimal > service interruption. > Design doc link: > [https://docs.google.com/document/d/1D24zRETMEetXvc3MSZj9WKYeUnooQX5BLv6gSDEmwfk/edit?usp=sharing] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] gokceni commented on pull request #1368: PHOENIX-6373 IndexRegionObserver changes for creating mutations for transforming t…
gokceni commented on pull request #1368: URL: https://github.com/apache/phoenix/pull/1368#issuecomment-999170830 @gjacoby126 @virajjasani -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6373) Schema changes that require table re-writes can be supported (Online data format changes)
[ https://issues.apache.org/jira/browse/PHOENIX-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463518#comment-17463518 ] ASF GitHub Bot commented on PHOENIX-6373: - gokceni opened a new pull request #1368: URL: https://github.com/apache/phoenix/pull/1368 …ransforming table (#701) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Schema changes that require table re-writes can be supported (Online data > format changes) > - > > Key: PHOENIX-6373 > URL: https://issues.apache.org/jira/browse/PHOENIX-6373 > Project: Phoenix > Issue Type: Improvement >Reporter: Gokcen Iskender >Assignee: Gokcen Iskender >Priority: Major > > Today, using ALTER TABLE or ALTER INDEX commands, the user can make certain > changes to the schema. For example, changing certain table properties like > TTL and immutability, adding nullable columns/dropping non-pk columns are > allowed as well as certain index state changes. All of the allowed changes > don’t require the table to be re-written. As soon as the ALTER command > returns, most of the changes became available immediately (eg. index disable, > TTL) but some of them might take some time and the syntax lets you to specify > async (eg. Rebuild index) and depending on the client cache settings, some > changes never make it to the client (eg. select * that run from a client > never seeing the new column since its schema cache is not updated). > If the user wants to change the schema properties that require table > re-writes, it is blocked and ALTER fails. Phoenix lacks of the ability to > change some of the table schemas and attributes, such changing the row key > (primary keys), the type of a column, the table storage format, the column > encoding, etc. There is no way to make this changes with no or very minimal > service interruption. > Design doc link: > [https://docs.google.com/document/d/1D24zRETMEetXvc3MSZj9WKYeUnooQX5BLv6gSDEmwfk/edit?usp=sharing] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] gokceni opened a new pull request #1368: PHOENIX-6373 IndexRegionObserver changes for creating mutations for transforming t…
gokceni opened a new pull request #1368: URL: https://github.com/apache/phoenix/pull/1368 …ransforming table (#701) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6612) Add TransformTool
[ https://issues.apache.org/jira/browse/PHOENIX-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463509#comment-17463509 ] ASF GitHub Bot commented on PHOENIX-6612: - gokceni commented on pull request #1367: URL: https://github.com/apache/phoenix/pull/1367#issuecomment-999143739 merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add TransformTool > - > > Key: PHOENIX-6612 > URL: https://issues.apache.org/jira/browse/PHOENIX-6612 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] gokceni closed pull request #1367: PHOENIX-6612 Add TransformTool
gokceni closed pull request #1367: URL: https://github.com/apache/phoenix/pull/1367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6612) Add TransformTool
[ https://issues.apache.org/jira/browse/PHOENIX-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463507#comment-17463507 ] ASF GitHub Bot commented on PHOENIX-6612: - gokceni closed pull request #1366: URL: https://github.com/apache/phoenix/pull/1366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add TransformTool > - > > Key: PHOENIX-6612 > URL: https://issues.apache.org/jira/browse/PHOENIX-6612 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6612) Add TransformTool
[ https://issues.apache.org/jira/browse/PHOENIX-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463506#comment-17463506 ] ASF GitHub Bot commented on PHOENIX-6612: - gokceni commented on pull request #1366: URL: https://github.com/apache/phoenix/pull/1366#issuecomment-999143621 merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add TransformTool > - > > Key: PHOENIX-6612 > URL: https://issues.apache.org/jira/browse/PHOENIX-6612 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] gokceni commented on pull request #1366: PHOENIX-6612 Add TransformTool
gokceni commented on pull request #1366: URL: https://github.com/apache/phoenix/pull/1366#issuecomment-999143621 merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6612) Add TransformTool
[ https://issues.apache.org/jira/browse/PHOENIX-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463508#comment-17463508 ] ASF GitHub Bot commented on PHOENIX-6612: - gokceni closed pull request #1367: URL: https://github.com/apache/phoenix/pull/1367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add TransformTool > - > > Key: PHOENIX-6612 > URL: https://issues.apache.org/jira/browse/PHOENIX-6612 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] gokceni closed pull request #1366: PHOENIX-6612 Add TransformTool
gokceni closed pull request #1366: URL: https://github.com/apache/phoenix/pull/1366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [phoenix] gokceni commented on pull request #1367: PHOENIX-6612 Add TransformTool
gokceni commented on pull request #1367: URL: https://github.com/apache/phoenix/pull/1367#issuecomment-999143739 merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6612) Add TransformTool
[ https://issues.apache.org/jira/browse/PHOENIX-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463476#comment-17463476 ] ASF GitHub Bot commented on PHOENIX-6612: - gokceni opened a new pull request #1367: URL: https://github.com/apache/phoenix/pull/1367 Co-authored-by: Gokcen Iskender <47044859+gokc...@users.noreply.github.com> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add TransformTool > - > > Key: PHOENIX-6612 > URL: https://issues.apache.org/jira/browse/PHOENIX-6612 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] gokceni opened a new pull request #1367: PHOENIX-6612 Add TransformTool
gokceni opened a new pull request #1367: URL: https://github.com/apache/phoenix/pull/1367 Co-authored-by: Gokcen Iskender <47044859+gokc...@users.noreply.github.com> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6612) Add TransformTool
[ https://issues.apache.org/jira/browse/PHOENIX-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463473#comment-17463473 ] ASF GitHub Bot commented on PHOENIX-6612: - gokceni commented on pull request #1366: URL: https://github.com/apache/phoenix/pull/1366#issuecomment-999110611 Jenkins test this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add TransformTool > - > > Key: PHOENIX-6612 > URL: https://issues.apache.org/jira/browse/PHOENIX-6612 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] gokceni commented on pull request #1366: PHOENIX-6612 Add TransformTool
gokceni commented on pull request #1366: URL: https://github.com/apache/phoenix/pull/1366#issuecomment-999110611 Jenkins test this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6611) Fix IndexTool -snap option and set VERIFIED in PhoenixIndexImportDirectReducer
[ https://issues.apache.org/jira/browse/PHOENIX-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463428#comment-17463428 ] ASF GitHub Bot commented on PHOENIX-6611: - gokceni commented on pull request #1365: URL: https://github.com/apache/phoenix/pull/1365#issuecomment-999030788 @stoty Thank you for implementing suggestions. There seems to be unit test failures. Could you please check if they are related to this patch? It LGTM but please make sure that tests are not broken. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix IndexTool -snap option and set VERIFIED in > PhoenixIndexImportDirectReducer > > > Key: PHOENIX-6611 > URL: https://issues.apache.org/jira/browse/PHOENIX-6611 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 5.2.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > > While fixing a trivial problem I introduced in PHOENIX-6601, I have expanded > the ITs, and found a several bugs with IndexTool. > * Indexes generated by PhoenixIndexImportDirectReducer do not handle rows > modified after the index table was created. (related to PHOENIX-5375, > PHOENIX-5376) > * Global secondary index cells generated by PhoenixIndexImportDirectReducer > do not have the VERIFIED flag set (similar to PHOENIX-6386) > * IndexTool cannot be run on the same table multiple withh -snapshot, unless > the snapshot is manually deleted. > * Global secondary index cells on transactional tables do not have the > VERIFIED flag set. > * Invalid snapshot name is generated for snapshot with namespace mapping > enabled. (PHOENIX-6601 regression) > Some more test issues: > * IndexToolIT is not run with snapshots, or with namespace mapping enabled. > * The obsolete directApi option is all over the test code. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] gokceni commented on pull request #1365: PHOENIX-6611 Multiple IndexTool bugs
gokceni commented on pull request #1365: URL: https://github.com/apache/phoenix/pull/1365#issuecomment-999030788 @stoty Thank you for implementing suggestions. There seems to be unit test failures. Could you please check if they are related to this patch? It LGTM but please make sure that tests are not broken. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6612) Add TransformTool
[
https://issues.apache.org/jira/browse/PHOENIX-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463417#comment-17463417
]
ASF GitHub Bot commented on PHOENIX-6612:
-
gokceni commented on a change in pull request #1366:
URL: https://github.com/apache/phoenix/pull/1366#discussion_r773376594
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/transform/TransformMaintainer.java
##
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.schema.transform;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.Lists;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Sets;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.ByteStringer;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.io.WritableUtils;
+import org.apache.phoenix.coprocessor.generated.ServerCachingProtos;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.ExpressionType;
+
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.covered.update.ColumnReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.util.KeyValueBuilder;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.ColumnFamilyNotFoundException;
+import org.apache.phoenix.schema.ColumnNotFoundException;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PColumnFamily;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.SaltingUtil;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.EncodedColumnsUtil;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TrustedByteArrayOutputStream;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static
org.apache.phoenix.schema.PTable.QualifierEncodingScheme.NON_ENCODED_QUALIFIERS;
+
+public class TransformMaintainer extends IndexMaintainer {
+private boolean isMultiTenant;
+// indexed expressions that are not present in the row key of the data
table, the expression can also refer to a regular column
+private List newTableExpressions;
+private Set newTableColumns;
+
+private List newTableColumnTypes;
+private int newTableColumnCount;
+private byte[] newTableName;
+private int nNewTableSaltBuckets;
+private byte[] oldTableEmptyKeyValueCF;
+private ImmutableBytesPtr emptyKeyValueCFPtr;
+private int nOldTableCFs;
+private boolean newTableWALDisabled;
+private boolean newTableImmutableRows;
+// Transient state
+private final boolean isOldTableSalted;
+private final RowKeySchema oldTableRowKeySchema;
+
+private int estimatedNewTableRowKeyBytes;
+private ColumnReference newTableEmptyKeyValueRef;
+private ColumnReference oldTableEmptyKeyValueRef;
+private boolean newTableRowKeyOrderOptimizable;
+
+private PTable.QualifierEncodingScheme newTableEncodingScheme;
+private PTable.ImmutableStorageScheme newTableImmutableStorageScheme;
+private PTable.QualifierEncodingScheme oldTableEncodingScheme;
+private PTable.ImmutableStorageScheme oldTableImmutableStorageScheme;
+/*
+ * The first part of the pair is column family name
+ * and second part is the column name. The reason we need to track this
state is because for certain storage
[GitHub] [phoenix] gokceni commented on a change in pull request #1366: PHOENIX-6612 Add TransformTool
gokceni commented on a change in pull request #1366:
URL: https://github.com/apache/phoenix/pull/1366#discussion_r773376594
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/transform/TransformMaintainer.java
##
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.schema.transform;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.Lists;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Sets;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.ByteStringer;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.io.WritableUtils;
+import org.apache.phoenix.coprocessor.generated.ServerCachingProtos;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.ExpressionType;
+
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.covered.update.ColumnReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.util.KeyValueBuilder;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.ColumnFamilyNotFoundException;
+import org.apache.phoenix.schema.ColumnNotFoundException;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PColumnFamily;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.PTableType;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.SaltingUtil;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.EncodedColumnsUtil;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TrustedByteArrayOutputStream;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static
org.apache.phoenix.schema.PTable.QualifierEncodingScheme.NON_ENCODED_QUALIFIERS;
+
+public class TransformMaintainer extends IndexMaintainer {
+private boolean isMultiTenant;
+// indexed expressions that are not present in the row key of the data
table, the expression can also refer to a regular column
+private List newTableExpressions;
+private Set newTableColumns;
+
+private List newTableColumnTypes;
+private int newTableColumnCount;
+private byte[] newTableName;
+private int nNewTableSaltBuckets;
+private byte[] oldTableEmptyKeyValueCF;
+private ImmutableBytesPtr emptyKeyValueCFPtr;
+private int nOldTableCFs;
+private boolean newTableWALDisabled;
+private boolean newTableImmutableRows;
+// Transient state
+private final boolean isOldTableSalted;
+private final RowKeySchema oldTableRowKeySchema;
+
+private int estimatedNewTableRowKeyBytes;
+private ColumnReference newTableEmptyKeyValueRef;
+private ColumnReference oldTableEmptyKeyValueRef;
+private boolean newTableRowKeyOrderOptimizable;
+
+private PTable.QualifierEncodingScheme newTableEncodingScheme;
+private PTable.ImmutableStorageScheme newTableImmutableStorageScheme;
+private PTable.QualifierEncodingScheme oldTableEncodingScheme;
+private PTable.ImmutableStorageScheme oldTableImmutableStorageScheme;
+/*
+ * The first part of the pair is column family name
+ * and second part is the column name. The reason we need to track this
state is because for certain storage schemes
+ * like ImmutableStorageScheme#SINGLE_CELL_ARRAY_WITH_OFFSETS, the column
for which we need to generate an new
+ * table put/delete is different from the old columns in the phoenix
schema.
+ */
+private Set>
[jira] [Commented] (PHOENIX-6612) Add TransformTool
[
https://issues.apache.org/jira/browse/PHOENIX-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463409#comment-17463409
]
ASF GitHub Bot commented on PHOENIX-6612:
-
gokceni commented on a change in pull request #1366:
URL: https://github.com/apache/phoenix/pull/1366#discussion_r773370346
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/transform/Transform.java
##
@@ -157,64 +307,70 @@ public static void addTransform(
}
-public static SystemTransformRecord getTransformRecord(
-String schema, String logicalTableName, String logicalParentName,
String tenantId, PhoenixConnection connection) throws SQLException {
-try (ResultSet resultSet = connection.prepareStatement("SELECT " +
-PhoenixDatabaseMetaData.TENANT_ID + ", " +
-PhoenixDatabaseMetaData.TABLE_SCHEM + ", " +
-PhoenixDatabaseMetaData.LOGICAL_TABLE_NAME + ", " +
-PhoenixDatabaseMetaData.NEW_PHYS_TABLE_NAME + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_TYPE + ", " +
-PhoenixDatabaseMetaData.LOGICAL_PARENT_NAME + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_STATUS + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_JOB_ID + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_RETRY_COUNT + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_START_TS + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_END_TS + ", " +
-PhoenixDatabaseMetaData.OLD_METADATA + " , " +
-PhoenixDatabaseMetaData.NEW_METADATA + " , " +
-PhoenixDatabaseMetaData.TRANSFORM_FUNCTION +
-" FROM " + PhoenixDatabaseMetaData.SYSTEM_TRANSFORM_NAME + "
WHERE " +
-(Strings.isNullOrEmpty(tenantId) ? "" :
(PhoenixDatabaseMetaData.TENANT_ID + " ='" + tenantId + "' AND ")) +
-(Strings.isNullOrEmpty(schema) ? "" :
(PhoenixDatabaseMetaData.TABLE_SCHEM + " ='" + schema + "' AND ")) +
-PhoenixDatabaseMetaData.LOGICAL_TABLE_NAME + " ='" +
logicalTableName + "'" +
-(Strings.isNullOrEmpty(logicalParentName) ? "": (" AND " +
PhoenixDatabaseMetaData.LOGICAL_PARENT_NAME + "='" + logicalParentName + "'" ))
-).executeQuery()) {
-if (resultSet.next()) {
-return
SystemTransformRecord.SystemTransformBuilder.build(resultSet);
+public static void completeTransform(Connection connection, Configuration
configuration) throws Exception{
+// Will be called from Reducer
+long timestmp= EnvironmentEdgeManager.currentTimeMillis();
+String tenantId = configuration.get(MAPREDUCE_TENANT_ID, null);
+String fullOldTableName =
PhoenixConfigurationUtil.getInputTableName(configuration);
+String schemaName =
SchemaUtil.getSchemaNameFromFullName(fullOldTableName);
+String oldTableLogicalName =
SchemaUtil.getTableNameFromFullName(fullOldTableName);
+String indexTableName =
SchemaUtil.getTableNameFromFullName(PhoenixConfigurationUtil.getIndexToolIndexTableName(configuration));
+String logicaTableName = oldTableLogicalName;
+String logicalParentName = null;
+if (PhoenixConfigurationUtil.getTransformingTableType(configuration)
== IndexScrutinyTool.SourceTable.INDEX_TABLE_SOURCE)
+if (!Strings.isNullOrEmpty(indexTableName)) {
+logicaTableName = indexTableName;
+logicalParentName = SchemaUtil.getTableName(schemaName,
oldTableLogicalName);
}
-return null;
-}
-}
-
-public static boolean checkIsTransformNeeded(MetaDataClient.MetaProperties
metaProperties, String schemaName,
- PTable table, String
logicalTableName, String parentTableName,
- String tenantId,
PhoenixConnection connection) throws SQLException {
-boolean isTransformNeeded = isTransformNeeded(metaProperties, table);
-if (isTransformNeeded) {
-SystemTransformRecord existingTransform =
Transform.getTransformRecord(schemaName, logicalTableName, parentTableName,
tenantId,connection);
-if (existingTransform != null && existingTransform.isActive()) {
-throw new SQLExceptionInfo.Builder(
-
SQLExceptionCode.CANNOT_TRANSFORM_ALREADY_TRANSFORMING_TABLE)
-.setMessage(" Only one transform at a time is allowed
")
-
.setSchemaName(schemaName).setTableName(logicalTableName).build().buildException();
+boolean isPartial =
PhoenixConfigurationUtil.getIsPartialTransform(configuration);
+SystemTransformRecord transformRecord = getTransformRecord(schemaName,
logicaTableName, logicalParentName,
+tenantId,
[GitHub] [phoenix] gokceni commented on a change in pull request #1366: PHOENIX-6612 Add TransformTool
gokceni commented on a change in pull request #1366:
URL: https://github.com/apache/phoenix/pull/1366#discussion_r773370346
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/transform/Transform.java
##
@@ -157,64 +307,70 @@ public static void addTransform(
}
-public static SystemTransformRecord getTransformRecord(
-String schema, String logicalTableName, String logicalParentName,
String tenantId, PhoenixConnection connection) throws SQLException {
-try (ResultSet resultSet = connection.prepareStatement("SELECT " +
-PhoenixDatabaseMetaData.TENANT_ID + ", " +
-PhoenixDatabaseMetaData.TABLE_SCHEM + ", " +
-PhoenixDatabaseMetaData.LOGICAL_TABLE_NAME + ", " +
-PhoenixDatabaseMetaData.NEW_PHYS_TABLE_NAME + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_TYPE + ", " +
-PhoenixDatabaseMetaData.LOGICAL_PARENT_NAME + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_STATUS + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_JOB_ID + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_RETRY_COUNT + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_START_TS + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_END_TS + ", " +
-PhoenixDatabaseMetaData.OLD_METADATA + " , " +
-PhoenixDatabaseMetaData.NEW_METADATA + " , " +
-PhoenixDatabaseMetaData.TRANSFORM_FUNCTION +
-" FROM " + PhoenixDatabaseMetaData.SYSTEM_TRANSFORM_NAME + "
WHERE " +
-(Strings.isNullOrEmpty(tenantId) ? "" :
(PhoenixDatabaseMetaData.TENANT_ID + " ='" + tenantId + "' AND ")) +
-(Strings.isNullOrEmpty(schema) ? "" :
(PhoenixDatabaseMetaData.TABLE_SCHEM + " ='" + schema + "' AND ")) +
-PhoenixDatabaseMetaData.LOGICAL_TABLE_NAME + " ='" +
logicalTableName + "'" +
-(Strings.isNullOrEmpty(logicalParentName) ? "": (" AND " +
PhoenixDatabaseMetaData.LOGICAL_PARENT_NAME + "='" + logicalParentName + "'" ))
-).executeQuery()) {
-if (resultSet.next()) {
-return
SystemTransformRecord.SystemTransformBuilder.build(resultSet);
+public static void completeTransform(Connection connection, Configuration
configuration) throws Exception{
+// Will be called from Reducer
+long timestmp= EnvironmentEdgeManager.currentTimeMillis();
+String tenantId = configuration.get(MAPREDUCE_TENANT_ID, null);
+String fullOldTableName =
PhoenixConfigurationUtil.getInputTableName(configuration);
+String schemaName =
SchemaUtil.getSchemaNameFromFullName(fullOldTableName);
+String oldTableLogicalName =
SchemaUtil.getTableNameFromFullName(fullOldTableName);
+String indexTableName =
SchemaUtil.getTableNameFromFullName(PhoenixConfigurationUtil.getIndexToolIndexTableName(configuration));
+String logicaTableName = oldTableLogicalName;
+String logicalParentName = null;
+if (PhoenixConfigurationUtil.getTransformingTableType(configuration)
== IndexScrutinyTool.SourceTable.INDEX_TABLE_SOURCE)
+if (!Strings.isNullOrEmpty(indexTableName)) {
+logicaTableName = indexTableName;
+logicalParentName = SchemaUtil.getTableName(schemaName,
oldTableLogicalName);
}
-return null;
-}
-}
-
-public static boolean checkIsTransformNeeded(MetaDataClient.MetaProperties
metaProperties, String schemaName,
- PTable table, String
logicalTableName, String parentTableName,
- String tenantId,
PhoenixConnection connection) throws SQLException {
-boolean isTransformNeeded = isTransformNeeded(metaProperties, table);
-if (isTransformNeeded) {
-SystemTransformRecord existingTransform =
Transform.getTransformRecord(schemaName, logicalTableName, parentTableName,
tenantId,connection);
-if (existingTransform != null && existingTransform.isActive()) {
-throw new SQLExceptionInfo.Builder(
-
SQLExceptionCode.CANNOT_TRANSFORM_ALREADY_TRANSFORMING_TABLE)
-.setMessage(" Only one transform at a time is allowed
")
-
.setSchemaName(schemaName).setTableName(logicalTableName).build().buildException();
+boolean isPartial =
PhoenixConfigurationUtil.getIsPartialTransform(configuration);
+SystemTransformRecord transformRecord = getTransformRecord(schemaName,
logicaTableName, logicalParentName,
+tenantId, connection.unwrap(PhoenixConnection.class));
+if (!isPartial) {
+String newTableName =
SchemaUtil.getTableNameFromFullName(transformRecord.getNewPhysicalTableName());
+PTable pNewTable = PhoenixRuntime.getTable(connection,
[jira] [Commented] (PHOENIX-6615) The Tephra transaction processor cannot be loaded anymore.
[ https://issues.apache.org/jira/browse/PHOENIX-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463377#comment-17463377 ] Lars Hofhansl commented on PHOENIX-6615: I guess the breakage was hard to notice, since the tests are (necessarily) disabled for the NotAvailableTransactionProvider. > The Tephra transaction processor cannot be loaded anymore. > -- > > Key: PHOENIX-6615 > URL: https://issues.apache.org/jira/browse/PHOENIX-6615 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.1.2 >Reporter: Lars Hofhansl >Priority: Major > Attachments: 6615.txt > > > See > # TransactionFactory > # TephraTransactionProvider > Can you spot the problem? :) (Hint: The constructor is private.) > Broken since PHOENIX-6064. [~stoty] . > Can I just say... Unless I am missing something... How could we not have > noticed that one of the transaction processors has not been working since > August (in 5.x at least)? Is really nobody using the transaction engines? > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6615) The Tephra transaction processor cannot be loaded anymore.
[ https://issues.apache.org/jira/browse/PHOENIX-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463376#comment-17463376 ] Lars Hofhansl commented on PHOENIX-6615: Thanks [~stoty] . I agree the bitrot is around Tephra, not Phoenix. And the job you do maintaining Tephra is appreciated! Omid and Tephra have different pros and cons. Tephra has no (or negligible) per row cost, but a higher per transaction cost (since all failed or rolled-back transactions have to be sent to all future transaction until they are collected), Omid on the other hand has a fairly high per row cost (as the shadow columns need to be updated) but no lasting per transactions cost (once the shadow columns are updated no further information is needed). So I'd use Omid for many small transactions, and Tephra for few but large transactions... And maybe it's really not worth it. :) I agree with you that maintaining the perception that a project is maintained is worth than ripping it out. As for transactions in general... Phoenix is the only engine I know that supports OLTP like interactions (JDBC et al), and high-volume interactions (via regions, guideposts and M/R, Spark, and Trino integrations), and transactions. It's something that really sets Phoenix apart. It would be just a shame to let that rot. Yes, let's take it up on the dev list. My priorities have since changed more into the bigger ecosystem of data (Trino, Iceberg, Spark, Federation, Real-time Ingestion, M/L, Governance, etc). That said, Phoenix has still a place in that ecosystem as I mentioned before, and so I am still interested at that level. > The Tephra transaction processor cannot be loaded anymore. > -- > > Key: PHOENIX-6615 > URL: https://issues.apache.org/jira/browse/PHOENIX-6615 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.1.2 >Reporter: Lars Hofhansl >Priority: Major > Attachments: 6615.txt > > > See > # TransactionFactory > # TephraTransactionProvider > Can you spot the problem? :) (Hint: The constructor is private.) > Broken since PHOENIX-6064. [~stoty] . > Can I just say... Unless I am missing something... How could we not have > noticed that one of the transaction processors has not been working since > August (in 5.x at least)? Is really nobody using the transaction engines? > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6612) Add TransformTool
[
https://issues.apache.org/jira/browse/PHOENIX-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463150#comment-17463150
]
ASF GitHub Bot commented on PHOENIX-6612:
-
virajjasani commented on a change in pull request #1366:
URL: https://github.com/apache/phoenix/pull/1366#discussion_r773012137
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/transform/Transform.java
##
@@ -157,64 +307,70 @@ public static void addTransform(
}
-public static SystemTransformRecord getTransformRecord(
-String schema, String logicalTableName, String logicalParentName,
String tenantId, PhoenixConnection connection) throws SQLException {
-try (ResultSet resultSet = connection.prepareStatement("SELECT " +
-PhoenixDatabaseMetaData.TENANT_ID + ", " +
-PhoenixDatabaseMetaData.TABLE_SCHEM + ", " +
-PhoenixDatabaseMetaData.LOGICAL_TABLE_NAME + ", " +
-PhoenixDatabaseMetaData.NEW_PHYS_TABLE_NAME + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_TYPE + ", " +
-PhoenixDatabaseMetaData.LOGICAL_PARENT_NAME + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_STATUS + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_JOB_ID + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_RETRY_COUNT + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_START_TS + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_END_TS + ", " +
-PhoenixDatabaseMetaData.OLD_METADATA + " , " +
-PhoenixDatabaseMetaData.NEW_METADATA + " , " +
-PhoenixDatabaseMetaData.TRANSFORM_FUNCTION +
-" FROM " + PhoenixDatabaseMetaData.SYSTEM_TRANSFORM_NAME + "
WHERE " +
-(Strings.isNullOrEmpty(tenantId) ? "" :
(PhoenixDatabaseMetaData.TENANT_ID + " ='" + tenantId + "' AND ")) +
-(Strings.isNullOrEmpty(schema) ? "" :
(PhoenixDatabaseMetaData.TABLE_SCHEM + " ='" + schema + "' AND ")) +
-PhoenixDatabaseMetaData.LOGICAL_TABLE_NAME + " ='" +
logicalTableName + "'" +
-(Strings.isNullOrEmpty(logicalParentName) ? "": (" AND " +
PhoenixDatabaseMetaData.LOGICAL_PARENT_NAME + "='" + logicalParentName + "'" ))
-).executeQuery()) {
-if (resultSet.next()) {
-return
SystemTransformRecord.SystemTransformBuilder.build(resultSet);
+public static void completeTransform(Connection connection, Configuration
configuration) throws Exception{
+// Will be called from Reducer
+long timestmp= EnvironmentEdgeManager.currentTimeMillis();
+String tenantId = configuration.get(MAPREDUCE_TENANT_ID, null);
+String fullOldTableName =
PhoenixConfigurationUtil.getInputTableName(configuration);
+String schemaName =
SchemaUtil.getSchemaNameFromFullName(fullOldTableName);
+String oldTableLogicalName =
SchemaUtil.getTableNameFromFullName(fullOldTableName);
+String indexTableName =
SchemaUtil.getTableNameFromFullName(PhoenixConfigurationUtil.getIndexToolIndexTableName(configuration));
+String logicaTableName = oldTableLogicalName;
+String logicalParentName = null;
+if (PhoenixConfigurationUtil.getTransformingTableType(configuration)
== IndexScrutinyTool.SourceTable.INDEX_TABLE_SOURCE)
+if (!Strings.isNullOrEmpty(indexTableName)) {
+logicaTableName = indexTableName;
+logicalParentName = SchemaUtil.getTableName(schemaName,
oldTableLogicalName);
}
-return null;
-}
-}
-
-public static boolean checkIsTransformNeeded(MetaDataClient.MetaProperties
metaProperties, String schemaName,
- PTable table, String
logicalTableName, String parentTableName,
- String tenantId,
PhoenixConnection connection) throws SQLException {
-boolean isTransformNeeded = isTransformNeeded(metaProperties, table);
-if (isTransformNeeded) {
-SystemTransformRecord existingTransform =
Transform.getTransformRecord(schemaName, logicalTableName, parentTableName,
tenantId,connection);
-if (existingTransform != null && existingTransform.isActive()) {
-throw new SQLExceptionInfo.Builder(
-
SQLExceptionCode.CANNOT_TRANSFORM_ALREADY_TRANSFORMING_TABLE)
-.setMessage(" Only one transform at a time is allowed
")
-
.setSchemaName(schemaName).setTableName(logicalTableName).build().buildException();
+boolean isPartial =
PhoenixConfigurationUtil.getIsPartialTransform(configuration);
+SystemTransformRecord transformRecord = getTransformRecord(schemaName,
logicaTableName, logicalParentName,
+tenantId,
[GitHub] [phoenix] virajjasani commented on a change in pull request #1366: PHOENIX-6612 Add TransformTool
virajjasani commented on a change in pull request #1366:
URL: https://github.com/apache/phoenix/pull/1366#discussion_r773012137
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/transform/Transform.java
##
@@ -157,64 +307,70 @@ public static void addTransform(
}
-public static SystemTransformRecord getTransformRecord(
-String schema, String logicalTableName, String logicalParentName,
String tenantId, PhoenixConnection connection) throws SQLException {
-try (ResultSet resultSet = connection.prepareStatement("SELECT " +
-PhoenixDatabaseMetaData.TENANT_ID + ", " +
-PhoenixDatabaseMetaData.TABLE_SCHEM + ", " +
-PhoenixDatabaseMetaData.LOGICAL_TABLE_NAME + ", " +
-PhoenixDatabaseMetaData.NEW_PHYS_TABLE_NAME + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_TYPE + ", " +
-PhoenixDatabaseMetaData.LOGICAL_PARENT_NAME + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_STATUS + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_JOB_ID + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_RETRY_COUNT + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_START_TS + ", " +
-PhoenixDatabaseMetaData.TRANSFORM_END_TS + ", " +
-PhoenixDatabaseMetaData.OLD_METADATA + " , " +
-PhoenixDatabaseMetaData.NEW_METADATA + " , " +
-PhoenixDatabaseMetaData.TRANSFORM_FUNCTION +
-" FROM " + PhoenixDatabaseMetaData.SYSTEM_TRANSFORM_NAME + "
WHERE " +
-(Strings.isNullOrEmpty(tenantId) ? "" :
(PhoenixDatabaseMetaData.TENANT_ID + " ='" + tenantId + "' AND ")) +
-(Strings.isNullOrEmpty(schema) ? "" :
(PhoenixDatabaseMetaData.TABLE_SCHEM + " ='" + schema + "' AND ")) +
-PhoenixDatabaseMetaData.LOGICAL_TABLE_NAME + " ='" +
logicalTableName + "'" +
-(Strings.isNullOrEmpty(logicalParentName) ? "": (" AND " +
PhoenixDatabaseMetaData.LOGICAL_PARENT_NAME + "='" + logicalParentName + "'" ))
-).executeQuery()) {
-if (resultSet.next()) {
-return
SystemTransformRecord.SystemTransformBuilder.build(resultSet);
+public static void completeTransform(Connection connection, Configuration
configuration) throws Exception{
+// Will be called from Reducer
+long timestmp= EnvironmentEdgeManager.currentTimeMillis();
+String tenantId = configuration.get(MAPREDUCE_TENANT_ID, null);
+String fullOldTableName =
PhoenixConfigurationUtil.getInputTableName(configuration);
+String schemaName =
SchemaUtil.getSchemaNameFromFullName(fullOldTableName);
+String oldTableLogicalName =
SchemaUtil.getTableNameFromFullName(fullOldTableName);
+String indexTableName =
SchemaUtil.getTableNameFromFullName(PhoenixConfigurationUtil.getIndexToolIndexTableName(configuration));
+String logicaTableName = oldTableLogicalName;
+String logicalParentName = null;
+if (PhoenixConfigurationUtil.getTransformingTableType(configuration)
== IndexScrutinyTool.SourceTable.INDEX_TABLE_SOURCE)
+if (!Strings.isNullOrEmpty(indexTableName)) {
+logicaTableName = indexTableName;
+logicalParentName = SchemaUtil.getTableName(schemaName,
oldTableLogicalName);
}
-return null;
-}
-}
-
-public static boolean checkIsTransformNeeded(MetaDataClient.MetaProperties
metaProperties, String schemaName,
- PTable table, String
logicalTableName, String parentTableName,
- String tenantId,
PhoenixConnection connection) throws SQLException {
-boolean isTransformNeeded = isTransformNeeded(metaProperties, table);
-if (isTransformNeeded) {
-SystemTransformRecord existingTransform =
Transform.getTransformRecord(schemaName, logicalTableName, parentTableName,
tenantId,connection);
-if (existingTransform != null && existingTransform.isActive()) {
-throw new SQLExceptionInfo.Builder(
-
SQLExceptionCode.CANNOT_TRANSFORM_ALREADY_TRANSFORMING_TABLE)
-.setMessage(" Only one transform at a time is allowed
")
-
.setSchemaName(schemaName).setTableName(logicalTableName).build().buildException();
+boolean isPartial =
PhoenixConfigurationUtil.getIsPartialTransform(configuration);
+SystemTransformRecord transformRecord = getTransformRecord(schemaName,
logicaTableName, logicalParentName,
+tenantId, connection.unwrap(PhoenixConnection.class));
+if (!isPartial) {
+String newTableName =
SchemaUtil.getTableNameFromFullName(transformRecord.getNewPhysicalTableName());
+PTable pNewTable =
[jira] [Commented] (PHOENIX-6616) Alter table command can be used to set normalization_enabled=true on salted tables
[ https://issues.apache.org/jira/browse/PHOENIX-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463079#comment-17463079 ] Aman Poonia commented on PHOENIX-6616: -- [~stoty] [~dbwong] FYI > Alter table command can be used to set normalization_enabled=true on salted > tables > -- > > Key: PHOENIX-6616 > URL: https://issues.apache.org/jira/browse/PHOENIX-6616 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.16.1, 5.1.2 >Reporter: Aman Poonia >Priority: Major > > Here is what i found > CREATE TABLE IF NOT EXISTS table1(a BIGINT NOT NULL, b BIGINT NOT NULL > CONSTRAINT PK PRIMARY KEY (a, b)) TTL=7776000, NORMALIZATION_ENABLED=true, > SALT_BUCKETS=16, DISABLE_TABLE_SOR=true, NORMALIZER_TARGET_REGION_SIZE=5200; > Error: ERROR 1147 (42Y86): Should not enable normalizer on salted table. > tableName=TABLE1 (state=42Y86,code=1147) > java.sql.SQLException: ERROR 1147 (42Y86): Should not enable normalizer on > salted table. tableName=TABLE1 > at > org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:618) > at > org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:228) > at > org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1462) > at > org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1983) > at > org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:3094) > at > org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1118) > at > org.apache.phoenix.compile.CreateTableCompiler$CreateTableMutationPlan.execute(CreateTableCompiler.java:421) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:516) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:482) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:481) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:469) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:2036) > at sqlline.Commands.execute(Commands.java:814) > at sqlline.Commands.sql(Commands.java:754) > > > 0: jdbc:phoenix:localhost:50141> CREATE TABLE IF NOT EXISTS table1(a BIGINT > NOT NULL, b BIGINT NOT NULL CONSTRAINT PK PRIMARY KEY (a, b)) TTL=7776000, > NORMALIZATION_ENABLED=false, SALT_BUCKETS=16, DISABLE_TABLE_SOR=true, > NORMALIZER_TARGET_REGION_SIZE=5200; > No rows affected (2.295 seconds) > 0: jdbc:phoenix:localhost:50141> ALTER TABLE table1 set > NORMALIZATION_ENABLED=true; > No rows affected (1.374 seconds) > So basically we are still able to set the normalization if we go through the > alter table. -- This message was sent by Atlassian Jira (v8.20.1#820001)
