[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[
https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457504#comment-17457504
]
ASF GitHub Bot commented on PHOENIX-6603:
-
gokceni commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r767076745
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/TableProperty.java
##
@@ -182,7 +182,7 @@ public Object getPTableValue(PTable table) {
},
-COLUMN_ENCODED_BYTES(PhoenixDatabaseMetaData.ENCODING_SCHEME,
COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY, false, false, false) {
+COLUMN_ENCODED_BYTES(PhoenixDatabaseMetaData.ENCODING_SCHEME,
COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY, true, false, false) {
Review comment:
This property was used for immutable tables only I think. Now we changed
the behavior so that mutable indexes can benefit from this. It is not a
property reserved for immutable tables. And we didn't have any test to use this
property on mutable tables/indexes I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create SYSTEM.TRANSFORM table
> -
>
> Key: PHOENIX-6603
> URL: https://issues.apache.org/jira/browse/PHOENIX-6603
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Gokcen Iskender
>Priority: Major
>
> SYSTEM.TRANSFORM is a table for bookkeeping the transform process
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [phoenix] gokceni commented on a change in pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
gokceni commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r767076745
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/TableProperty.java
##
@@ -182,7 +182,7 @@ public Object getPTableValue(PTable table) {
},
-COLUMN_ENCODED_BYTES(PhoenixDatabaseMetaData.ENCODING_SCHEME,
COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY, false, false, false) {
+COLUMN_ENCODED_BYTES(PhoenixDatabaseMetaData.ENCODING_SCHEME,
COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY, true, false, false) {
Review comment:
This property was used for immutable tables only I think. Now we changed
the behavior so that mutable indexes can benefit from this. It is not a
property reserved for immutable tables. And we didn't have any test to use this
property on mutable tables/indexes I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[
https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457503#comment-17457503
]
ASF GitHub Bot commented on PHOENIX-6603:
-
gokceni commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r767076397
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
##
@@ -5384,7 +5435,7 @@ else if (metaProperties.getImmutableStorageSchemeProp()
!= table.getImmutableSto
return changingPhoenixTableProperty;
}
-private static class MetaProperties {
+public static class MetaProperties {
Review comment:
It is actually required. Transform.java refers this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create SYSTEM.TRANSFORM table
> -
>
> Key: PHOENIX-6603
> URL: https://issues.apache.org/jira/browse/PHOENIX-6603
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Gokcen Iskender
>Priority: Major
>
> SYSTEM.TRANSFORM is a table for bookkeeping the transform process
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [phoenix] gokceni commented on a change in pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
gokceni commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r767076397
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
##
@@ -5384,7 +5435,7 @@ else if (metaProperties.getImmutableStorageSchemeProp()
!= table.getImmutableSto
return changingPhoenixTableProperty;
}
-private static class MetaProperties {
+public static class MetaProperties {
Review comment:
It is actually required. Transform.java refers this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (PHOENIX-6608) DISCUSS: Rethink MapReduce split generation
[ https://issues.apache.org/jira/browse/PHOENIX-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457296#comment-17457296 ] Lars Hofhansl commented on PHOENIX-6608: Yeah, if each worker uses a JDBC client and replans the query, that's a problem. Maybe not the right place here... But how does each worker know what part of the overall query it should execute? Could look at how exactly the guidepost table is queried. Does it always scan along the PK? If not we're doing full scans. There is clearly some work to do in Phoenix in this in general. The notion that scans are starting when they are placed into the task queue is insane. It might buy a little bit on latency for small queries since we start scanning before the iterators are scheduled in Phoenix, but for large queries it's a disaster... Imagine a 5000 scans start fetching up to 2mb (~4mb on the heap)... You'd have 20GB of fairly useless prefetched results for Phoenix iterators that are not even scheduled to run. Anyway. Once the Trino patch is committed, this prefetching is completely disabled there. > DISCUSS: Rethink MapReduce split generation > --- > > Key: PHOENIX-6608 > URL: https://issues.apache.org/jira/browse/PHOENIX-6608 > Project: Phoenix > Issue Type: Improvement >Reporter: Lars Hofhansl >Priority: Major > > I just ran into an issue with Trino, which uses Phoenix' M/R integration to > generate splits for its worker nodes. > See: [https://github.com/trinodb/trino/issues/10143] > And a fix: [https://github.com/trinodb/trino/pull/10153] > In short the issue is that with large data size and guideposts enabled > (default) Phoenix' RoundRobinResultIterator starts scanning when tasks are > submitted to the queue. For large datasets (per client) this fills the heap > with pre-fetches HBase result objects. > MapReduce (and Spark) integrations have presumably the same issue. > My proposed solution is instead of allowing Phoenix to do intra-split > parallelism we create more splits (the fix above groups 20 scans into a split > - 20 turned out to be a good number). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[ https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457294#comment-17457294 ] ASF GitHub Bot commented on PHOENIX-6603: - virajjasani commented on pull request #1363: URL: https://github.com/apache/phoenix/pull/1363#issuecomment-991164193 Also, while not all might be worth resolving, here is the list of spotbugs reported by Jenkins: [here](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1363/4/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create SYSTEM.TRANSFORM table > - > > Key: PHOENIX-6603 > URL: https://issues.apache.org/jira/browse/PHOENIX-6603 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > > SYSTEM.TRANSFORM is a table for bookkeeping the transform process -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] virajjasani commented on pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
virajjasani commented on pull request #1363: URL: https://github.com/apache/phoenix/pull/1363#issuecomment-991164193 Also, while not all might be worth resolving, here is the list of spotbugs reported by Jenkins: [here](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1363/4/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[ https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457292#comment-17457292 ] ASF GitHub Bot commented on PHOENIX-6603: - virajjasani commented on pull request #1363: URL: https://github.com/apache/phoenix/pull/1363#issuecomment-991162650 > > Btw the recent QA results from Jenkins look great [here](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1363/4/testReport/) > > Thanks @virajjasani I am trying to understand what the error is on the failed test No worries, that one is flaky test, not relevant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create SYSTEM.TRANSFORM table > - > > Key: PHOENIX-6603 > URL: https://issues.apache.org/jira/browse/PHOENIX-6603 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > > SYSTEM.TRANSFORM is a table for bookkeeping the transform process -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[ https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457289#comment-17457289 ] ASF GitHub Bot commented on PHOENIX-6603: - gokceni commented on pull request #1363: URL: https://github.com/apache/phoenix/pull/1363#issuecomment-991162027 > Btw the recent QA results from Jenkins look great [here](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1363/4/testReport/) Thanks @virajjasani I am trying to understand what the error is on the failed test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create SYSTEM.TRANSFORM table > - > > Key: PHOENIX-6603 > URL: https://issues.apache.org/jira/browse/PHOENIX-6603 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > > SYSTEM.TRANSFORM is a table for bookkeeping the transform process -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] virajjasani commented on pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
virajjasani commented on pull request #1363: URL: https://github.com/apache/phoenix/pull/1363#issuecomment-991162650 > > Btw the recent QA results from Jenkins look great [here](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1363/4/testReport/) > > Thanks @virajjasani I am trying to understand what the error is on the failed test No worries, that one is flaky test, not relevant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [phoenix] gokceni commented on pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
gokceni commented on pull request #1363: URL: https://github.com/apache/phoenix/pull/1363#issuecomment-991162027 > Btw the recent QA results from Jenkins look great [here](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1363/4/testReport/) Thanks @virajjasani I am trying to understand what the error is on the failed test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[ https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457288#comment-17457288 ] ASF GitHub Bot commented on PHOENIX-6603: - virajjasani edited a comment on pull request #1363: URL: https://github.com/apache/phoenix/pull/1363#issuecomment-991160634 Btw the recent QA results from Jenkins look great [here](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1363/4/testReport/) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create SYSTEM.TRANSFORM table > - > > Key: PHOENIX-6603 > URL: https://issues.apache.org/jira/browse/PHOENIX-6603 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > > SYSTEM.TRANSFORM is a table for bookkeeping the transform process -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[ https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457286#comment-17457286 ] ASF GitHub Bot commented on PHOENIX-6603: - virajjasani commented on pull request #1363: URL: https://github.com/apache/phoenix/pull/1363#issuecomment-991160634 Btw the recent QA results from Jenkins look great [here]([url](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1363/4/testReport/)). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create SYSTEM.TRANSFORM table > - > > Key: PHOENIX-6603 > URL: https://issues.apache.org/jira/browse/PHOENIX-6603 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > > SYSTEM.TRANSFORM is a table for bookkeeping the transform process -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] virajjasani commented on pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
virajjasani commented on pull request #1363: URL: https://github.com/apache/phoenix/pull/1363#issuecomment-991160634 Btw the recent QA results from Jenkins look great [here]([url](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1363/4/testReport/)). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [phoenix] virajjasani edited a comment on pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
virajjasani edited a comment on pull request #1363: URL: https://github.com/apache/phoenix/pull/1363#issuecomment-991160634 Btw the recent QA results from Jenkins look great [here](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1363/4/testReport/) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[
https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457277#comment-17457277
]
ASF GitHub Bot commented on PHOENIX-6603:
-
gokceni commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r766842493
##
File path: phoenix-core/src/main/java/org/apache/phoenix/schema/PTable.java
##
@@ -247,6 +247,58 @@ public String toString() {
},
}
+public enum TransformType {
+METADATA_TRANSFORM((byte)1);
+
+private final byte[] byteValue;
+private final int serializedValue;
+
+TransformType(int serializedValue) {
+this.serializedValue = serializedValue;
+this.byteValue = Bytes.toBytes(this.name());
+}
+
+public byte[] getBytes() {
+return byteValue;
+}
+
+public int getSerializedValue() {
+return this.serializedValue;
+}
+public static TransformType getDefault() {
+return METADATA_TRANSFORM;
+}
+public static TransformType fromSerializedValue(int serializedValue) {
+if (serializedValue < 1 || serializedValue >
TaskType.values().length) {
Review comment:
Yes. It was a copy paste error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create SYSTEM.TRANSFORM table
> -
>
> Key: PHOENIX-6603
> URL: https://issues.apache.org/jira/browse/PHOENIX-6603
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Gokcen Iskender
>Priority: Major
>
> SYSTEM.TRANSFORM is a table for bookkeeping the transform process
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [phoenix] gokceni commented on a change in pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
gokceni commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r766842493
##
File path: phoenix-core/src/main/java/org/apache/phoenix/schema/PTable.java
##
@@ -247,6 +247,58 @@ public String toString() {
},
}
+public enum TransformType {
+METADATA_TRANSFORM((byte)1);
+
+private final byte[] byteValue;
+private final int serializedValue;
+
+TransformType(int serializedValue) {
+this.serializedValue = serializedValue;
+this.byteValue = Bytes.toBytes(this.name());
+}
+
+public byte[] getBytes() {
+return byteValue;
+}
+
+public int getSerializedValue() {
+return this.serializedValue;
+}
+public static TransformType getDefault() {
+return METADATA_TRANSFORM;
+}
+public static TransformType fromSerializedValue(int serializedValue) {
+if (serializedValue < 1 || serializedValue >
TaskType.values().length) {
Review comment:
Yes. It was a copy paste error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[
https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457029#comment-17457029
]
ASF GitHub Bot commented on PHOENIX-6603:
-
virajjasani commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r765829755
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/SingleCellIndexIT.java
##
@@ -476,7 +476,7 @@ public static void dumpTable(String tableName) throws
Exception {
.entrySet()) {
byte[] family = entryF.getKey();
Review comment:
Not related to this change, but looks like we are not using `family`
anywhere? Unless we want to log family, we can remove this for-loop? But no
strong opinion, it's upto you :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create SYSTEM.TRANSFORM table
> -
>
> Key: PHOENIX-6603
> URL: https://issues.apache.org/jira/browse/PHOENIX-6603
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Gokcen Iskender
>Priority: Major
>
> SYSTEM.TRANSFORM is a table for bookkeeping the transform process
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [phoenix] virajjasani commented on a change in pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
virajjasani commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r765829755
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/SingleCellIndexIT.java
##
@@ -476,7 +476,7 @@ public static void dumpTable(String tableName) throws
Exception {
.entrySet()) {
byte[] family = entryF.getKey();
Review comment:
Not related to this change, but looks like we are not using `family`
anywhere? Unless we want to log family, we can remove this for-loop? But no
strong opinion, it's upto you :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[
https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457028#comment-17457028
]
ASF GitHub Bot commented on PHOENIX-6603:
-
virajjasani commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r766550840
##
File path: phoenix-core/src/main/java/org/apache/phoenix/schema/PTable.java
##
@@ -247,6 +247,58 @@ public String toString() {
},
}
+public enum TransformType {
+METADATA_TRANSFORM((byte)1);
+
+private final byte[] byteValue;
+private final int serializedValue;
+
+TransformType(int serializedValue) {
+this.serializedValue = serializedValue;
+this.byteValue = Bytes.toBytes(this.name());
+}
+
+public byte[] getBytes() {
+return byteValue;
+}
+
+public int getSerializedValue() {
+return this.serializedValue;
+}
+public static TransformType getDefault() {
+return METADATA_TRANSFORM;
+}
+public static TransformType fromSerializedValue(int serializedValue) {
+if (serializedValue < 1 || serializedValue >
TaskType.values().length) {
Review comment:
I was wondering how TransformIT didn't fail due to this but then
realized that since we have only one Enum value available for TransformType
i.e. `METADATA_TRANSFORM` with value 1, so when we deserialize, it will anyways
result into `TransformType.values()[0]` so we were good. Only if we had
multiple enum values, we might have faced some issues. Is that correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create SYSTEM.TRANSFORM table
> -
>
> Key: PHOENIX-6603
> URL: https://issues.apache.org/jira/browse/PHOENIX-6603
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Gokcen Iskender
>Priority: Major
>
> SYSTEM.TRANSFORM is a table for bookkeeping the transform process
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [phoenix] virajjasani commented on a change in pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
virajjasani commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r766550840
##
File path: phoenix-core/src/main/java/org/apache/phoenix/schema/PTable.java
##
@@ -247,6 +247,58 @@ public String toString() {
},
}
+public enum TransformType {
+METADATA_TRANSFORM((byte)1);
+
+private final byte[] byteValue;
+private final int serializedValue;
+
+TransformType(int serializedValue) {
+this.serializedValue = serializedValue;
+this.byteValue = Bytes.toBytes(this.name());
+}
+
+public byte[] getBytes() {
+return byteValue;
+}
+
+public int getSerializedValue() {
+return this.serializedValue;
+}
+public static TransformType getDefault() {
+return METADATA_TRANSFORM;
+}
+public static TransformType fromSerializedValue(int serializedValue) {
+if (serializedValue < 1 || serializedValue >
TaskType.values().length) {
Review comment:
I was wondering how TransformIT didn't fail due to this but then
realized that since we have only one Enum value available for TransformType
i.e. `METADATA_TRANSFORM` with value 1, so when we deserialize, it will anyways
result into `TransformType.values()[0]` so we were good. Only if we had
multiple enum values, we might have faced some issues. Is that correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[
https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457025#comment-17457025
]
ASF GitHub Bot commented on PHOENIX-6603:
-
virajjasani commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r766535782
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/TableProperty.java
##
@@ -182,7 +182,7 @@ public Object getPTableValue(PTable table) {
},
-COLUMN_ENCODED_BYTES(PhoenixDatabaseMetaData.ENCODING_SCHEME,
COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY, false, false, false) {
+COLUMN_ENCODED_BYTES(PhoenixDatabaseMetaData.ENCODING_SCHEME,
COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY, true, false, false) {
Review comment:
Nice one, curious any test was failing to validate table props without
this change earlier?
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/TransformIT.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end.index;
+
+import org.apache.phoenix.thirdparty.com.google.common.base.Strings;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Properties;
+
+import static
org.apache.phoenix.end2end.index.SingleCellIndexIT.assertMetadata;
Review comment:
Now that `assertMetadata` is moved to BaseTest, we can remove this to
avoid confusion (because SingleCellIndexIT doesn't have assertMetadata anymore
but this import still doesn't result into compilation failure because
`SingleCellIndexIT.assertMetadata` = `BaseTest.assertMetadata`)
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
##
@@ -5220,12 +5260,8 @@ private boolean evaluateStmtProperties(MetaProperties
metaProperties, MetaProper
throws SQLException {
boolean changingPhoenixTableProperty = false;
-if (metaProperties.getImmutableRowsProp() != null) {
+if (metaProperties.getImmutableRowsProp() != null && table.getType()
!= INDEX) {
if (metaProperties.getImmutableRowsProp().booleanValue() !=
table.isImmutableRows()) {
-if (table.getImmutableStorageScheme() !=
ImmutableStorageScheme.ONE_CELL_PER_COLUMN) {
Review comment:
Got it, saw the ignored tests. Thanks
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
##
@@ -5384,7 +5435,7 @@ else if (metaProperties.getImmutableStorageSchemeProp()
!= table.getImmutableSto
return changingPhoenixTableProperty;
}
-private static class MetaProperties {
+public static class MetaProperties {
Review comment:
Unless I missed something specific, looks like this change is not
required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create SYSTEM.TRANSFORM table
> -
>
> Key: PHOENIX-6603
> URL: https://issues.apache.org/jira/browse/PHOENIX-6603
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Gokcen Iskender
>Priority: Major
>
> SYSTEM.TRANSFORM is a table for bookkeeping the transform process
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [phoenix] virajjasani commented on a change in pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
virajjasani commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r766535782
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/TableProperty.java
##
@@ -182,7 +182,7 @@ public Object getPTableValue(PTable table) {
},
-COLUMN_ENCODED_BYTES(PhoenixDatabaseMetaData.ENCODING_SCHEME,
COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY, false, false, false) {
+COLUMN_ENCODED_BYTES(PhoenixDatabaseMetaData.ENCODING_SCHEME,
COLUMN_FAMILY_NOT_ALLOWED_TABLE_PROPERTY, true, false, false) {
Review comment:
Nice one, curious any test was failing to validate table props without
this change earlier?
##
File path:
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/TransformIT.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end.index;
+
+import org.apache.phoenix.thirdparty.com.google.common.base.Strings;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Properties;
+
+import static
org.apache.phoenix.end2end.index.SingleCellIndexIT.assertMetadata;
Review comment:
Now that `assertMetadata` is moved to BaseTest, we can remove this to
avoid confusion (because SingleCellIndexIT doesn't have assertMetadata anymore
but this import still doesn't result into compilation failure because
`SingleCellIndexIT.assertMetadata` = `BaseTest.assertMetadata`)
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
##
@@ -5220,12 +5260,8 @@ private boolean evaluateStmtProperties(MetaProperties
metaProperties, MetaProper
throws SQLException {
boolean changingPhoenixTableProperty = false;
-if (metaProperties.getImmutableRowsProp() != null) {
+if (metaProperties.getImmutableRowsProp() != null && table.getType()
!= INDEX) {
if (metaProperties.getImmutableRowsProp().booleanValue() !=
table.isImmutableRows()) {
-if (table.getImmutableStorageScheme() !=
ImmutableStorageScheme.ONE_CELL_PER_COLUMN) {
Review comment:
Got it, saw the ignored tests. Thanks
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
##
@@ -5384,7 +5435,7 @@ else if (metaProperties.getImmutableStorageSchemeProp()
!= table.getImmutableSto
return changingPhoenixTableProperty;
}
-private static class MetaProperties {
+public static class MetaProperties {
Review comment:
Unless I missed something specific, looks like this change is not
required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (PHOENIX-5375) Building indexes from a snapshot image of a data table leads to incorrect timestamps
[ https://issues.apache.org/jira/browse/PHOENIX-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456974#comment-17456974 ] Istvan Toth commented on PHOENIX-5375: -- Testing to see whether this applies to the new indexing. I have added a new snapshot parameter to IndexBuildTimestampIT, and found that the combination of _localindex_, _async_, and _snapshot_ parameters causes the test to fail. > Building indexes from a snapshot image of a data table leads to incorrect > timestamps > - > > Key: PHOENIX-5375 > URL: https://issues.apache.org/jira/browse/PHOENIX-5375 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.0.0, 4.14.2 >Reporter: Kadir OZDEMIR >Priority: Major > > IndexTool after PHOENIX-5018 uses the UngroupedAggregateRegionObserver > coprocessor in order to generate index mutations with the right timestamps > except when indexes are build from snapshots. This is because aggregate > operations are not supported for snapshots yet. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6603) Create SYSTEM.TRANSFORM table
[
https://issues.apache.org/jira/browse/PHOENIX-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456961#comment-17456961
]
ASF GitHub Bot commented on PHOENIX-6603:
-
virajjasani commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r766478021
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/transform/SystemTransformRecord.java
##
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.schema.transform;
+
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Timestamp;
+
+/**
+ * Task params to be used while upserting records in SYSTEM.TRANSFORM table.
+ * This POJO is mainly used while upserting(and committing) or generating
+ * upsert mutations plan in {@link Transform} class
+ */
+@edu.umd.cs.findbugs.annotations.SuppressWarnings(
+value = {"EI_EXPOSE_REP", "EI_EXPOSE_REP2"},
+justification = "endTs and startTs are not used for mutation")
+public class SystemTransformRecord {
+private final PTable.TransformType transformType;
+private final String schemaName;
+private final String logicalTableName;
+private final String tenantId;
+private final String logicalParentName;
+private final String newPhysicalTableName;
+private final String transformStatus;
+private final String transformJobId;
+private final Integer transformRetryCount;
+private final Timestamp startTs;
+private final Timestamp endTs;
+private final String oldMetadata;
+private final String newMetadata;
+private final String transformFunction;
+
+public SystemTransformRecord(PTable.TransformType transformType,
+ String schemaName, String logicalTableName,
String tenantId, String newPhysicalTableName, String logicalParentName,
+ String transformStatus, String
transformJobId, Integer transformRetryCount, Timestamp startTs,
+ Timestamp endTs, String oldMetadata, String
newMetadata, String transformFunction) {
+this.transformType = transformType;
+this.schemaName = schemaName;
+this.tenantId = tenantId;
+this.logicalTableName = logicalTableName;
+this.newPhysicalTableName = newPhysicalTableName;
+this.logicalParentName = logicalParentName;
+this.transformStatus = transformStatus;
+this.transformJobId = transformJobId;
+this.transformRetryCount = transformRetryCount;
+this.startTs = startTs;
+this.endTs = endTs;
+this.oldMetadata = oldMetadata;
+this.newMetadata = newMetadata;
+this.transformFunction = transformFunction;
+}
+
+public String getString() {
+return String.format("transformType: %s, schameName: %s,
logicalTableName: %s, newPhysicalTableName: %s, logicalParentName: %s "
+, String.valueOf(transformType), String.valueOf(schemaName),
String.valueOf(logicalTableName), String.valueOf(newPhysicalTableName),
+String.valueOf(logicalParentName));
+}
+
+public PTable.TransformType getTransformType() {
+return transformType;
+}
+
+public String getSchemaName() {
+return schemaName;
+}
+
+public String getTenantId() {
+return tenantId;
+}
+
+public String getLogicalTableName() {
+return logicalTableName;
+}
+
+public String getLogicalParentName() {
+return logicalParentName;
+}
+
+public String getNewPhysicalTableName() {
+return newPhysicalTableName;
+}
+
+public String getTransformStatus() {
+return transformStatus;
+}
+
+public String getTransformJobId() {
+return transformJobId;
+}
+
+public int getTransformRetryCount() {
+return transformRetryCount;
+}
+
+public Timestamp getTransformStartTs() {
+return startTs;
+}
+
+public Timestamp getTransformEndTs() {
+return
[GitHub] [phoenix] virajjasani commented on a change in pull request #1363: PHOENIX-6603: Add SYSTEM.TRANSFORM table (#656)
virajjasani commented on a change in pull request #1363:
URL: https://github.com/apache/phoenix/pull/1363#discussion_r766478021
##
File path:
phoenix-core/src/main/java/org/apache/phoenix/schema/transform/SystemTransformRecord.java
##
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.schema.transform;
+
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Timestamp;
+
+/**
+ * Task params to be used while upserting records in SYSTEM.TRANSFORM table.
+ * This POJO is mainly used while upserting(and committing) or generating
+ * upsert mutations plan in {@link Transform} class
+ */
+@edu.umd.cs.findbugs.annotations.SuppressWarnings(
+value = {"EI_EXPOSE_REP", "EI_EXPOSE_REP2"},
+justification = "endTs and startTs are not used for mutation")
+public class SystemTransformRecord {
+private final PTable.TransformType transformType;
+private final String schemaName;
+private final String logicalTableName;
+private final String tenantId;
+private final String logicalParentName;
+private final String newPhysicalTableName;
+private final String transformStatus;
+private final String transformJobId;
+private final Integer transformRetryCount;
+private final Timestamp startTs;
+private final Timestamp endTs;
+private final String oldMetadata;
+private final String newMetadata;
+private final String transformFunction;
+
+public SystemTransformRecord(PTable.TransformType transformType,
+ String schemaName, String logicalTableName,
String tenantId, String newPhysicalTableName, String logicalParentName,
+ String transformStatus, String
transformJobId, Integer transformRetryCount, Timestamp startTs,
+ Timestamp endTs, String oldMetadata, String
newMetadata, String transformFunction) {
+this.transformType = transformType;
+this.schemaName = schemaName;
+this.tenantId = tenantId;
+this.logicalTableName = logicalTableName;
+this.newPhysicalTableName = newPhysicalTableName;
+this.logicalParentName = logicalParentName;
+this.transformStatus = transformStatus;
+this.transformJobId = transformJobId;
+this.transformRetryCount = transformRetryCount;
+this.startTs = startTs;
+this.endTs = endTs;
+this.oldMetadata = oldMetadata;
+this.newMetadata = newMetadata;
+this.transformFunction = transformFunction;
+}
+
+public String getString() {
+return String.format("transformType: %s, schameName: %s,
logicalTableName: %s, newPhysicalTableName: %s, logicalParentName: %s "
+, String.valueOf(transformType), String.valueOf(schemaName),
String.valueOf(logicalTableName), String.valueOf(newPhysicalTableName),
+String.valueOf(logicalParentName));
+}
+
+public PTable.TransformType getTransformType() {
+return transformType;
+}
+
+public String getSchemaName() {
+return schemaName;
+}
+
+public String getTenantId() {
+return tenantId;
+}
+
+public String getLogicalTableName() {
+return logicalTableName;
+}
+
+public String getLogicalParentName() {
+return logicalParentName;
+}
+
+public String getNewPhysicalTableName() {
+return newPhysicalTableName;
+}
+
+public String getTransformStatus() {
+return transformStatus;
+}
+
+public String getTransformJobId() {
+return transformJobId;
+}
+
+public int getTransformRetryCount() {
+return transformRetryCount;
+}
+
+public Timestamp getTransformStartTs() {
+return startTs;
+}
+
+public Timestamp getTransformEndTs() {
+return endTs;
+}
+
+public String getOldMetadata() {
+return oldMetadata;
+}
+public String getNewMetadata() {
+return newMetadata;
+}
+public String getTransformFunction() { return transformFunction; }
+
+
