[jira] [Commented] (PHOENIX-4130) Avoid server retries for mutable indexes
[ https://issues.apache.org/jira/browse/PHOENIX-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333103#comment-16333103 ] Vincent Poon commented on PHOENIX-4130: --- Discussed with James - l'll work on a v3 next week where we retry index failures at the Phoenix layer, so that we know if subsequent retries succeed, we can reset the index to ACTIVE > Avoid server retries for mutable indexes > > > Key: PHOENIX-4130 > URL: https://issues.apache.org/jira/browse/PHOENIX-4130 > Project: Phoenix > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vincent Poon >Priority: Major > Fix For: 4.14.0 > > Attachments: PHOENIX-4130.v1.master.patch, > PHOENIX-4130.v2.master.patch > > > Had some discussions with [~jamestaylor], [~samarthjain], and [~vincentpoon], > during which I suggested that we can possibly eliminate retry loops happening > at the server that cause the handler threads to be stuck potentially for > quite a while (at least multiple seconds to ride over common scenarios like > splits). > Instead we can do the retries at the Phoenix client that. > So: > # The index updates are not retried on the server. (retries = 0) > # A failed index update would set the failed index timestamp but leave the > index enabled. > # Now the handler thread is done, it throws an appropriate exception back to > the client. > # The Phoenix client can now retry. When those retries fail the index is > disabled (if the policy dictates that) and throw the exception back to its > caller. > So no more waiting is needed on the server, handler threads are freed > immediately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index
[ https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333101#comment-16333101 ] Vincent Poon commented on PHOENIX-4531: --- Attached a v5, where if we're comparing two index tables, we still consider the size of the tables in the comparison. This fixes a test failure in CostBasedDecisionIT Will commit on Monday unless any objections > Delete on a table with a global mutable index can issue client-side deletes > against the index > - > > Key: PHOENIX-4531 > URL: https://issues.apache.org/jira/browse/PHOENIX-4531 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.0 > Environment: >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Major > Attachments: PHOENIX-4531.v1.master.patch, > PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, > PHOENIX-4531.v5.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, > PartialIndexRebuilderIT.java > > > For a table with a global mutable index, I found the following result in > client-side deletes against both the data table and index table. > "DELETE FROM data_table" > "DELETE FROM data_table WHERE indexed_col='v'" > We only need the delete to be issued against the data table, because > 1) It's redundant since a delete against the index will be issued on the > server side when we process the delete of the data table row > 2) Deletes issued from the client-side won't have the index failure policy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index
[ https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated PHOENIX-4531: -- Attachment: PHOENIX-4531.v5.master.patch > Delete on a table with a global mutable index can issue client-side deletes > against the index > - > > Key: PHOENIX-4531 > URL: https://issues.apache.org/jira/browse/PHOENIX-4531 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.0 > Environment: >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Major > Attachments: PHOENIX-4531.v1.master.patch, > PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, > PHOENIX-4531.v5.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, > PartialIndexRebuilderIT.java > > > For a table with a global mutable index, I found the following result in > client-side deletes against both the data table and index table. > "DELETE FROM data_table" > "DELETE FROM data_table WHERE indexed_col='v'" > We only need the delete to be issued against the data table, because > 1) It's redundant since a delete against the index will be issued on the > server side when we process the delete of the data table row > 2) Deletes issued from the client-side won't have the index failure policy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4542) Update cryptographic sig file names per ASF policy
[
https://issues.apache.org/jira/browse/PHOENIX-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333086#comment-16333086
]
Hudson commented on PHOENIX-4542:
-
FAILURE: Integrated in Jenkins build Phoenix-master #1916 (See
[https://builds.apache.org/job/Phoenix-master/1916/])
PHOENIX-4542 Use .sha256 and .sha512 (elserj: rev
5fb3f7fa86bb9496f91f1f861f000c0089157340)
* (edit) dev/make_rc.sh
> Update cryptographic sig file names per ASF policy
> --
>
> Key: PHOENIX-4542
> URL: https://issues.apache.org/jira/browse/PHOENIX-4542
> Project: Phoenix
> Issue Type: Task
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4542.001.patch
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
> {quote}
> An [SHA|https://www.apache.org/dev/release-signing#sha-checksum] checksum
> SHOULD also be created and MUST be suffixed as:
> * {{.sha1}} for a SHA-1 checksum
> * {{.sha256}} for a SHA-256 checksum
> * {{.sha512}} for a SHA-512 checksum
> {quote}
> We should be using .sha256 and .sha512 instead of .sha as the filename's
> suffix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4541) rat-check fails on argparse.py
[ https://issues.apache.org/jira/browse/PHOENIX-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333085#comment-16333085 ] Hudson commented on PHOENIX-4541: - FAILURE: Integrated in Jenkins build Phoenix-master #1916 (See [https://builds.apache.org/job/Phoenix-master/1916/]) PHOENIX-4541 Fix apache-rat-check failures (elserj: rev 0aa1cfb4e14f52c9770d5da5bae7f4868152ca7e) * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/TablesNotInSyncException.java * (edit) pom.xml > rat-check fails on argparse.py > -- > > Key: PHOENIX-4541 > URL: https://issues.apache.org/jira/browse/PHOENIX-4541 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 5.0.0, 4.14.0 > > Attachments: PHOENIX-4541.001.patch > > > PHOENIX-4449 bundles a version of argparse.py into the Phoenix installation > to work around some system Python compatibility issues. This is not ALv2 but > is compatible. Need to add an exclusion to the plugin's configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4130) Avoid server retries for mutable indexes
[ https://issues.apache.org/jira/browse/PHOENIX-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332971#comment-16332971 ] Vincent Poon commented on PHOENIX-4130: --- [~jamestaylor] Attached v2 patch, please review. Implemented the following based on your suggestions: * add PENDING_DISABLE state. If it's been in this state past a certain threshold, the client no longer uses it, and the rebuilder will change the state to DISABLE * add failure handling to UngroupedAggregateRegionObserver.commit() , for mutations that don't go through the client's MutationState * pass back the QueryServices.INDEX_FAILURE_DISABLE_INDEX config setting via the exception that is thrown back to the client, so that it only needs to be set on the server side * use IndexUtil.updateIndexState instead of "alter index" statement I don't think there's a way to detect # of retries on a successful mutation, so currently if we transition to PENDING_DISABLE on the first attempt, then the subsequent attempt succeeds, we rely on the rebuilder to transition back to ACTIVE (and duplicating mutations in the process). Would be good if you can double check that the PENDING_DISABLE checks are in the right places - I looked through the usages of PENDING_ACTIVE and modified the ones that looked appropriate. > Avoid server retries for mutable indexes > > > Key: PHOENIX-4130 > URL: https://issues.apache.org/jira/browse/PHOENIX-4130 > Project: Phoenix > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vincent Poon >Priority: Major > Fix For: 4.14.0 > > Attachments: PHOENIX-4130.v1.master.patch, > PHOENIX-4130.v2.master.patch > > > Had some discussions with [~jamestaylor], [~samarthjain], and [~vincentpoon], > during which I suggested that we can possibly eliminate retry loops happening > at the server that cause the handler threads to be stuck potentially for > quite a while (at least multiple seconds to ride over common scenarios like > splits). > Instead we can do the retries at the Phoenix client that. > So: > # The index updates are not retried on the server. (retries = 0) > # A failed index update would set the failed index timestamp but leave the > index enabled. > # Now the handler thread is done, it throws an appropriate exception back to > the client. > # The Phoenix client can now retry. When those retries fail the index is > disabled (if the policy dictates that) and throw the exception back to its > caller. > So no more waiting is needed on the server, handler threads are freed > immediately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4130) Avoid server retries for mutable indexes
[ https://issues.apache.org/jira/browse/PHOENIX-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated PHOENIX-4130: -- Attachment: PHOENIX-4130.v2.master.patch > Avoid server retries for mutable indexes > > > Key: PHOENIX-4130 > URL: https://issues.apache.org/jira/browse/PHOENIX-4130 > Project: Phoenix > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vincent Poon >Priority: Major > Fix For: 4.14.0 > > Attachments: PHOENIX-4130.v1.master.patch, > PHOENIX-4130.v2.master.patch > > > Had some discussions with [~jamestaylor], [~samarthjain], and [~vincentpoon], > during which I suggested that we can possibly eliminate retry loops happening > at the server that cause the handler threads to be stuck potentially for > quite a while (at least multiple seconds to ride over common scenarios like > splits). > Instead we can do the retries at the Phoenix client that. > So: > # The index updates are not retried on the server. (retries = 0) > # A failed index update would set the failed index timestamp but leave the > index enabled. > # Now the handler thread is done, it throws an appropriate exception back to > the client. > # The Phoenix client can now retry. When those retries fail the index is > disabled (if the policy dictates that) and throw the exception back to its > caller. > So no more waiting is needed on the server, handler threads are freed > immediately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4544) Update statistics inconsistent behavior
[
https://issues.apache.org/jira/browse/PHOENIX-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov updated PHOENIX-4544:
-
Affects Version/s: (was: 5.x)
5.0
> Update statistics inconsistent behavior
>
>
> Key: PHOENIX-4544
> URL: https://issues.apache.org/jira/browse/PHOENIX-4544
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Romil Choksi
>Priority: Major
>
> Update statistics may not generate the stats information for all dependent
> indexes. And this behavior may depend on whether the command executed
> synchronously or asynchronously.
> I have a table GIGANTIC_TABLE with ~500k rows with global index I1 and local
> index I2.
> If async is turned on (the default value):
> {noformat}
> 0: jdbc:phoenix:> update statistics GIGANTIC_TABLE ALL;
> No rows affected (0.081 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='I1' AND COLUMN_FAMILY='0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 5 |
> +---+
> 1 row selected (0.009 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 520 |
> +---+
> 1 row selected (0.014 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='L#0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 0 |
> +---+
> 1 row selected (0.008 seconds)
> 0: jdbc:phoenix:>
> {noformat}
> As we can see there is no records for local index I2. But if we run
> statistics for indexes:
> {noformat}
> 0: jdbc:phoenix:> update statistics GIGANTIC_TABLE INDEX;
> No rows affected (0.036 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='L#0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 20|
> +---+
> 1 row selected (0.007 seconds)
> {noformat}
> the statistic for local index is generated correctly.
> Now we turn async off:
> {noformat}
> 0: jdbc:phoenix:> delete from SYSTEM.STATS;
> 547 rows affected (0.079 seconds)
> 0: jdbc:phoenix:> update statistics GIGANTIC_TABLE ALL;
> 999,998 rows affected (4.671 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 520 |
> +---+
> 1 row selected (0.04 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='L#0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 20|
> +---+
> 1 row selected (0.012 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='I1' AND COLUMN_FAMILY='0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 0 |
> +---+
> 1 row selected (0.011 seconds)
> {noformat}
> As we can see we got statistics for the table itself and local index. But not
> for the global index.
> Moreover, if we try to update statistics for indexes:
> {noformat}
> 0: jdbc:phoenix:> update statistics GIGANTIC_TABLE INDEX;
> 499,999 rows affected (0.332 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='I1' AND COLUMN_FAMILY='0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 0 |
> +---+
> 1 row selected (0.009 seconds)
> {noformat}
> So, still no records for global index.
> But if we delete statistics first and run update for indexes:
> {noformat}
> 0: jdbc:phoenix:> delete from SYSTEM.STATS;
> 541 rows affected (0.024 seconds)
> 0: jdbc:phoenix:> update statistics GIGANTIC_TABLE INDEX;
> 999,998 rows affected (0.41 seconds)
> 0: jdbc:phoenix:> select
[jira] [Commented] (PHOENIX-4544) Update statistics inconsistent behavior
[
https://issues.apache.org/jira/browse/PHOENIX-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332914#comment-16332914
]
Sergey Soldatov commented on PHOENIX-4544:
--
Checked with 5.x branch, but it also may affect master as well.
> Update statistics inconsistent behavior
>
>
> Key: PHOENIX-4544
> URL: https://issues.apache.org/jira/browse/PHOENIX-4544
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Romil Choksi
>Priority: Major
>
> Update statistics may not generate the stats information for all dependent
> indexes. And this behavior may depend on whether the command executed
> synchronously or asynchronously.
> I have a table GIGANTIC_TABLE with ~500k rows with global index I1 and local
> index I2.
> If async is turned on (the default value):
> {noformat}
> 0: jdbc:phoenix:> update statistics GIGANTIC_TABLE ALL;
> No rows affected (0.081 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='I1' AND COLUMN_FAMILY='0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 5 |
> +---+
> 1 row selected (0.009 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 520 |
> +---+
> 1 row selected (0.014 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='L#0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 0 |
> +---+
> 1 row selected (0.008 seconds)
> 0: jdbc:phoenix:>
> {noformat}
> As we can see there is no records for local index I2. But if we run
> statistics for indexes:
> {noformat}
> 0: jdbc:phoenix:> update statistics GIGANTIC_TABLE INDEX;
> No rows affected (0.036 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='L#0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 20|
> +---+
> 1 row selected (0.007 seconds)
> {noformat}
> the statistic for local index is generated correctly.
> Now we turn async off:
> {noformat}
> 0: jdbc:phoenix:> delete from SYSTEM.STATS;
> 547 rows affected (0.079 seconds)
> 0: jdbc:phoenix:> update statistics GIGANTIC_TABLE ALL;
> 999,998 rows affected (4.671 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 520 |
> +---+
> 1 row selected (0.04 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='L#0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 20|
> +---+
> 1 row selected (0.012 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='I1' AND COLUMN_FAMILY='0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 0 |
> +---+
> 1 row selected (0.011 seconds)
> {noformat}
> As we can see we got statistics for the table itself and local index. But not
> for the global index.
> Moreover, if we try to update statistics for indexes:
> {noformat}
> 0: jdbc:phoenix:> update statistics GIGANTIC_TABLE INDEX;
> 499,999 rows affected (0.332 seconds)
> 0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
> PHYSICAL_NAME='I1' AND COLUMN_FAMILY='0';
> +---+
> | COUNT(GUIDE_POSTS_ROW_COUNT) |
> +---+
> | 0 |
> +---+
> 1 row selected (0.009 seconds)
> {noformat}
> So, still no records for global index.
> But if we delete statistics first and run update for indexes:
> {noformat}
> 0: jdbc:phoenix:> delete from SYSTEM.STATS;
> 541 rows affected (0.024 seconds)
> 0: jdbc:phoenix:> update statistics GIGANTIC_TABLE INDEX;
> 999,998 rows affected (0.41 seconds)
> 0:
[jira] [Created] (PHOENIX-4544) Update statistics inconsistent behavior
Sergey Soldatov created PHOENIX-4544:
Summary: Update statistics inconsistent behavior
Key: PHOENIX-4544
URL: https://issues.apache.org/jira/browse/PHOENIX-4544
Project: Phoenix
Issue Type: Bug
Affects Versions: 5.x
Reporter: Romil Choksi
Update statistics may not generate the stats information for all dependent
indexes. And this behavior may depend on whether the command executed
synchronously or asynchronously.
I have a table GIGANTIC_TABLE with ~500k rows with global index I1 and local
index I2.
If async is turned on (the default value):
{noformat}
0: jdbc:phoenix:> update statistics GIGANTIC_TABLE ALL;
No rows affected (0.081 seconds)
0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
PHYSICAL_NAME='I1' AND COLUMN_FAMILY='0';
+---+
| COUNT(GUIDE_POSTS_ROW_COUNT) |
+---+
| 5 |
+---+
1 row selected (0.009 seconds)
0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='0';
+---+
| COUNT(GUIDE_POSTS_ROW_COUNT) |
+---+
| 520 |
+---+
1 row selected (0.014 seconds)
0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='L#0';
+---+
| COUNT(GUIDE_POSTS_ROW_COUNT) |
+---+
| 0 |
+---+
1 row selected (0.008 seconds)
0: jdbc:phoenix:>
{noformat}
As we can see there is no records for local index I2. But if we run statistics
for indexes:
{noformat}
0: jdbc:phoenix:> update statistics GIGANTIC_TABLE INDEX;
No rows affected (0.036 seconds)
0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='L#0';
+---+
| COUNT(GUIDE_POSTS_ROW_COUNT) |
+---+
| 20|
+---+
1 row selected (0.007 seconds)
{noformat}
the statistic for local index is generated correctly.
Now we turn async off:
{noformat}
0: jdbc:phoenix:> delete from SYSTEM.STATS;
547 rows affected (0.079 seconds)
0: jdbc:phoenix:> update statistics GIGANTIC_TABLE ALL;
999,998 rows affected (4.671 seconds)
0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='0';
+---+
| COUNT(GUIDE_POSTS_ROW_COUNT) |
+---+
| 520 |
+---+
1 row selected (0.04 seconds)
0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='L#0';
+---+
| COUNT(GUIDE_POSTS_ROW_COUNT) |
+---+
| 20|
+---+
1 row selected (0.012 seconds)
0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
PHYSICAL_NAME='I1' AND COLUMN_FAMILY='0';
+---+
| COUNT(GUIDE_POSTS_ROW_COUNT) |
+---+
| 0 |
+---+
1 row selected (0.011 seconds)
{noformat}
As we can see we got statistics for the table itself and local index. But not
for the global index.
Moreover, if we try to update statistics for indexes:
{noformat}
0: jdbc:phoenix:> update statistics GIGANTIC_TABLE INDEX;
499,999 rows affected (0.332 seconds)
0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
PHYSICAL_NAME='I1' AND COLUMN_FAMILY='0';
+---+
| COUNT(GUIDE_POSTS_ROW_COUNT) |
+---+
| 0 |
+---+
1 row selected (0.009 seconds)
{noformat}
So, still no records for global index.
But if we delete statistics first and run update for indexes:
{noformat}
0: jdbc:phoenix:> delete from SYSTEM.STATS;
541 rows affected (0.024 seconds)
0: jdbc:phoenix:> update statistics GIGANTIC_TABLE INDEX;
999,998 rows affected (0.41 seconds)
0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
PHYSICAL_NAME='I1' AND COLUMN_FAMILY='0';
+---+
| COUNT(GUIDE_POSTS_ROW_COUNT) |
+---+
| 5 |
+---+
1 row selected (0.01 seconds)
0: jdbc:phoenix:> select count(GUIDE_POSTS_ROW_COUNT) from SYSTEM.STATS WHERE
PHYSICAL_NAME='GIGANTIC_TABLE' AND COLUMN_FAMILY='L#0';
+---+
|
[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system
[
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332891#comment-16332891
]
Josh Elser commented on PHOENIX-4537:
-
bq. Are we saying that migration would happen before the first client
connection? That's bad. Can we change that?
There are two points here: one is migration (modification of the schema and
contents of our system tables). This happens on first client connection.
The issue here is that "renaming" of system tables from "SYSTEM.CATALOG" to the
namespace-mapped variants ("SYSTEM:CATALOG") is not presently subject to the
same logic. It presently happens for any Phoenix connection, client or server.
I think we just haven't noticed this before. The only time it would really
cause an issue would be this deadlock where a compaction gets queued which
keeps the table from becoming disabled (until that RS is taken down)
> RegionServer initiating compaction can trigger schema migration and deadlock
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
> Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}}
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a
> Phoenix connection. While the RegionServer won't upgrade system tables, it
> *will* try to migrate them into the schema mapped variants (e.g.
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled
> until the region is CLOSED, and the region cannot be CLOSED until the
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table
> migrations, but Sergey and [~elserj] both think that this will end badly
> (RegionServers falling over because they expect the tables to be migrated and
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4542) Update cryptographic sig file names per ASF policy
[
https://issues.apache.org/jira/browse/PHOENIX-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332887#comment-16332887
]
Sergey Soldatov commented on PHOENIX-4542:
--
+1
> Update cryptographic sig file names per ASF policy
> --
>
> Key: PHOENIX-4542
> URL: https://issues.apache.org/jira/browse/PHOENIX-4542
> Project: Phoenix
> Issue Type: Task
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4542.001.patch
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
> {quote}
> An [SHA|https://www.apache.org/dev/release-signing#sha-checksum] checksum
> SHOULD also be created and MUST be suffixed as:
> * {{.sha1}} for a SHA-1 checksum
> * {{.sha256}} for a SHA-256 checksum
> * {{.sha512}} for a SHA-512 checksum
> {quote}
> We should be using .sha256 and .sha512 instead of .sha as the filename's
> suffix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4541) rat-check fails on argparse.py
[ https://issues.apache.org/jira/browse/PHOENIX-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332885#comment-16332885 ] Sergey Soldatov commented on PHOENIX-4541: -- LGTM +1. > rat-check fails on argparse.py > -- > > Key: PHOENIX-4541 > URL: https://issues.apache.org/jira/browse/PHOENIX-4541 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 5.0.0, 4.14.0 > > Attachments: PHOENIX-4541.001.patch > > > PHOENIX-4449 bundles a version of argparse.py into the Phoenix installation > to work around some system Python compatibility issues. This is not ALv2 but > is compatible. Need to add an exclusion to the plugin's configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system
[
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332884#comment-16332884
]
James Taylor commented on PHOENIX-4537:
---
Are we saying that migration would happen before the first client connection?
That's bad. Can we change that? We send the client version along in
MetaDataEndpointImpl calls. Perhaps we could send it through our region
observers if need be.
My preference would be to implement PHOENIX-4530 and get rid of
clearTsOnDisabledIndexes as it wouldn't be necessary ([~an...@apache.org]
excellent idea). We might need to start adding secondary indexes to system
tables, so I'd hate to have too many assumptions here.
> RegionServer initiating compaction can trigger schema migration and deadlock
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
> Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}}
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a
> Phoenix connection. While the RegionServer won't upgrade system tables, it
> *will* try to migrate them into the schema mapped variants (e.g.
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled
> until the region is CLOSED, and the region cannot be CLOSED until the
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table
> migrations, but Sergey and [~elserj] both think that this will end badly
> (RegionServers falling over because they expect the tables to be migrated and
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system
[
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332876#comment-16332876
]
Josh Elser commented on PHOENIX-4537:
-
bq. LGTM. The only concern I have is that in our documentation, we are saying
that the migration would happen on the first connection
Yeah, agreed.
[~jamestaylor], don't want to misinterpret the silence, but how does this
approach strike you?
> RegionServer initiating compaction can trigger schema migration and deadlock
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
> Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}}
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a
> Phoenix connection. While the RegionServer won't upgrade system tables, it
> *will* try to migrate them into the schema mapped variants (e.g.
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled
> until the region is CLOSED, and the region cannot be CLOSED until the
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table
> migrations, but Sergey and [~elserj] both think that this will end badly
> (RegionServers falling over because they expect the tables to be migrated and
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4534) upsert/delete/upsert for the same row corrupts the indexes
[
https://issues.apache.org/jira/browse/PHOENIX-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332868#comment-16332868
]
Rajeshbabu Chintaguntla commented on PHOENIX-4534:
--
[~sergey.soldatov] Here are the test failures in master branch which are
failing without the patch also. Will do patch cleanup and upload new one.
{noformat}
[http://issues.apache.org/jira/secure/attachment/12906811/PHOENIX-4534_v2.patch]
failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.LocalIndexSplitMergeIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.OrderByIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.RowValueConstructorIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.IndexMetadataIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.DefaultColumnValueIT
{noformat}
> upsert/delete/upsert for the same row corrupts the indexes
> --
>
> Key: PHOENIX-4534
> URL: https://issues.apache.org/jira/browse/PHOENIX-4534
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Romil Choksi
>Assignee: Rajeshbabu Chintaguntla
>Priority: Critical
> Labels: HBase-2.0
> Fix For: 5.0
>
> Attachments: PHOENIX-4534.patch, PHOENIX-4534_v2.patch
>
>
> If we delete and upsert again the same row, the corresponding index has a
> null value.
> {noformat}
> 0: jdbc:phoenix:> create table a (id integer primary key, f float);
> No rows affected (2.272 seconds)
> 0: jdbc:phoenix:> create index i1 on a (f);
> No rows affected (5.769 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.021 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F | :ID |
> +--+--+
> | 0.5 | 1|
> +--+--+
> 1 row selected (0.016 seconds)
> 0: jdbc:phoenix:> delete from a where id = 1;
> 1 row affected (0.009 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F | :ID |
> +--+--+
> +--+--+
> No rows selected (0.015 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.008 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +---+--+
> | 0:F | :ID |
> +---+--+
> | null | 1|
> +---+--+
> 1 row selected (0.013 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system
[
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332864#comment-16332864
]
Sergey Soldatov commented on PHOENIX-4537:
--
LGTM. The only concern I have is that in our documentation, we are saying that
the migration would happen on the first connection. That might surprise the end
user that it actually happens without his involvement.
> RegionServer initiating compaction can trigger schema migration and deadlock
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
> Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}}
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a
> Phoenix connection. While the RegionServer won't upgrade system tables, it
> *will* try to migrate them into the schema mapped variants (e.g.
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled
> until the region is CLOSED, and the region cannot be CLOSED until the
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table
> migrations, but Sergey and [~elserj] both think that this will end badly
> (RegionServers falling over because they expect the tables to be migrated and
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index
[ https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332856#comment-16332856 ] James Taylor commented on PHOENIX-4531: --- +1. Thanks, [~vincentpoon]. > Delete on a table with a global mutable index can issue client-side deletes > against the index > - > > Key: PHOENIX-4531 > URL: https://issues.apache.org/jira/browse/PHOENIX-4531 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.0 > Environment: >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Major > Attachments: PHOENIX-4531.v1.master.patch, > PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, > PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, PartialIndexRebuilderIT.java > > > For a table with a global mutable index, I found the following result in > client-side deletes against both the data table and index table. > "DELETE FROM data_table" > "DELETE FROM data_table WHERE indexed_col='v'" > We only need the delete to be issued against the data table, because > 1) It's redundant since a delete against the index will be issued on the > server side when we process the delete of the data table row > 2) Deletes issued from the client-side won't have the index failure policy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index
[ https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated PHOENIX-4531: -- Attachment: PHOENIX-4531.v4.master.patch > Delete on a table with a global mutable index can issue client-side deletes > against the index > - > > Key: PHOENIX-4531 > URL: https://issues.apache.org/jira/browse/PHOENIX-4531 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.0 > Environment: >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Major > Attachments: PHOENIX-4531.v1.master.patch, > PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, > PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, PartialIndexRebuilderIT.java > > > For a table with a global mutable index, I found the following result in > client-side deletes against both the data table and index table. > "DELETE FROM data_table" > "DELETE FROM data_table WHERE indexed_col='v'" > We only need the delete to be issued against the data table, because > 1) It's redundant since a delete against the index will be issued on the > server side when we process the delete of the data table row > 2) Deletes issued from the client-side won't have the index failure policy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system
[
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated PHOENIX-4537:
Attachment: PHOENIX-4537.001.patch
> RegionServer initiating compaction can trigger schema migration and deadlock
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
> Issue Type: Bug
>Reporter: Romil Choksi
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}}
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a
> Phoenix connection. While the RegionServer won't upgrade system tables, it
> *will* try to migrate them into the schema mapped variants (e.g.
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled
> until the region is CLOSED, and the region cannot be CLOSED until the
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table
> migrations, but Sergey and [~elserj] both think that this will end badly
> (RegionServers falling over because they expect the tables to be migrated and
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index
[ https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332841#comment-16332841 ] Vincent Poon commented on PHOENIX-4531: --- v4 patch with a test that a DELETE statement without WHERE clause favors the data table. Including a WHERE clause favors the index. > Delete on a table with a global mutable index can issue client-side deletes > against the index > - > > Key: PHOENIX-4531 > URL: https://issues.apache.org/jira/browse/PHOENIX-4531 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.0 > Environment: >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Major > Attachments: PHOENIX-4531.v1.master.patch, > PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, > PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, PartialIndexRebuilderIT.java > > > For a table with a global mutable index, I found the following result in > client-side deletes against both the data table and index table. > "DELETE FROM data_table" > "DELETE FROM data_table WHERE indexed_col='v'" > We only need the delete to be issued against the data table, because > 1) It's redundant since a delete against the index will be issued on the > server side when we process the delete of the data table row > 2) Deletes issued from the client-side won't have the index failure policy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system
[
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser reassigned PHOENIX-4537:
---
Assignee: Josh Elser
> RegionServer initiating compaction can trigger schema migration and deadlock
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
> Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}}
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a
> Phoenix connection. While the RegionServer won't upgrade system tables, it
> *will* try to migrate them into the schema mapped variants (e.g.
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled
> until the region is CLOSED, and the region cannot be CLOSED until the
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table
> migrations, but Sergey and [~elserj] both think that this will end badly
> (RegionServers falling over because they expect the tables to be migrated and
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (PHOENIX-4519) Index rebuild MR jobs not created for "alter index rebuild async" rebuilds
[ https://issues.apache.org/jira/browse/PHOENIX-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332673#comment-16332673 ] James Taylor edited comment on PHOENIX-4519 at 1/19/18 6:17 PM: Thanks for the info, [~an...@apache.org]. Couple of more specific questions: * Where are the unit tests for IndexToolForPartialBuildIT? * Do they test all the corner cases as is done by PartialIndexRebuilderIT? ** Multiple versions of same row ** Multiple versions of same row with family deletes intermixed ** Null values of columns ** Index write failure while executing raw scan while partially rebuilding (with multiple batches) ** Data table taking writes to same rows being partially rebuilt (see testUpperBoundSetOnRebuild) ** Disable or rebuild of index during partial rebuild I filed PHOENIX-4543 for the MR partial index rebuilder to handle the case in which the index is left active while the partial rebuild is happening. Some use cases would rather tolerate some drift between the index and data table than take the read hit of having a disabled index. Since it's use case dependent, we allow this to be set on a per table basis. This is based on the DISABLE_INDEX_ON_WRITE_FAILURE property on the htable (true means it's disabled, false means it's left active) and REBUILD_INDEX_ON_WRITE_FAILURE (true means to partially rebuild the index while false means not to). was (Author: jamestaylor): Thanks for the info, [~an...@apache.org]. Couple of more specific questions: * Where are the unit tests for IndexToolForPartialBuildIT? * Do they test all the corner cases as is done by PartialIndexRebuilderIT? ** Multiple versions of same row ** Multiple versions of same row with family deletes intermixed ** Null values of columns ** Index write failure while executing raw scan while partially rebuilding (with multiple batches) ** Data table taking writes to same rows being partially rebuilt (see testUpperBoundSetOnRebuild) ** Disable or rebuild of index during partial rebuild I'll file a JIRA for handling the case in which the index is left active while the partial rebuild is happening. Some use cases would rather tolerate some drift between the index and data table than take the read hit of having a disabled index. Since it's use case dependent, we allow this to be set on a per table basis. This is based on the DISABLE_INDEX_ON_WRITE_FAILURE property on the htable (true means it's disabled, false means it's left active) and REBUILD_INDEX_ON_WRITE_FAILURE (true means to partially rebuild the index while false means not to). > Index rebuild MR jobs not created for "alter index rebuild async" rebuilds > -- > > Key: PHOENIX-4519 > URL: https://issues.apache.org/jira/browse/PHOENIX-4519 > Project: Phoenix > Issue Type: Bug >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Major > > It seems we have two ASYNC flags for index rebuilds: > ASYNC_CREATED_DATE - when an index is created async > ASYNC_REBUILD_TIMESTAMP - created by "alter index ... rebuild async" > The PhoenixMRJobSubmitter only submits MR jobs for the former. We should > also submit jobs for the latter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (PHOENIX-4543) Ensure that MR partial index rebuilder handles the case when an index is left active during the rebuild
James Taylor created PHOENIX-4543:
-
Summary: Ensure that MR partial index rebuilder handles the case
when an index is left active during the rebuild
Key: PHOENIX-4543
URL: https://issues.apache.org/jira/browse/PHOENIX-4543
Project: Phoenix
Issue Type: Bug
Reporter: James Taylor
We have an option used by the online partial index rebuilder
(MetaDataRegionObserver,BuildIndexScheduleTask) to keep an index active when a
write failure occurs, based on the htable property of
DISABLE_INDEX_ON_WRITE_FAILURE and falling back to the
{{QueryServices.INDEX_FAILURE_DISABLE_INDEX}} config property. Also, there's an
orthogonal option that controls if the partial index is done at all:
{{QueryServices.INDEX_FAILURE_HANDLING_REBUILD_ATTRIB}} turns this off globally
and the htable property {{REBUILD_INDEX_ON_WRITE_FAILURE}} turns it off on a
per table basis.
We should support the same options in the MR partial index rebuilder
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4519) Index rebuild MR jobs not created for "alter index rebuild async" rebuilds
[ https://issues.apache.org/jira/browse/PHOENIX-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332673#comment-16332673 ] James Taylor commented on PHOENIX-4519: --- Thanks for the info, [~an...@apache.org]. Couple of more specific questions: * Where are the unit tests for IndexToolForPartialBuildIT? * Do they test all the corner cases as is done by PartialIndexRebuilderIT? ** Multiple versions of same row ** Multiple versions of same row with family deletes intermixed ** Null values of columns ** Index write failure while executing raw scan while partially rebuilding (with multiple batches) ** Data table taking writes to same rows being partially rebuilt (see testUpperBoundSetOnRebuild) ** Disable or rebuild of index during partial rebuild I'll file a JIRA for handling the case in which the index is left active while the partial rebuild is happening. Some use cases would rather tolerate some drift between the index and data table than take the read hit of having a disabled index. Since it's use case dependent, we allow this to be set on a per table basis. This is based on the DISABLE_INDEX_ON_WRITE_FAILURE property on the htable (true means it's disabled, false means it's left active) and REBUILD_INDEX_ON_WRITE_FAILURE (true means to partially rebuild the index while false means not to). > Index rebuild MR jobs not created for "alter index rebuild async" rebuilds > -- > > Key: PHOENIX-4519 > URL: https://issues.apache.org/jira/browse/PHOENIX-4519 > Project: Phoenix > Issue Type: Bug >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Major > > It seems we have two ASYNC flags for index rebuilds: > ASYNC_CREATED_DATE - when an index is created async > ASYNC_REBUILD_TIMESTAMP - created by "alter index ... rebuild async" > The PhoenixMRJobSubmitter only submits MR jobs for the former. We should > also submit jobs for the latter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4540) Client side evaluation of group by Expression in projection gives erroneous result
[
https://issues.apache.org/jira/browse/PHOENIX-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332623#comment-16332623
]
James Taylor commented on PHOENIX-4540:
---
select round(k/v,0) x from round_test group by x,v
bq. Why we need to re-evaluate the expression here, can't we use the same
result evaluated at server side during the "group by"
We don't re-evaluate the expression on the client side. We get the value of
{{round(k/v,0)}} from the row key of the rows we get back from the server. The
server returns the aggregated rows with a row key based on the GROUP BY
expressions, so the first part of the returned row key should have the
evaluated value of {{round(k/v,0)}}.
Does this slightly simpler test fail too? It's a bit strange to group by both
round(k/v,0) and v.
select round(k/v,0) x from round_test group by x
I suspect it may be due to a type mismatch.
> Client side evaluation of group by Expression in projection gives erroneous
> result
> --
>
> Key: PHOENIX-4540
> URL: https://issues.apache.org/jira/browse/PHOENIX-4540
> Project: Phoenix
> Issue Type: Bug
>Reporter: Ankit Singhal
>Priority: Major
> Attachments: PHOENIX-4540_unittest.patch
>
>
> If the columns involved in projected expression are not present in "group by"
> clause, the client evaluation of the same expression will give an erroneous
> result because of the absence of involved column value.
> Following queries will produce wrong result
> >select round(k/v,0) x from round_test group by x,v
> >select k/v x from round_test group by x,v
> but query runs fine if we add all columns so that client expression can be
> evaluated
> >select round(k/v,0) x from round_test group by x,k,v //will produce right
> >result
> >select k/v x from round_test group by x,k,v;
> Why we need to re-evaluate the expression here, can't we use the same result
> evaluated at server side during the "group by"
> thoughts [~jamestaylor]?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4538) Phoenix/Spark unit tests fail to compile
[ https://issues.apache.org/jira/browse/PHOENIX-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332587#comment-16332587 ] James Taylor commented on PHOENIX-4538: --- Never mind, [~elserj]. I must have forgotten to do a mvn clean after switching branches. Works for me now. Sorry for the noise. > Phoenix/Spark unit tests fail to compile > > > Key: PHOENIX-4538 > URL: https://issues.apache.org/jira/browse/PHOENIX-4538 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Priority: Major > > The phoenix-spark unit test fail to compile with these two commits for > PHOENIX-4466: > - > https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594 > - > https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d > If I revert those, the test compile and pass (FYI, I'm on a Mac, so not sure > if this a contributing factor). This may be the reason our pre-commit Jenkins > is busted too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (PHOENIX-4538) Phoenix/Spark unit tests fail to compile
[ https://issues.apache.org/jira/browse/PHOENIX-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor resolved PHOENIX-4538. --- Resolution: Invalid > Phoenix/Spark unit tests fail to compile > > > Key: PHOENIX-4538 > URL: https://issues.apache.org/jira/browse/PHOENIX-4538 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Priority: Major > > The phoenix-spark unit test fail to compile with these two commits for > PHOENIX-4466: > - > https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594 > - > https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d > If I revert those, the test compile and pass (FYI, I'm on a Mac, so not sure > if this a contributing factor). This may be the reason our pre-commit Jenkins > is busted too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4541) rat-check fails on argparse.py
[ https://issues.apache.org/jira/browse/PHOENIX-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332583#comment-16332583 ] Josh Elser commented on PHOENIX-4541: - bq. Thanks Josh Elser for fixing this. Can we add this license check and rat-check in Hadoop QA? It's already there :) That's how I noticed it. https://builds.apache.org/job/PreCommit-PHOENIX-Build/1717/artifact/patchprocess/patchReleaseAuditProblems.txt However, because it got stuck on the argparse.py goof from me, it didn't also report on the missing license on this java file :) > rat-check fails on argparse.py > -- > > Key: PHOENIX-4541 > URL: https://issues.apache.org/jira/browse/PHOENIX-4541 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 5.0.0, 4.14.0 > > Attachments: PHOENIX-4541.001.patch > > > PHOENIX-4449 bundles a version of argparse.py into the Phoenix installation > to work around some system Python compatibility issues. This is not ALv2 but > is compatible. Need to add an exclusion to the plugin's configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4541) rat-check fails on argparse.py
[ https://issues.apache.org/jira/browse/PHOENIX-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332575#comment-16332575 ] Karan Mehta commented on PHOENIX-4541: -- Thanks [~elserj] for fixing this. Can we add this license check and rat-check in Hadoop QA? > rat-check fails on argparse.py > -- > > Key: PHOENIX-4541 > URL: https://issues.apache.org/jira/browse/PHOENIX-4541 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 5.0.0, 4.14.0 > > Attachments: PHOENIX-4541.001.patch > > > PHOENIX-4449 bundles a version of argparse.py into the Phoenix installation > to work around some system Python compatibility issues. This is not ALv2 but > is compatible. Need to add an exclusion to the plugin's configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4541) rat-check fails on argparse.py
[ https://issues.apache.org/jira/browse/PHOENIX-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-4541: -- Fix Version/s: (was: 4.13.0) 4.14.0 > rat-check fails on argparse.py > -- > > Key: PHOENIX-4541 > URL: https://issues.apache.org/jira/browse/PHOENIX-4541 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 5.0.0, 4.14.0 > > Attachments: PHOENIX-4541.001.patch > > > PHOENIX-4449 bundles a version of argparse.py into the Phoenix installation > to work around some system Python compatibility issues. This is not ALv2 but > is compatible. Need to add an exclusion to the plugin's configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4542) Update cryptographic sig file names per ASF policy
[
https://issues.apache.org/jira/browse/PHOENIX-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated PHOENIX-4542:
Attachment: PHOENIX-4542.001.patch
> Update cryptographic sig file names per ASF policy
> --
>
> Key: PHOENIX-4542
> URL: https://issues.apache.org/jira/browse/PHOENIX-4542
> Project: Phoenix
> Issue Type: Task
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4542.001.patch
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
> {quote}
> An [SHA|https://www.apache.org/dev/release-signing#sha-checksum] checksum
> SHOULD also be created and MUST be suffixed as:
> * {{.sha1}} for a SHA-1 checksum
> * {{.sha256}} for a SHA-256 checksum
> * {{.sha512}} for a SHA-512 checksum
> {quote}
> We should be using .sha256 and .sha512 instead of .sha as the filename's
> suffix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (PHOENIX-4542) Update cryptographic sig file names per ASF policy
Josh Elser created PHOENIX-4542:
---
Summary: Update cryptographic sig file names per ASF policy
Key: PHOENIX-4542
URL: https://issues.apache.org/jira/browse/PHOENIX-4542
Project: Phoenix
Issue Type: Task
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: 5.0.0, 4.14.0
https://www.apache.org/dev/release-distribution#sigs-and-sums
{quote}
An [SHA|https://www.apache.org/dev/release-signing#sha-checksum] checksum
SHOULD also be created and MUST be suffixed as:
* {{.sha1}} for a SHA-1 checksum
* {{.sha256}} for a SHA-256 checksum
* {{.sha512}} for a SHA-512 checksum
{quote}
We should be using .sha256 and .sha512 instead of .sha as the filename's suffix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system
[
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332490#comment-16332490
]
Josh Elser commented on PHOENIX-4537:
-
{quote}A simple fix could be to skip
UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes if fullTableName is a
SYSTEM table as we are not expecting indexes on a system table.
{quote}
That would prevent this deadlock, but are we OK with the RegionServers
initiating the migration? Maybe we just have to be OK with it as things will
break badly otherwise?
Let me try to put this together.
> RegionServer initiating compaction can trigger schema migration and deadlock
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
> Issue Type: Bug
>Reporter: Romil Choksi
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}}
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a
> Phoenix connection. While the RegionServer won't upgrade system tables, it
> *will* try to migrate them into the schema mapped variants (e.g.
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled
> until the region is CLOSED, and the region cannot be CLOSED until the
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table
> migrations, but Sergey and [~elserj] both think that this will end badly
> (RegionServers falling over because they expect the tables to be migrated and
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (PHOENIX-4541) rat-check fails on argparse.py
[ https://issues.apache.org/jira/browse/PHOENIX-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated PHOENIX-4541: Attachment: PHOENIX-4541.001.patch > rat-check fails on argparse.py > -- > > Key: PHOENIX-4541 > URL: https://issues.apache.org/jira/browse/PHOENIX-4541 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.13.0, 5.0.0 > > Attachments: PHOENIX-4541.001.patch > > > PHOENIX-4449 bundles a version of argparse.py into the Phoenix installation > to work around some system Python compatibility issues. This is not ALv2 but > is compatible. Need to add an exclusion to the plugin's configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (PHOENIX-4541) rat-check fails on argparse.py
Josh Elser created PHOENIX-4541: --- Summary: rat-check fails on argparse.py Key: PHOENIX-4541 URL: https://issues.apache.org/jira/browse/PHOENIX-4541 Project: Phoenix Issue Type: Bug Reporter: Josh Elser Assignee: Josh Elser Fix For: 4.13.0, 5.0.0 PHOENIX-4449 bundles a version of argparse.py into the Phoenix installation to work around some system Python compatibility issues. This is not ALv2 but is compatible. Need to add an exclusion to the plugin's configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4538) Phoenix/Spark unit tests fail to compile
[
https://issues.apache.org/jira/browse/PHOENIX-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332387#comment-16332387
]
Josh Elser commented on PHOENIX-4538:
-
I'm not seeing an obvious failure, and I have no idea why the changes in
PHOENIX-4466 would have any bearing on the phoenix-spark module – phoenix-spark
and phoenix-queryserver-client are disjoint modules and do not interact with
each other (at least going off of pom dependencies).
With java version "1.8.0_131", a {{mvn clean package -DskipTests}} passed
locally for me. Can you share some more context of what you're seeing fail,
[~jamestaylor], or a link to a PreCommit you think is busted because of this?
The only thing I have noticed is that PHOENIX-4449 broke a rat-check. Will put
up a patch to fix that.
> Phoenix/Spark unit tests fail to compile
>
>
> Key: PHOENIX-4538
> URL: https://issues.apache.org/jira/browse/PHOENIX-4538
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Priority: Major
>
> The phoenix-spark unit test fail to compile with these two commits for
> PHOENIX-4466:
> -
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594
> -
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d
> If I revert those, the test compile and pass (FYI, I'm on a Mac, so not sure
> if this a contributing factor). This may be the reason our pre-commit Jenkins
> is busted too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4538) Phoenix/Spark unit tests fail to compile
[ https://issues.apache.org/jira/browse/PHOENIX-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332354#comment-16332354 ] Josh Elser commented on PHOENIX-4538: - That's strange. I didn't this phoenix-spark was using the thin-client. Let me take a look. Sorry for the pain, James. > Phoenix/Spark unit tests fail to compile > > > Key: PHOENIX-4538 > URL: https://issues.apache.org/jira/browse/PHOENIX-4538 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Priority: Major > > The phoenix-spark unit test fail to compile with these two commits for > PHOENIX-4466: > - > https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594 > - > https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d > If I revert those, the test compile and pass (FYI, I'm on a Mac, so not sure > if this a contributing factor). This may be the reason our pre-commit Jenkins > is busted too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4534) upsert/delete/upsert for the same row corrupts the indexes
[
https://issues.apache.org/jira/browse/PHOENIX-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332182#comment-16332182
]
Rajeshbabu Chintaguntla commented on PHOENIX-4534:
--
uploaded patch works with master branch. Let's see all tests goes fine with the
patch.
> upsert/delete/upsert for the same row corrupts the indexes
> --
>
> Key: PHOENIX-4534
> URL: https://issues.apache.org/jira/browse/PHOENIX-4534
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Romil Choksi
>Assignee: Rajeshbabu Chintaguntla
>Priority: Critical
> Labels: HBase-2.0
> Fix For: 5.0
>
> Attachments: PHOENIX-4534.patch, PHOENIX-4534_v2.patch
>
>
> If we delete and upsert again the same row, the corresponding index has a
> null value.
> {noformat}
> 0: jdbc:phoenix:> create table a (id integer primary key, f float);
> No rows affected (2.272 seconds)
> 0: jdbc:phoenix:> create index i1 on a (f);
> No rows affected (5.769 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.021 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F | :ID |
> +--+--+
> | 0.5 | 1|
> +--+--+
> 1 row selected (0.016 seconds)
> 0: jdbc:phoenix:> delete from a where id = 1;
> 1 row affected (0.009 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F | :ID |
> +--+--+
> +--+--+
> No rows selected (0.015 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.008 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +---+--+
> | 0:F | :ID |
> +---+--+
> | null | 1|
> +---+--+
> 1 row selected (0.013 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (PHOENIX-4534) upsert/delete/upsert for the same row corrupts the indexes
[
https://issues.apache.org/jira/browse/PHOENIX-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajeshbabu Chintaguntla updated PHOENIX-4534:
-
Attachment: PHOENIX-4534_v2.patch
> upsert/delete/upsert for the same row corrupts the indexes
> --
>
> Key: PHOENIX-4534
> URL: https://issues.apache.org/jira/browse/PHOENIX-4534
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Romil Choksi
>Assignee: Rajeshbabu Chintaguntla
>Priority: Critical
> Labels: HBase-2.0
> Fix For: 5.0
>
> Attachments: PHOENIX-4534.patch, PHOENIX-4534_v2.patch
>
>
> If we delete and upsert again the same row, the corresponding index has a
> null value.
> {noformat}
> 0: jdbc:phoenix:> create table a (id integer primary key, f float);
> No rows affected (2.272 seconds)
> 0: jdbc:phoenix:> create index i1 on a (f);
> No rows affected (5.769 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.021 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F | :ID |
> +--+--+
> | 0.5 | 1|
> +--+--+
> 1 row selected (0.016 seconds)
> 0: jdbc:phoenix:> delete from a where id = 1;
> 1 row affected (0.009 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F | :ID |
> +--+--+
> +--+--+
> No rows selected (0.015 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.008 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +---+--+
> | 0:F | :ID |
> +---+--+
> | null | 1|
> +---+--+
> 1 row selected (0.013 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (PHOENIX-4540) Client side evaluation of group by Expression in projection gives erroneous result
[ https://issues.apache.org/jira/browse/PHOENIX-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4540: --- Attachment: PHOENIX-4540_unittest.patch > Client side evaluation of group by Expression in projection gives erroneous > result > -- > > Key: PHOENIX-4540 > URL: https://issues.apache.org/jira/browse/PHOENIX-4540 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal >Priority: Major > Attachments: PHOENIX-4540_unittest.patch > > > If the columns involved in projected expression are not present in "group by" > clause, the client evaluation of the same expression will give an erroneous > result because of the absence of involved column value. > Following queries will produce wrong result > >select round(k/v,0) x from round_test group by x,v > >select k/v x from round_test group by x,v > but query runs fine if we add all columns so that client expression can be > evaluated > >select round(k/v,0) x from round_test group by x,k,v //will produce right > >result > >select k/v x from round_test group by x,k,v; > Why we need to re-evaluate the expression here, can't we use the same result > evaluated at server side during the "group by" > thoughts [~jamestaylor]? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4540) Client side evaluation of group by Expression in projection gives erroneous result
[ https://issues.apache.org/jira/browse/PHOENIX-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332142#comment-16332142 ] Ankit Singhal commented on PHOENIX-4540: Attaching an altered test case from https://issues.apache.org/jira/browse/PHOENIX-2039 to reproduce the issue. > Client side evaluation of group by Expression in projection gives erroneous > result > -- > > Key: PHOENIX-4540 > URL: https://issues.apache.org/jira/browse/PHOENIX-4540 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal >Priority: Major > Attachments: PHOENIX-4540_unittest.patch > > > If the columns involved in projected expression are not present in "group by" > clause, the client evaluation of the same expression will give an erroneous > result because of the absence of involved column value. > Following queries will produce wrong result > >select round(k/v,0) x from round_test group by x,v > >select k/v x from round_test group by x,v > but query runs fine if we add all columns so that client expression can be > evaluated > >select round(k/v,0) x from round_test group by x,k,v //will produce right > >result > >select k/v x from round_test group by x,k,v; > Why we need to re-evaluate the expression here, can't we use the same result > evaluated at server side during the "group by" > thoughts [~jamestaylor]? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (PHOENIX-4540) Client side evaluation of group by Expression in projection gives erroneous result
Ankit Singhal created PHOENIX-4540: -- Summary: Client side evaluation of group by Expression in projection gives erroneous result Key: PHOENIX-4540 URL: https://issues.apache.org/jira/browse/PHOENIX-4540 Project: Phoenix Issue Type: Bug Reporter: Ankit Singhal If the columns involved in projected expression are not present in "group by" clause, the client evaluation of the same expression will give an erroneous result because of the absence of involved column value. Following queries will produce wrong result >select round(k/v,0) x from round_test group by x,v >select k/v x from round_test group by x,v but query runs fine if we add all columns so that client expression can be evaluated >select round(k/v,0) x from round_test group by x,k,v //will produce right >result >select k/v x from round_test group by x,k,v; Why we need to re-evaluate the expression here, can't we use the same result evaluated at server side during the "group by" thoughts [~jamestaylor]? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
