[jira] [Commented] (PHOENIX-6981) Bump Jackson version to 2.14.1
[ https://issues.apache.org/jira/browse/PHOENIX-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733570#comment-17733570 ] Geoffrey Jacoby commented on PHOENIX-6981: -- Also added [~kudivuhadi] to the contributor list for Phoenix so you can assign JIRAs to yourself now. > Bump Jackson version to 2.14.1 > -- > > Key: PHOENIX-6981 > URL: https://issues.apache.org/jira/browse/PHOENIX-6981 > Project: Phoenix > Issue Type: Improvement >Reporter: Krzysztof Sobolewski >Assignee: Krzysztof Sobolewski >Priority: Major > Fix For: 5.2.0, 5.1.4 > > > A never-ending quest to stamp out CVEs ad such. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PHOENIX-6883) Phoenix metadata caching redesign
[ https://issues.apache.org/jira/browse/PHOENIX-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17691718#comment-17691718 ] Geoffrey Jacoby commented on PHOENIX-6883: -- I like it, this sounds like a good improvement. Glad to see LAST_DDL_TIMESTAMP getting some use -- the original purpose I designed it for as part of change detection got replaced by the external schema column in 5.2. However, this will need some modification to the LAST_DDL_TIMESTAMP update semantics. Currently it only updates for DDL that changes the table schema -- i.e, table creation, column creation, column deletion. I don't think it updates if, say, you create an index on a table. Also, what about DML which is executed on the client side? (In particular, immutable tables with no index (yet) or UPSERT SELECTs with the server-side flag off) There's also an edge case if the client is further in time than the server (imagine Server A and B both have regions of the same table that Client 1 wants to write to. Client 1 writes to Server A, gets the stale metadata cache exception, and updates its metadata, then continues trying to write to Server B, which hasn't been able to update its server side metadata cache yet, either because it missed the signal from the System Catalog RS, or because its refresh query failed.) > Phoenix metadata caching redesign > - > > Key: PHOENIX-6883 > URL: https://issues.apache.org/jira/browse/PHOENIX-6883 > Project: Phoenix > Issue Type: Improvement >Reporter: Kadir Ozdemir >Priority: Major > > PHOENIX-6761 improves the client side metadata caching by eliminating the > separate cache for each connection. This improvement results in memory and > compute savings since it eliminates copying CQSI level cache every time a > Phoenix connection is created, and also replaces the inefficient the CQSI > level cache implementation with Guava Cache from Google. > Despite this improvement, the overall metadata caching architecture begs for > redesign. This is because every operation in Phoenix need to make multiple > RPCs to metadata servers for the SYSTEM.CATALOG table (please see > PHOENIX-6860) to ensure the latest metadata changes are visible to clients. > These constant RPCs makes the region servers serving SYSTEM.CATALOG hot spot > and thus leads to poor performance and availability issues. > The UPDATE_CACHE_FREQUENCY configuration parameter specifies how frequently > the client cache is updated. However, setting this parameter to a non-zero > value results in stale caching. Stale caching can cause data integrity > issues. For example, if an index table creation is not visible to the client, > Phoenix would skip updating the index table in the write path. That's why is > this parameter is typically set to zero. However, this defeats the purpose of > client side metadata caching. > The redesign of the metadata caching architecture is to directly address this > issue by making sure that the client metadata caching is always used (that > is, UPDATE_CACHE_FREQUENCY is set to NEVER) but still ensures the data > integrity. This is achieved by three main changes. > The first change is to introduce server side metadata caching in all region > servers. Currently, the server side metadata caching is used on the region > servers serving SYSTEM.CATALOG. This metadata caching should be strongly > consistent such that the metadata updates should include invalidating the > corresponding entries on the server side caches. This would ensure the server > cache would not become stale. > The second change is that the Phoenix client passes the LAST_DDL_TIMESTAMP > table attribute along with scan and mutation operations to the server regions > (more accurately to the Phoenix coprocessors). Then the Phoenix coprocessors > would check the timestamp on a given operation against with the timestamp in > its server side cache to validate that the client did not use stale metadata > when it prepared the operation. If the client did use stale metadata then the > coprocessor would return an exception (this exception can be called > StaleClientMetadataCacheException) to the client. > The third change is that upon receiving StaleClientMetadataCacheException the > Phoenix client makes an RPC call to the metadata server to update the client > cache, reconstruct the operation with the updated cached, and retry the > operation. > This redesign would require updating client and server metadata caches only > when metadata is stale instead of updating the client metadata cache for each > (scan or mutation) operation. This would eliminate hot spotting on the > metadata servers and thus poor performance and availability issues caused by > this hot spotting. > > -- This message was sent by Atlassian Jira
[jira] [Commented] (PHOENIX-6847) Provide a crisp API to check if a view exists.
[
https://issues.apache.org/jira/browse/PHOENIX-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649919#comment-17649919
]
Geoffrey Jacoby commented on PHOENIX-6847:
--
[~shahrs87] - if we had a viewExists method, we'd probably want separate
parameters for schemaName and viewName.
> Provide a crisp API to check if a view exists.
> --
>
> Key: PHOENIX-6847
> URL: https://issues.apache.org/jira/browse/PHOENIX-6847
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rushabh Shah
>Priority: Major
>
> Currently customer using phoenix runs the following statements to check if a
> view exists or not.
> {noformat}
> public static boolean checkTableViewExists(Connection connection, String
> schema, String viewName)
> throws SQLException {
> DatabaseMetaData meta = connection.getMetaData();
> ResultSet resultSet = meta.getTables(null, schema, viewName, new String[]
> {"VIEW"});
> return resultSet.next();
> }
> {noformat}
> IMHO this is NOT very user friendly.
> We need to provide either of the 2 APIs
> {noformat}
> public PTable getView(String viewName, String tenantID); OR
> public boolean viewExists(String viewName, String tenantID);
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PHOENIX-6847) Provide a crisp API to check if a view exists.
[
https://issues.apache.org/jira/browse/PHOENIX-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648739#comment-17648739
]
Geoffrey Jacoby commented on PHOENIX-6847:
--
Some years ago I asked James Taylor, who led the original development of
Phoenix, about a similar issue, and he said something like "The only public API
in Phoenix is JDBC". In this case, the above way using DatabaseMetaData is the
correct way to do it in JDBC.
That said, I've often treated PhoenixRuntime.getTable() and the PTable data
structure as quasi-public APIs, because they're both easier to work with and
much more performant than using JDBC.
This is because the Phoenix client can cache PTables, but every call to JDBC's
metadata APIs hits SYSTEM.CATALOG. And there are some subtleties in how you
structure the JDBC call -- whether you use null or "" in the catalog field if
you're looking for a non-tenant object -- that have a huge perf impact on query
perf if you have a large SYSTEM.CATALOG.
This also can create hard-to-spot correctness bugs. For example, in the code
above, the customer is _probably_ wrong to pass in null for the catalog. They
probably either are interested in a view for a particular tenant, in which case
they should pass in the tenant ID for the catalog, or they know it's a globally
owned view, in which case they should pass in the empty string (which I believe
enforces "TENANT_ID IS NULL"). Otherwise they could get a false positive if
some other tenant has a view by the same name.
So in this case, I don't have a problem with wrapping it in a PhoenixRuntime
API, though I wouldn't want _all_ of PhoenixRuntime to be considered public.
Another option would be to do it as a SQL command -- we already have a SHOW
TABLES command in PHOENIX-5543, so there could be a specific SHOW VIEWS.
> Provide a crisp API to check if a view exists.
> --
>
> Key: PHOENIX-6847
> URL: https://issues.apache.org/jira/browse/PHOENIX-6847
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rushabh Shah
>Priority: Major
>
> Currently customer using phoenix runs the following statements to check if a
> view exists or not.
> {noformat}
> public static boolean checkTableViewExists(Connection connection, String
> schema, String viewName)
> throws SQLException {
> DatabaseMetaData meta = connection.getMetaData();
> ResultSet resultSet = meta.getTables(null, schema, viewName, new String[]
> {"VIEW"});
> return resultSet.next();
> }
> {noformat}
> IMHO this is NOT very user friendly.
> We need to provide either of the 2 APIs
> {noformat}
> public PTable getView(String viewName, String tenantID); OR
> public boolean viewExists(String viewName, String tenantID);
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PHOENIX-5215) Remove and replace HTrace
[ https://issues.apache.org/jira/browse/PHOENIX-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597374#comment-17597374 ] Geoffrey Jacoby commented on PHOENIX-5215: -- [~kiran.maturi], I marked the HTrace retirement JIRA as scheduled for 5.3 since no progress has been made on it recently. If this is a critical need for 5.2, and can be completed soon, please let me know and we can put it back. Thanks! > Remove and replace HTrace > - > > Key: PHOENIX-5215 > URL: https://issues.apache.org/jira/browse/PHOENIX-5215 > Project: Phoenix > Issue Type: Bug >Reporter: Andrew Kyle Purtell >Assignee: Kiran Kumar Maturi >Priority: Major > Fix For: 5.3.0 > > > HTrace is dead. > Hadoop is discussing a replacement of HTrace with OpenTracing, see > HADOOP-15566 > HBase is having the same discussion on HBASE-22120 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PHOENIX-6771) Allow only "squash and merge" from GitHub UI
[ https://issues.apache.org/jira/browse/PHOENIX-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580361#comment-17580361 ] Geoffrey Jacoby commented on PHOENIX-6771: -- [~stoty] - while I agree there's never a need for the simple Merge commit option, the Rebase option is occasionally useful, particularly when trying to merge in a multi-JIRA feature branch. Phoenix doesn't use them too often, but [~giskender] used one for her online transform work, and some giant PRs such as the HA client (PR 1491) could have been avoided if we used them for really big features. > Allow only "squash and merge" from GitHub UI > > > Key: PHOENIX-6771 > URL: https://issues.apache.org/jira/browse/PHOENIX-6771 > Project: Phoenix > Issue Type: Bug >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: connectors-6.0.0, 5.2.0, queryserver-6.0.1, > thirdparty-2.0.1 > > > Github has three possible merging modes from the UI: > [https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/about-pull-request-merges] > Of those, we want "squash and merge" 99% of the time, but sometimes other > methods are chosen, resulting in a less than ideal commit graph. > This is configurable via .asf.yml : > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Mergebuttons > I propose that we disable the "merge" and "rebase" options for every Phoenix > repo via the .asf.yml file. > This only applies to the GitHub Web UI, the other methods can still be used > via the standard git operations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PHOENIX-6363) Review isRawFilterSupported usage in master
[ https://issues.apache.org/jira/browse/PHOENIX-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577485#comment-17577485 ] Geoffrey Jacoby commented on PHOENIX-6363: -- Right, the isRawFilterSupported flag was only relevant when run against HBase 2.1, I think, which we no longer support. > Review isRawFilterSupported usage in master > --- > > Key: PHOENIX-6363 > URL: https://issues.apache.org/jira/browse/PHOENIX-6363 > Project: Phoenix > Issue Type: Task > Components: core >Affects Versions: 5.1.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > > At fisrt glance, there seem to be places in the code where we use setRaw() > with filters without gating it by isRawFilterSupported(). > Review if this is a real problem, and fix it if it is. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PHOENIX-6733) Ref count leaked test failures
[ https://issues.apache.org/jira/browse/PHOENIX-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568667#comment-17568667 ] Geoffrey Jacoby commented on PHOENIX-6733: -- I checked the logging from the store ref check, and found that in the couple of examples I looked at it was always hbase:meta that was the region leaking. Since it's totally fine for some background process in a minicluster to be reading meta, I'm going to push up a PR with meta skipped in the check to see if that clears up the test failures. If not we may need to similarly exempt SYSTEM.CATALOG and SYSTEM.TASK. > Ref count leaked test failures > -- > > Key: PHOENIX-6733 > URL: https://issues.apache.org/jira/browse/PHOENIX-6733 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.2.0 >Reporter: Geoffrey Jacoby >Priority: Blocker > Fix For: 5.2.0 > > > In pretty much every recent Yetus test run, some tests have flapped in the > AfterClass teardown logic which tries to check for HBase Store reference > resource leaks. The error message is "Ref count leaked", and some common > suites this happens to are: > DateTimeIT > InListIT > SequenceIT > IndexToolForDeleteBeforeRebuildIT > SpooledTmpFileDeleteIT > I haven't had much luck trying to reproduce this locally. It's also not clear > yet whether the root cause is an HBase error or a Phoenix one. (And if it's a > Phoenix one, is the bug with something in Phoenix or with the resource > check?) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PHOENIX-6733) Ref count leaked test failures
[ https://issues.apache.org/jira/browse/PHOENIX-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17562837#comment-17562837 ] Geoffrey Jacoby commented on PHOENIX-6733: -- [~vjasani] [~stoty] - this appears to be because of a check introduced in PHOENIX-5296 to insure that we don't leak scanners by checking to make sure that store files don't have any references to them when an IT test ends. The check fails frequently (at least a few times in all CI runs), and it always appears to be in the ParallelStatsDisabled test runs. I don't recall ever seeing a failure in the NeedsOwnCluster test runs. That got me thinking: the difference between ParallelStats[Enabled|Disabled] and NeedsOwnCluster is of course that each of the latter gets its own dedicated minicluster, but for the former we share a minicluster between multiple IT tests. Is this check safe to run when two IT tests are using the same minicluster at the same time? Doesn't seem like it would be, because it's checking all store files, not just store files of tables created in a particular IT test. Seems like we'd either need to limit it to certain tables created in an IT test (which is hard to do generically), or limit it to only NeedsOwnCluster IT tests. Or have I missed something? > Ref count leaked test failures > -- > > Key: PHOENIX-6733 > URL: https://issues.apache.org/jira/browse/PHOENIX-6733 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.2.0 >Reporter: Geoffrey Jacoby >Priority: Blocker > Fix For: 5.2.0 > > > In pretty much every recent Yetus test run, some tests have flapped in the > AfterClass teardown logic which tries to check for HBase Store reference > resource leaks. The error message is "Ref count leaked", and some common > suites this happens to are: > DateTimeIT > InListIT > SequenceIT > IndexToolForDeleteBeforeRebuildIT > SpooledTmpFileDeleteIT > I haven't had much luck trying to reproduce this locally. It's also not clear > yet whether the root cause is an HBase error or a Phoenix one. (And if it's a > Phoenix one, is the bug with something in Phoenix or with the resource > check?) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PHOENIX-5686) MetaDataUtil local index methods return incorrect results
Title: Message Title
Geoffrey Jacoby commented on PHOENIX-5686
Re: MetaDataUtil local index methods return incorrect results
There's some additional cleanup in related methods: getLocalIndexPhysicalName and getLocalIndexTableName are also incorrect. Local indexes are stored in a special column family in the physical HBase table that holds the base table they index; the physical name should be the same as the base table's physical name. A related change necessary to this is in UpgradeUtil where these MetaDataUtil methods are incorrectly used to try to do extra logic when turning on namespace mapping for Phoenix schemas; I think much of that logic can be removed or at least grealty simplified. This, from UpgradeUtil:2416, is incorrect:
if (MetaDataUtil.isLocalIndex(srcTableName)) {
LOGGER.info(String.format("local index '%s' found with physical hbase table name ''..",
index.getName(), srcTableName));
destTableName = Bytes
.toString(MetaDataUtil.getLocalIndexPhysicalName(
newPhysicalTablename.getBytes(StandardCharsets.UTF_8)));
// update parent_table property in local index table descriptor
conn.createStatement()
.execute(String.format("ALTER TABLE %s set " +
MetaDataUtil.PARENT_TABLE_KEY + "='%s'",
phoenixTableName, table.getPhysicalName()));
}
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (PHOENIX-5066) The TimeZone is incorrectly used during writing or reading data
[
https://issues.apache.org/jira/browse/PHOENIX-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558219#comment-17558219
]
Geoffrey Jacoby commented on PHOENIX-5066:
--
Pushing this to the next version because the work seems at a standstill for
awhile, and this is something we want to be really careful about getting right
so I don't want to rush it by the 5.2 deadline.
> The TimeZone is incorrectly used during writing or reading data
> ---
>
> Key: PHOENIX-5066
> URL: https://issues.apache.org/jira/browse/PHOENIX-5066
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.14.1
>Reporter: Jaanai Zhang
>Assignee: Istvan Toth
>Priority: Critical
> Fix For: 5.3.0
>
> Attachments: DateTest.java, PHOENIX-5066.4x.v1.patch,
> PHOENIX-5066.4x.v2.patch, PHOENIX-5066.4x.v3.patch,
> PHOENIX-5066.master.v1.patch, PHOENIX-5066.master.v2.patch,
> PHOENIX-5066.master.v3.patch, PHOENIX-5066.master.v4.patch,
> PHOENIX-5066.master.v5.patch, PHOENIX-5066.master.v6.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We have two methods to write data when uses JDBC API.
> #1. Uses _the exceuteUpdate_ method to execute a string that is an upsert SQL.
> #2. Uses the _prepareStatement_ method to set some objects and execute.
> The _string_ data needs to convert to a new object by the schema information
> of tables. we'll use some date formatters to convert string data to object
> for Date/Time/Timestamp types when writes data and the formatters are used
> when reads data as well.
>
> *Uses default timezone test*
> Writing 3 records by the different ways.
> {code:java}
> UPSERT INTO date_test VALUES (1,'2018-12-10 15:40:47','2018-12-10
> 15:40:47','2018-12-10 15:40:47')
> UPSERT INTO date_test VALUES (2,to_date('2018-12-10
> 15:40:47'),to_time('2018-12-10 15:40:47'),to_timestamp('2018-12-10 15:40:47'))
> stmt.setInt(1, 3);stmt.setDate(2, date);stmt.setTime(3,
> time);stmt.setTimestamp(4, ts);
> {code}
> Reading the table by the getObject(getDate/getTime/getTimestamp) methods.
> {code:java}
> 1 | 2018-12-10 | 23:45:07 | 2018-12-10 23:45:07.0
> 2 | 2018-12-10 | 23:45:07 | 2018-12-10 23:45:07.0
> 3 | 2018-12-10 | 15:45:07 | 2018-12-10 15:45:07.66
> {code}
> Reading the table by the getString methods
> {code:java}
> 1 | 2018-12-10 15:45:07.000 | 2018-12-10 15:45:07.000 | 2018-12-10
> 15:45:07.000
> 2 | 2018-12-10 15:45:07.000 | 2018-12-10 15:45:07.000 | 2018-12-10
> 15:45:07.000
> 3 | 2018-12-10 07:45:07.660 | 2018-12-10 07:45:07.660 | 2018-12-10
> 07:45:07.660
> {code}
> *Uses GMT+8 test*
> Writing 3 records by the different ways.
> {code:java}
> UPSERT INTO date_test VALUES (1,'2018-12-10 15:40:47','2018-12-10
> 15:40:47','2018-12-10 15:40:47')
> UPSERT INTO date_test VALUES (2,to_date('2018-12-10
> 15:40:47'),to_time('2018-12-10 15:40:47'),to_timestamp('2018-12-10 15:40:47'))
> stmt.setInt(1, 3);stmt.setDate(2, date);stmt.setTime(3,
> time);stmt.setTimestamp(4, ts);
> {code}
> Reading the table by the getObject(getDate/getTime/getTimestamp) methods.
> {code:java}
> 1 | 2018-12-10 | 23:40:47 | 2018-12-10 23:40:47.0
> 2 | 2018-12-10 | 15:40:47 | 2018-12-10 15:40:47.0
> 3 | 2018-12-10 | 15:40:47 | 2018-12-10 15:40:47.106 {code}
> Reading the table by the getString methods
> {code:java}
> 1 | 2018-12-10 23:40:47.000 | 2018-12-10 23:40:47.000 | 2018-12-10
> 23:40:47.000
> 2 | 2018-12-10 15:40:47.000 | 2018-12-10 15:40:47.000 | 2018-12-10
> 15:40:47.000
> 3 | 2018-12-10 15:40:47.106 | 2018-12-10 15:40:47.106 | 2018-12-10
> 15:40:47.106
> {code}
>
> _We_ have a historical problem, we'll parse the string to
> Date/Time/Timestamp objects with timezone in #1, which means the actual data
> is going to be changed when stored in HBase table。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (PHOENIX-6740) Upgrade default supported Hadoop 3 version to 3.2.3 for HBase 2.5 profile
[ https://issues.apache.org/jira/browse/PHOENIX-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558189#comment-17558189 ] Geoffrey Jacoby commented on PHOENIX-6740: -- [~stoty] - HBASE-27148 has a Fix Version of 2.5 and an open PR, so my assumption is that it will make it into 2.5 before it ships. If it turns out that it slips to a future release, then I agree we should delay this JIRA as well. > Upgrade default supported Hadoop 3 version to 3.2.3 for HBase 2.5 profile > - > > Key: PHOENIX-6740 > URL: https://issues.apache.org/jira/browse/PHOENIX-6740 > Project: Phoenix > Issue Type: Task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 5.2.0 > > > HBase is upgrading the minimum supported Hadoop to 3.2.3 for HBase 2.5, and > we have a similar request from dependabot. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-5495) Avoid server-server RPCs when row locks are held inside MetaDataEndpointImpl
[ https://issues.apache.org/jira/browse/PHOENIX-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556562#comment-17556562 ] Geoffrey Jacoby commented on PHOENIX-5495: -- Deferring this, because any change to this logic risks consistency problems and should be done with care > Avoid server-server RPCs when row locks are held inside MetaDataEndpointImpl > > > Key: PHOENIX-5495 > URL: https://issues.apache.org/jira/browse/PHOENIX-5495 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.15.0, 5.1.0 >Reporter: Chinmay Kulkarni >Priority: Major > Fix For: 5.3.0 > > > At various spots in MetaDataEndpointImpl, we acquire row locks and then while > locks are held, do server-to-server RPC calls. This can lead to lock > starvation if RPCs take too long. We should decouple such interactions as > much as possible. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-5632) Add more information to SYSTEM.TASK TASK_DATA field apart from the task status
[ https://issues.apache.org/jira/browse/PHOENIX-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556558#comment-17556558 ] Geoffrey Jacoby commented on PHOENIX-5632: -- This requires schema changes to a system table, so I'll defer this to 5.3 unless someone wants to take it up soon in time for the 5.2 release (in which case please feel free to move the Fix Version back to 5.2 after posting a PR.) > Add more information to SYSTEM.TASK TASK_DATA field apart from the task status > -- > > Key: PHOENIX-5632 > URL: https://issues.apache.org/jira/browse/PHOENIX-5632 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Minor > Labels: beginner, newbie > Fix For: 4.17.0, 5.2.0, 4.16.2 > > Time Spent: 10m > Remaining Estimate: 0h > > It would be helpful for debugging if we could add more information to the > TASK_DATA json that is upserted into SYSTEM.TASK apart from just the task > status. For example, in failures cases, perhaps we can add the stack trace > for the failing task. > > Ideas: > * Stacktrace in case of error > * Time taken for task to complete > * Name(s) of deleted child view(s)/table(s) per task > * Task_type column is represented by int; may be useful to include task type > in task_data column -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6085) Remove duplicate calls to getSysMutexPhysicalTableNameBytes() during the upgrade path
[ https://issues.apache.org/jira/browse/PHOENIX-6085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556557#comment-17556557 ] Geoffrey Jacoby commented on PHOENIX-6085: -- I'm going to mark the Fix Version for the next version since this doesn't seem to be urgent based on the earlier conversation, but if anyone wants to take this up please feel free to move it back if/when the patch is rebased. > Remove duplicate calls to getSysMutexPhysicalTableNameBytes() during the > upgrade path > - > > Key: PHOENIX-6085 > URL: https://issues.apache.org/jira/browse/PHOENIX-6085 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Richárd Antal >Priority: Minor > Labels: phoenix-hardening, quality-improvement > Fix For: 4.17.0, 5.2.0 > > Attachments: PHOENIX-6085.4.x.v1.patch, PHOENIX-6085.master.v1.patch > > > We already make this call inside > [CQSI.acquireUpgradeMutex()|https://github.com/apache/phoenix/blob/1922895dfe5960dc025709b04acfaf974d3959dc/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java#L4220] > and then call writeMutexCell() which calls this again > [here|https://github.com/apache/phoenix/blob/1922895dfe5960dc025709b04acfaf974d3959dc/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java#L4244]. > > We should move this to inside writeMutexCell() itself and throw > UpgradeInProgressException if required there to avoid unnecessary expensive > HBase admin API calls. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6085) Remove duplicate calls to getSysMutexPhysicalTableNameBytes() during the upgrade path
[ https://issues.apache.org/jira/browse/PHOENIX-6085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556556#comment-17556556 ] Geoffrey Jacoby commented on PHOENIX-6085: -- Ah, I see from [~richardantal]'s earlier comment that PHOENIX-6085 and PHOENIX-6129 are distinct. > Remove duplicate calls to getSysMutexPhysicalTableNameBytes() during the > upgrade path > - > > Key: PHOENIX-6085 > URL: https://issues.apache.org/jira/browse/PHOENIX-6085 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Richárd Antal >Priority: Minor > Labels: phoenix-hardening, quality-improvement > Fix For: 4.17.0, 5.2.0 > > Attachments: PHOENIX-6085.4.x.v1.patch, PHOENIX-6085.master.v1.patch > > > We already make this call inside > [CQSI.acquireUpgradeMutex()|https://github.com/apache/phoenix/blob/1922895dfe5960dc025709b04acfaf974d3959dc/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java#L4220] > and then call writeMutexCell() which calls this again > [here|https://github.com/apache/phoenix/blob/1922895dfe5960dc025709b04acfaf974d3959dc/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java#L4244]. > > We should move this to inside writeMutexCell() itself and throw > UpgradeInProgressException if required there to avoid unnecessary expensive > HBase admin API calls. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6085) Remove duplicate calls to getSysMutexPhysicalTableNameBytes() during the upgrade path
[ https://issues.apache.org/jira/browse/PHOENIX-6085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556547#comment-17556547 ] Geoffrey Jacoby commented on PHOENIX-6085: -- [~vjasani] - is this needed anymore following your patch in PHOENIX-6129, since getSysMutexPhysicalBytes doesn't exist anymore? > Remove duplicate calls to getSysMutexPhysicalTableNameBytes() during the > upgrade path > - > > Key: PHOENIX-6085 > URL: https://issues.apache.org/jira/browse/PHOENIX-6085 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Richárd Antal >Priority: Minor > Labels: phoenix-hardening, quality-improvement > Fix For: 4.17.0, 5.2.0 > > Attachments: PHOENIX-6085.4.x.v1.patch, PHOENIX-6085.master.v1.patch > > > We already make this call inside > [CQSI.acquireUpgradeMutex()|https://github.com/apache/phoenix/blob/1922895dfe5960dc025709b04acfaf974d3959dc/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java#L4220] > and then call writeMutexCell() which calls this again > [here|https://github.com/apache/phoenix/blob/1922895dfe5960dc025709b04acfaf974d3959dc/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java#L4244]. > > We should move this to inside writeMutexCell() itself and throw > UpgradeInProgressException if required there to avoid unnecessary expensive > HBase admin API calls. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6341) Enable running IT tests from PHERF module during builds and patch checkins
[ https://issues.apache.org/jira/browse/PHOENIX-6341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554793#comment-17554793 ] Geoffrey Jacoby commented on PHOENIX-6341: -- This does appear to already happen -- I'll close the JIRA. > Enable running IT tests from PHERF module during builds and patch checkins > -- > > Key: PHOENIX-6341 > URL: https://issues.apache.org/jira/browse/PHOENIX-6341 > Project: Phoenix > Issue Type: Test >Reporter: Jacob Isaac >Assignee: Kiran Kumar Maturi >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-5518) Unittests for global index read repair count
[ https://issues.apache.org/jira/browse/PHOENIX-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554681#comment-17554681 ] Geoffrey Jacoby commented on PHOENIX-5518: -- I checked and it appears that this is still the case. We have tests for the repair counts when running the IndexTool, and we have unit tests that verify that the GlobalIndexChecker's metrics are hooked up correctly. However, we don't appear to have an IT test that forces GlobalIndexChecker to do some in-line read repairs and verifies the counter is as expected. Doesn't seem like a blocker for 5.2 though, so I'll push out to the next release unless someone has bandwidth to take this in the short term > Unittests for global index read repair count > > > Key: PHOENIX-5518 > URL: https://issues.apache.org/jira/browse/PHOENIX-5518 > Project: Phoenix > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Kadir OZDEMIR >Priority: Major > Fix For: 4.17.0, 5.2.0, 4.16.2 > > > [~kadir] and I were tracking down a scenario where the read repair kept > increasing. > It turned out not to be a bug, but we realized that there is not test that > checks whether the read repair count is as expected as correctness is > guaranteed in any case. > So let's add a test case based on the read repairs metric we added some time > back. > I will not have time to work in, just filing in case somebody has. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-5534) Cursors With Request Metrics Enabled Throws Exception
[
https://issues.apache.org/jira/browse/PHOENIX-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554305#comment-17554305
]
Geoffrey Jacoby commented on PHOENIX-5534:
--
I confirmed that this still occurs.
> Cursors With Request Metrics Enabled Throws Exception
> -
>
> Key: PHOENIX-5534
> URL: https://issues.apache.org/jira/browse/PHOENIX-5534
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.13.0, 4.14.1, 4.14.2, 4.14.3
>Reporter: Daniel Wong
>Priority: Major
> Fix For: 4.17.0, 5.2.0, 4.16.2
>
>
> Metrics are attempted to be setup twice during this path which causes an
> exception to be thrown.
> Recreation:
> Adding
>
> {code:java}
> props.put("phoenix.query.request.metrics.enabled","true");
>
> {code}
> To the CursorLifecycleCompile() method in CursorCompilerTest and running the
> test.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (PHOENIX-5498) When dropping a view, send delete mutations for parent->child links from client to server rather than doing server-server RPCs
[ https://issues.apache.org/jira/browse/PHOENIX-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551279#comment-17551279 ] Geoffrey Jacoby commented on PHOENIX-5498: -- Postponed this to 5.2.1, but I'm a bit worried about this approach from an atomicity standpoint and if someone takes this back up later I hope we spend some time working through the implications of that. > When dropping a view, send delete mutations for parent->child links from > client to server rather than doing server-server RPCs > -- > > Key: PHOENIX-5498 > URL: https://issues.apache.org/jira/browse/PHOENIX-5498 > Project: Phoenix > Issue Type: Sub-task >Affects Versions: 4.15.0, 5.1.0 >Reporter: Chinmay Kulkarni >Priority: Major > Fix For: 5.2.1 > > > Once we are able to generate delete mutations using the child view and parent > PTable, we should send the mutations directly from the client to the endpoint > coprocessor on SYSTEM.CHILD_LINK rather than doing a server-server RPC from > the SYSTEM.CATALOG region to the SYSTEM.CHILD_LINK region. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-5338) Test the empty column
[ https://issues.apache.org/jira/browse/PHOENIX-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551236#comment-17551236 ] Geoffrey Jacoby commented on PHOENIX-5338: -- [~jisaac] - does the existing EmptyColumnIT fulfill this JIRA? Can it be marked Resolved? > Test the empty column > - > > Key: PHOENIX-5338 > URL: https://issues.apache.org/jira/browse/PHOENIX-5338 > Project: Phoenix > Issue Type: Test >Reporter: Kadir OZDEMIR >Assignee: Jacob Isaac >Priority: Major > Fix For: 4.17.0, 5.2.0, 4.16.2 > > > Every Phoenix table includes a shadow column that is called empty column. We > need an integration test to verify the following properties of the empty > column: > # Every Phoenix table (data or index) should have the empty column > # Every HBase mutation (full or partial row) for a Phoenix table should > include the empty column cell > # Removing/adding columns from/to a Phoenix table should not impact the > above empty column properties -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6395) Reusing Connection instance object instead of creating everytime in PhoenixAccessController class.
[ https://issues.apache.org/jira/browse/PHOENIX-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17550637#comment-17550637 ] Geoffrey Jacoby commented on PHOENIX-6395: -- [~vmeka]- just checking if are you still targeting this JIRA for the 5.2 release? Looks like your PR needs a rebase and it had some test failures. If not please set the Fix Version to 5.2.1. Thanks! > Reusing Connection instance object instead of creating everytime in > PhoenixAccessController class. > --- > > Key: PHOENIX-6395 > URL: https://issues.apache.org/jira/browse/PHOENIX-6395 > Project: Phoenix > Issue Type: Bug >Reporter: vikas meka >Assignee: vikas meka >Priority: Major > Fix For: 4.17.0, 5.2.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6404) Support Hadoop 2 in master
[ https://issues.apache.org/jira/browse/PHOENIX-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17550633#comment-17550633 ] Geoffrey Jacoby commented on PHOENIX-6404: -- I don't think there's much demand for this anymore. Should this be set to Won't Do? [~vjasani] [~larsh] [~stoty]. > Support Hadoop 2 in master > -- > > Key: PHOENIX-6404 > URL: https://issues.apache.org/jira/browse/PHOENIX-6404 > Project: Phoenix > Issue Type: Wish > Components: core >Affects Versions: 5.1.0 >Reporter: Istvan Toth >Priority: Major > Fix For: 5.2.0 > > > As discussed on the the dev list, being able to build and use Phoenix on > HBase 2.x on Hadoop 2.x would be a desirable. > This probably would require adding additional maven profiles to handle the > different Hadoop versions during the build and test phases, modelled after > the HBase hadoop profiles. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6492) Validate SQL with Minicluster before Synthesizing with SchemaTool
[ https://issues.apache.org/jira/browse/PHOENIX-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17550627#comment-17550627 ] Geoffrey Jacoby commented on PHOENIX-6492: -- [~swaroopa]- can this be postponed to 5.2.1 or 5.3, or is this scheduled to be done relatively soon? Thanks! > Validate SQL with Minicluster before Synthesizing with SchemaTool > - > > Key: PHOENIX-6492 > URL: https://issues.apache.org/jira/browse/PHOENIX-6492 > Project: Phoenix > Issue Type: Improvement >Reporter: Swaroopa Kadam >Assignee: Swaroopa Kadam >Priority: Minor > Fix For: 4.17.0, 5.2.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6636) Replace bundled log4j libraries with reload4j
[ https://issues.apache.org/jira/browse/PHOENIX-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17550626#comment-17550626 ] Geoffrey Jacoby commented on PHOENIX-6636: -- [~stoty]- can this be closed (since I believe we later decided to move to log4j2?) > Replace bundled log4j libraries with reload4j > - > > Key: PHOENIX-6636 > URL: https://issues.apache.org/jira/browse/PHOENIX-6636 > Project: Phoenix > Issue Type: Bug > Components: connectors, core >Affects Versions: 5.2.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 4.17.0, 5.2.0, 4.16.2, 5.1.3 > > > To reduce the number of dependecies with unresolved CVEs, replace the bundled > log4j libraries with reload4j ([https://reload4j.qos.ch/).] > This will also require bumping the slf4j version. > This is a quick fix, and does not preclude moving to some different backend > later (like log4j2 or logback) -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6702) ConcurrentMutationsExtendedIT and PartialIndexRebuilderIT fail on Hbase 2.4.11+
[ https://issues.apache.org/jira/browse/PHOENIX-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17547024#comment-17547024 ] Geoffrey Jacoby commented on PHOENIX-6702: -- [~kozdemir]- this is the issue I mentioned to you earlier today. I looked at this some yesterday and confirmed that the failure seems to be genuine. At the end of the concurrent upsert test we're doing an IndexScrutiny (the non-MR kind) and finding at least one row that was present in the _data_ table but missing in the index table, which should never happen in a case with no deletes (which is the case for this test). The HBase JIRA that [~stoty] links above changed the HBase scanners to return early in some circumstances; I'm wondering if there's a hidden assumption somewhere in Phoenix now that we get the full page back with the option of more rows, or a partial with no extra rows. I didn't see anything suspicious looking in the GlobalIndexScanner and PagedRegionScanner. > ConcurrentMutationsExtendedIT and PartialIndexRebuilderIT fail on Hbase > 2.4.11+ > --- > > Key: PHOENIX-6702 > URL: https://issues.apache.org/jira/browse/PHOENIX-6702 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 5.2.0, 5.1.3 >Reporter: Istvan Toth >Assignee: Kadir OZDEMIR >Priority: Blocker > Attachments: bisect.sh > > > On my local machine > ConcurrentMutationsExtendedIT.testConcurrentUpserts failed 6 out 10 times > while PartialIndexRebuilderIT.testConcurrentUpsertsWithRebuild failed 10 out > of 10 times with HBase 2.4.11 (the default build) > The same tests succeeded 3 out of 3 times with HBase 2.3.7. > Either HBase 2.4 has a bug, or our compatibility modules need to be fixed. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6722) Change Data Capture - Transform Compatibility
[ https://issues.apache.org/jira/browse/PHOENIX-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545109#comment-17545109 ] Geoffrey Jacoby commented on PHOENIX-6722: -- Talked about this offline with [~giskender] and we agreed that this can be deferred to 5.2.1. In 5.2 you just shouldn't use the new online schema transform on a CDC-enabled table. > Change Data Capture - Transform Compatibility > - > > Key: PHOENIX-6722 > URL: https://issues.apache.org/jira/browse/PHOENIX-6722 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Minor > Fix For: 5.2.1 > > > When transforming a change-detection enabled table/view or altering its > logical name, we need to re-export its schema to the external schema > registry. This is because some of the changed metadata may affect how edits > from the table or view are parsed in the CDC producer. > Until this JIRA is done, online data format transforms should not be done on > change detection enabled tables. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6710) Revert PHOENIX-3842 Turn on back default bloomFilter for Phoenix Tables
[ https://issues.apache.org/jira/browse/PHOENIX-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534098#comment-17534098 ] Geoffrey Jacoby commented on PHOENIX-6710: -- [~an...@apache.org] - Makes sense that there are valid cases where bloom filters would be helpful, such as the ones you list. Seems like we'd want it to be configurable for both a default and table-specific override in a CREATE statement. Also based on PHOENIX-3797, do we need an exception if you try to create a local index on a table with a bloom filter? (Or an exception if you try to turn bloom filters on on a table that already has a local index?) > Revert PHOENIX-3842 Turn on back default bloomFilter for Phoenix Tables > --- > > Key: PHOENIX-6710 > URL: https://issues.apache.org/jira/browse/PHOENIX-6710 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 4.11.0 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > > PHOENIX-3842 was done to workaround PHOENIX-3797 to unblock a release, and > with the assumption that Phoenix is not used for GETs. > > At one of our users, we saw that they have been doing heavy GETs in their > custom coprocessor to check if the key is present or not in the current. At > most 99% of the time, the key is not expected to be present during the > initial load as keys are expected to be random, but there is still some > chance that there is 1% of keys would be duplicated. But in the absence of > BloomFilter, HBase has to seek HFile to confirm if the key is not present, > which results in regression in performance for about 2x slower. > > Even in use cases like Index maintenance and "ON DUPLICATE KEY" queries will > also be impacted without bloom filters. > > As Phoenix is still used for GETs by the users (SELECT query with key as a > filter). and we also have constructs that intrinsically do GETs like Index > maintenance and > "On Duplicate key". So I believe it is always better to have a bloom filter > should be "ON" by default as I don't also see any implication of it, even if > it is not getting used. > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6709) Create new command to update cache size of an existing sequence.
[ https://issues.apache.org/jira/browse/PHOENIX-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534030#comment-17534030 ] Geoffrey Jacoby commented on PHOENIX-6709: -- [~shahrs87]- Agreed this would be a very nice feature. We'd need to figure out which fields of the sequence were safe to update this way (e.g is it OK to update the increment by, or cache wrap flap?) and if there are any concurrency issues that we need to protect. > Create new command to update cache size of an existing sequence. > > > Key: PHOENIX-6709 > URL: https://issues.apache.org/jira/browse/PHOENIX-6709 > Project: Phoenix > Issue Type: Bug > Components: core >Reporter: Rushabh Shah >Priority: Major > > Create a new command something like: > UPDATE SEQUENCE SET CACHE_SIZE=1000 > We faced this issue in our production cluster. We create sequence with > default caching size (i.e 100). We have a use case that runs around 440k > increment calls per 5 mins which around 1500 increment ops per second. This > is consuming all the active handler threads. Depending on the load, this > causes saturation of handler threads for hours prevent any other operation on > system tables. > We decided to increase the cache size for that particular sequence to 1000 > assuming this will reduce the rpc count by 90% and the change did help. > We ran the following query: > UPSERT INTO SYSTEM."SEQUENCE"(TENANT_ID, SEQUENCE_SCHEMA, SEQUENCE_NAME, > CACHE_SIZE) SELECT TENANT_ID, SEQUENCE_SCHEMA, SEQUENCE_NAME, 1000 FROM > SYSTEM."SEQUENCE" WHERE SEQUENCE_SCHEMA='' AND > SEQUENCE_NAME=''; > Instead of this upsert query we can run some new phoenix command like: > UPDATE SEQUENCE SET CACHE_SIZE=1000 > [~gjacoby] -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6703) Exclude Jetty and servlet-api from phoenix-client
[ https://issues.apache.org/jira/browse/PHOENIX-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531280#comment-17531280 ] Geoffrey Jacoby commented on PHOENIX-6703: -- If we're getting jetty from our dependency on hbase-server, it's possible we could be indirectly using it via some HBase API that relies on it > Exclude Jetty and servlet-api from phoenix-client > - > > Key: PHOENIX-6703 > URL: https://issues.apache.org/jira/browse/PHOENIX-6703 > Project: Phoenix > Issue Type: Improvement > Components: core >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > > phoenix-client-embedded includes a Jetty server, and the Jetty API. > I don't think that there is a legitimate reason to have a web server in a > JDBC driver.It adds bloat, and tries to do some stuff at startup which takes > time. > More pressingly, the shaded javax.servlet classes conflict with minicluster, > which breaks the tests in phoenix-queryserver. > Remove Jetty and the Servlet API from the shaded phoenix-client JAR. > We could also consider excluding these dependencies in phoenix-core. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6703) Exclude Jetty and servlet-api from phoenix-client
[ https://issues.apache.org/jira/browse/PHOENIX-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531267#comment-17531267 ] Geoffrey Jacoby commented on PHOENIX-6703: -- +1 to excluding jetty and servlet dependencies from phoenix-core if they're not necessary. A quick look of the codebase shows that jetty's being used in phoenix-server (just as an exclusion) and phoenix-tracing-webapp, which make sense, but I don't see any use of it in phoenix-core. > Exclude Jetty and servlet-api from phoenix-client > - > > Key: PHOENIX-6703 > URL: https://issues.apache.org/jira/browse/PHOENIX-6703 > Project: Phoenix > Issue Type: Improvement > Components: core >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > > phoenix-client-embedded includes a Jetty server, and the Jetty API. > I don't think that there is a legitimate reason to have a web server in a > JDBC driver.It adds bloat, and tries to do some stuff at startup which takes > time. > More pressingly, the shaded javax.servlet classes conflict with minicluster, > which breaks the tests in phoenix-queryserver. > Remove Jetty and the Servlet API from the shaded phoenix-client JAR. > We could also consider excluding these dependencies in phoenix-core. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PHOENIX-6685) Change Data Capture - Populate Table / Topic Mappings
[ https://issues.apache.org/jira/browse/PHOENIX-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522540#comment-17522540 ] Geoffrey Jacoby commented on PHOENIX-6685: -- [~bharathv] [~jain.ankit] - do you have any concerns with switching to use a field on System.Catalog to store the table / topic mapping? Earlier in the project I think we were trying to keep CDC details out of syscat, but now that we're already storing external schema id as a 1:1 mapping with a table or view there that doesn't seem to be a design goal anymore. > Change Data Capture - Populate Table / Topic Mappings > -- > > Key: PHOENIX-6685 > URL: https://issues.apache.org/jira/browse/PHOENIX-6685 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 5.2.0 > > > When creating a new table with CHANGE_DETECTION_ENABLED, or when altering an > existing table to set this property to true, we should persist a mapping > between the table being enabled, and the message broker topic which will > carry its change messages. > This will allow both producers and consumers to use Phoenix metadata APIs to > figure out which topic to produce or consume from. > In the CDC prototype my colleagues and I have been working with, this has > been using a distinct system table we've manually populated. However, we may > want to consider using System.Catalog linking rows instead, to make things > more atomic. If table and topic will always be 1:1, we can also consider just > adding an optional field to SYSTEM.CATALOG that views and tables can populate. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-4830) order by primary key desc return wrong results
[
https://issues.apache.org/jira/browse/PHOENIX-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521935#comment-17521935
]
Geoffrey Jacoby commented on PHOENIX-4830:
--
[~xucang]- do you know if this issue still occurs? I know there have been a
bunch of client-side fixes over the past couple of years in this general area
> order by primary key desc return wrong results
> --
>
> Key: PHOENIX-4830
> URL: https://issues.apache.org/jira/browse/PHOENIX-4830
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.14.0
> Environment: phoenix-4.14-hbase-1.2
>Reporter: JieChen
>Assignee: Xu Cang
>Priority: Major
> Labels: DESC
> Fix For: 4.17.0, 5.2.0, 4.16.2
>
> Attachments: PHOENIX-4830-4.x-HBase-1.3.001.patch,
> PHOENIX-4830-4.x-HBase-1.3.002.patch, PHOENIX-4830-4.x-HBase-1.3.003.patch,
> PHOENIX-4830-4.x-HBase-1.3.004.patch, PHOENIX-4830-4.x-HBase-1.3.005.patch,
> PHOENIX-4830-4.x-HBase-1.3.006.patch, PHOENIX-4830-4.x-HBase-1.3.007.patch,
> PHOENIX-4830-4.x-HBase-1.3.007.patch, PHOENIX-4830-4.x-HBase-1.3.008.patch
>
>
> {code:java}
> 0: jdbc:phoenix:localhost> create table test(id bigint not null primary key,
> a bigint);
> No rows affected (1.242 seconds)
> 0: jdbc:phoenix:localhost> upsert into test values(1,11);
> 1 row affected (0.01 seconds)
> 0: jdbc:phoenix:localhost> upsert into test values(2,22);
> 1 row affected (0.007 seconds)
> 0: jdbc:phoenix:localhost> upsert into test values(3,33);
> 1 row affected (0.005 seconds)
> 0: jdbc:phoenix:localhost> select * from test;
> +-+-+
> | ID | A |
> +-+-+
> | 1 | 11 |
> | 2 | 22 |
> | 3 | 33 |
> +-+-+
> 3 rows selected (0.015 seconds)
> 0: jdbc:phoenix:localhost> select * from test order by id desc limit 2 offset
> 0;
> +-+-+
> | ID | A |
> +-+-+
> | 3 | 33 |
> | 2 | 22 |
> +-+-+
> 2 rows selected (0.018 seconds)
> 0: jdbc:phoenix:localhost> select * from test where id in (select id from
> test ) order by id desc limit 2 offset 0;
> +-+-+
> | ID | A |
> +-+-+
> | 2 | 22 |
> | 1 | 11 |
> +-+-+
> wrong results.
> {code}
> there may be some errors. ScanUtil.setupReverseScan code.
> then
> {code:java}
> 0: jdbc:phoenix:localhost> upsert into test values(4,33);
> 1 row affected (0.005 seconds)
> 0: jdbc:phoenix:localhost> upsert into test values(5,23);
> 1 row affected (0.005 seconds)
> 0: jdbc:phoenix:localhost> upsert into test values(6,23);
> 1 row affected (0.005 seconds)
> 0: jdbc:phoenix:localhost> upsert into test values(7,33);
> 1 row affected (0.006 seconds)
> {code}
> execute sql
> {code:java}
> select * from test where id in (select id from test where a=33) order by id
> desc;
> {code}
> throw exception
> {code:java}
> Error: org.apache.phoenix.exception.PhoenixIOException:
> org.apache.hadoop.hbase.DoNotRetryIOException:
> TEST,,1533266754845.b8e521d4dc8e8b8f18c69cc7ef76973d.: The next hint must
> come after previous hint
> (prev=\x80\x00\x00\x00\x00\x00\x00\x07//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0,
>
> next=\x80\x00\x00\x00\x00\x00\x00\x07//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0,
>
> kv=\x80\x00\x00\x00\x00\x00\x00\x06/0:\x00\x00\x00\x00/1533266778944/Put/vlen=1/seqid=9)
> at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:96)
> at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:62)
> at
> org.apache.phoenix.iterate.RegionScannerFactory$1.nextRaw(RegionScannerFactory.java:212)
> at
> org.apache.phoenix.coprocessor.HashJoinRegionScanner.nextRaw(HashJoinRegionScanner.java:264)
> at
> org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:82)
> at
> org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:82)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:293)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2541)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33648)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2183)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)
> Caused by: java.lang.IllegalStateException: The next hint must come after
> previous hint
> (prev=\x80\x00\x00\x00\x00\x00\x00\x07//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0,
>
> next=\x80\x00\x00\x00\x00\x00\x00\x07//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0,
>
>
[jira] [Commented] (PHOENIX-6677) Parallelism within a batch of mutations
[ https://issues.apache.org/jira/browse/PHOENIX-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514132#comment-17514132 ] Geoffrey Jacoby commented on PHOENIX-6677: -- Great idea. I think it's really important that it be tunable though. I know some use cases in my environment execute on clients that are really CPU-bound, so I suspect that in some situations this might perform less well due to introducing more context switching. With less contention, or in a situation such as PQS, this sounds like it would be very helpful, however. > Parallelism within a batch of mutations > > > Key: PHOENIX-6677 > URL: https://issues.apache.org/jira/browse/PHOENIX-6677 > Project: Phoenix > Issue Type: Improvement >Reporter: Kadir OZDEMIR >Priority: Major > Fix For: 4.17.0, 5.2.0 > > > Currently, Phoenix client simply passes the batches of row mutations from the > application to HBase client without any parallelism or intelligent grouping > (except grouping mutations for the same row). > Assume that the application creates batches 1 row mutations for a given > table. Phoenix client divides these rows based on their arrival order into > HBase batches of n (e.g., 100) rows based on the configured batch size, i.e., > the number of rows and bytes. Then, Phoenix calls HBase batch API, one batch > at a time (i.e., serially). HBase client further divides a given batch of > rows into smaller batches based on their regions. This means that a large > batch created by the application is divided into many tiny batches and > executed mostly serially. For slated tables, this will result in even smaller > batches. > We can improve the current implementation greatly if we group the rows of the > batch prepared by the application into sub batches based on table region > boundaries and then execute these batches in parallel. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (PHOENIX-6561) Allow pherf to intake phoenix Connection properties as argument.
[ https://issues.apache.org/jira/browse/PHOENIX-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442442#comment-17442442 ] Geoffrey Jacoby edited comment on PHOENIX-6561 at 11/11/21, 8:24 PM: - [~jisaac] [~lokiore] - Reopening this JIRA because the 4.x patch is causing a build error. (PhoenixUtil is using a HashMap without importing it). Could you please fix when you get a minute? was (Author: gjacoby): [~jisaac] [~lokiore] - Reopening this JIRA because the 4.x patch is causing a build error. (PhoenixUtil is using a HashMap without importing it) > Allow pherf to intake phoenix Connection properties as argument. > > > Key: PHOENIX-6561 > URL: https://issues.apache.org/jira/browse/PHOENIX-6561 > Project: Phoenix > Issue Type: Improvement >Reporter: Lokesh Khurana >Assignee: Lokesh Khurana >Priority: Minor > Fix For: 4.17.0, 5.2.0, 4.16.2, 5.1.3 > > > Currently pherf doesn't allow connection properties to be passed as > arguments, it allows for some cases through scenario files, but for dynamic > properties selection that might not work, also for WriteWorkload no property > is being allowed to pass during connection creation. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6548) Race condition when triggering index rebuilds as regionserver closes
[ https://issues.apache.org/jira/browse/PHOENIX-6548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427291#comment-17427291 ] Geoffrey Jacoby commented on PHOENIX-6548: -- Also cherry-picked to 4.16 and 5.1. Thanks for the contribution, [~jainankit]! > Race condition when triggering index rebuilds as regionserver closes > > > Key: PHOENIX-6548 > URL: https://issues.apache.org/jira/browse/PHOENIX-6548 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.3, 4.16.1 >Reporter: Geoffrey Jacoby >Assignee: Ankit Jain >Priority: Minor > Fix For: 4.17.0, 5.2.0, 4.16.2, 5.1.3 > > > On each regionserver our coprocs keep a cache of HConnections with custom > settings (such as short timeouts) for talking to other regionservers. They're > used when coprocs need to make RPCs, such as during index rebuilds. > When a regionserver is closed, these HConnections are closed as well. > However, we've seen in our test pipelines a race condition where we may have > just given out one of the HConnections to a coprocessor, only to have the > connection closed just before it's used. > This will produce an IllegalArgumentException from the HBase Table object, > which (if the index rebuild was caused by a client Scan) will be thrown back > to the client as a DoNotRetryIOException. > In this case we want the client exception to be a normal retriable > IOException, because if they try again after the region comes up again > somewhere else, the Scan will likely succeed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6539) PHOENIX-6397 breaks 4.x builds for HBase 1.3 profile
[ https://issues.apache.org/jira/browse/PHOENIX-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412650#comment-17412650 ] Geoffrey Jacoby commented on PHOENIX-6539: -- [~stoty]- do we still support HBase 1.3? I thought that 4.16 was going to be the last release with 1.3 support. (No objection if you want to fix this, I'm just curious.) > PHOENIX-6397 breaks 4.x builds for HBase 1.3 profile > > > Key: PHOENIX-6539 > URL: https://issues.apache.org/jira/browse/PHOENIX-6539 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 4.17.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Blocker > > PHOENIX-6397 introduces org.apache.phoenix.monitoring.JmxMetricProvider, > which depends on HBase and Phoenix feature not present in 1.3, thus breaking > the build. > The best solution is probably pushing down the Class to the the compat > modules, with a dummy implementation for the HBase 1.3 profile. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6457) Optionally store schema version string in SYSTEM.CATALOG
[ https://issues.apache.org/jira/browse/PHOENIX-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343531#comment-17343531 ] Geoffrey Jacoby commented on PHOENIX-6457: -- [~larsh] - this changes the schema of System.Catalog, which makes it ineligible to be in a patch release. Hence it went in 4.x (4.17) and master (5.2) but not 5.1 (5.1.2) or 4.16 (4.16.2) > Optionally store schema version string in SYSTEM.CATALOG > > > Key: PHOENIX-6457 > URL: https://issues.apache.org/jira/browse/PHOENIX-6457 > Project: Phoenix > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 4.17.0, 5.2.0 > > > In many environments, schema changes to Phoenix tables are applied in batches > associated with a version of an application. (For example, v1.0 of an app may > start with one set of CREATE statements, v1.1 then adds some ALTER > statements, etc.) > It can be useful to be able to look up the latest app version in which a > table or view was changed; this could potentially be added as a feature of > the Schema Tool. > This change would add an optional property to CREATE and ALTER statements, > SCHEMA_VERSION, which would take a user-supplied string. > This is also a pre-req for PHOENIX-6227, because we would want to pass the > schema version string, if any, to an external schema repository in > environments where we're integrating with one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-3725) Add support for per VIEW level TTLs in addition to table level TTLs
[ https://issues.apache.org/jira/browse/PHOENIX-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340306#comment-17340306 ] Geoffrey Jacoby commented on PHOENIX-3725: -- [~jisaac] [~yanxinyi], can this umbrella JIRA for Phoenix/View TTL be marked resolved since all the subtasks look complete? Is there more work to do here? > Add support for per VIEW level TTLs in addition to table level TTLs > --- > > Key: PHOENIX-3725 > URL: https://issues.apache.org/jira/browse/PHOENIX-3725 > Project: Phoenix > Issue Type: New Feature >Reporter: Jan Fernando >Assignee: Jacob Isaac >Priority: Major > > We are seeing more and more use cases where it would be very helpful to > declare a TTL for data at the View level instead of the table level. We have > several use cases where a shared Phoenix table is partitioned by views and > each view has some shared columns with the base table and it's own unique > columns. In these cases it's not possible to declare a TTL for data at the > table level as each view may warrant a different TTL. > As a result, enforcement of a TTL for data relies on custom jobs that query > the data that has expired and then deletes them via SQL. This has 2 major > disadvantages a) developers have to write a custom job and b) we can > potentially create a lot of delete markers in HBase. > From an end-user perspective it would really simplify the development > experience if users could instead simply declare a TTL on the view and let > Phoenix/HBase take care of deletion at compaction time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6454) Add feature to SchemaTool to get the DDL in specification mode
[ https://issues.apache.org/jira/browse/PHOENIX-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329351#comment-17329351 ] Geoffrey Jacoby commented on PHOENIX-6454: -- Will there also be an ability to just get the effective DDL SQL as it currently exists, without providing a new create or alter statement? > Add feature to SchemaTool to get the DDL in specification mode > -- > > Key: PHOENIX-6454 > URL: https://issues.apache.org/jira/browse/PHOENIX-6454 > Project: Phoenix > Issue Type: Improvement >Reporter: Swaroopa Kadam >Assignee: Swaroopa Kadam >Priority: Major > Fix For: 4.17.0, 5.2.0 > > > Currently, SchemExtractionTool uses PTable representation to get the > effective DDL on the cluster. > Rename SchemaExtractionTool to SchemaTool, add a feature that accepts create > DDL and alter DDL to give effective DDL without using PTable implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5705) SYSTEM.CATALOG rows are rewritten with high frequency
[
https://issues.apache.org/jira/browse/PHOENIX-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304417#comment-17304417
]
Geoffrey Jacoby commented on PHOENIX-5705:
--
[~tkhurana] - I'm curious, how did we determine that this isn't a problem?
> SYSTEM.CATALOG rows are rewritten with high frequency
> -
>
> Key: PHOENIX-5705
> URL: https://issues.apache.org/jira/browse/PHOENIX-5705
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.14.3
>Reporter: Priyank Porwal
>Assignee: Tanuj Khurana
>Priority: Major
> Fix For: 4.14.4, 4.16.1, 4.17.0
>
>
> It seems every few ms, the following cells are being rewritten without any
> change in values - PENDING_DISABLE_COUNT,
> INDEX_STATE,INDEX_DISABLE_TIMESTAMP,ASYNC_REBUILD_TIMESTAMP,_0
> Sometimes the rewrites have the same cell timestamps as well.
> These cells are all relevant to secondary indexes to it is likely that
> automated index rebuild system or Phoenix MR job submitter has an issue. This
> is true even if the index state is Active 'a'.
>
> REPRO:
> scan 'SYSTEM.CATALOG',
> {LIMIT=>1,RAW=>true,VERSIONS=>10,STARTROW=>""
> Get syscatrowkey:
> SELECT ROWKEY_BYTES_STRING() FROM SYSTEM.CATALOG WHERE COLUMN_NAME IS NULL
> AND DATA_TABLE_NAME IS NOT NULL AND TABLE_NAME = ""
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-6377) phoenix-client has erronous maven dependecies
[ https://issues.apache.org/jira/browse/PHOENIX-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282753#comment-17282753 ] Geoffrey Jacoby commented on PHOENIX-6377: -- I'm a bit unclear on how to detect the problem -- running mvn dependency:tree seems to yield the right answers against both 4.x and master (hbase dependencies vary depending on hbase.profile param value) We're sure that this also affects the RC of 4.16 that's currently up for vote? I was about to +1 it, but if this affects 4.16 as well that seems worth an RC3 to fix, rather than having to do a 4.16.1 shortly. [~yanxinyi], fyi. > phoenix-client has erronous maven dependecies > - > > Key: PHOENIX-6377 > URL: https://issues.apache.org/jira/browse/PHOENIX-6377 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 5.1.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Critical > > depending on phoenix-client brings in phoenix-core, and all of its > dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6374) Publish perf workload results and analysis
[ https://issues.apache.org/jira/browse/PHOENIX-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282521#comment-17282521 ] Geoffrey Jacoby commented on PHOENIX-6374: -- Thanks for publishing this, [~jisaac]! Just want to point out that I believe the 4.14.3 perf run that Jacob did was from an internal SFDC version of Phoenix. (Please correct me if I'm wrong.) It has some significant differences from the original Apache release that might affect perf (mostly select pieces of 4.15 and 4.16 pulled in early). The 4.15 numbers are from the Apache release of 4.15. > Publish perf workload results and analysis > -- > > Key: PHOENIX-6374 > URL: https://issues.apache.org/jira/browse/PHOENIX-6374 > Project: Phoenix > Issue Type: Test >Affects Versions: 4.x >Reporter: Jacob Isaac >Assignee: Jacob Isaac >Priority: Major > > Ran perf workloads against 4.14.x, 4.15.x and 4.16RC1 build. > The results and observations are published here for review - > https://docs.google.com/document/d/19QHG6vvdxwCNkT3nqu8N-ift_1OIn161pqtJx1UcXiY/edit# > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6348) java.lang.NoClassDefFoundError: when running with hbase-1.6
[ https://issues.apache.org/jira/browse/PHOENIX-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278300#comment-17278300 ] Geoffrey Jacoby commented on PHOENIX-6348: -- [~yanxinyi] doesn't this need to be 4.16.0, not 4.16.1? > java.lang.NoClassDefFoundError: when running with hbase-1.6 > --- > > Key: PHOENIX-6348 > URL: https://issues.apache.org/jira/browse/PHOENIX-6348 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.16.0 >Reporter: Jacob Isaac >Priority: Blocker > Fix For: 4.16.1, 4.17.0 > > > Getting this error, when running with hbase-1.6 > I think this stems from the jar dependency mismatch between phoenix 4.x/4.16 > and hbase1.6 > hbase-1.6 : commons-cli-1.2.jar > (https://github.com/apache/hbase/blob/5ec5a5b115ee36fb28903667c008218abd21b3f5/pom.xml#L1260) > phoenix 4.x : commons-cli-1.4.jar > ([https://github.com/apache/phoenix/blob/44d44029597d032af1be54d5e9a70342c1fe4769/pom.xml#L100)] > > What is the best way to resolve this? Shading? > [~stoty] [~vjasani] > FYI > [~yanxinyi] [~ChinmayKulkarni] [~kadir] > > **Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/commons/cli/DefaultParser > at > org.apache.phoenix.mapreduce.index.IndexTool.parseOptions(IndexTool.java:354) > at org.apache.phoenix.mapreduce.index.IndexTool.run(IndexTool.java:788) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) > at org.apache.phoenix.mapreduce.index.IndexTool.main(IndexTool.java:1201) > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.cli.DefaultParser > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5874) IndexTool does not set TTL on its log tables correctly
[ https://issues.apache.org/jira/browse/PHOENIX-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277304#comment-17277304 ] Geoffrey Jacoby commented on PHOENIX-5874: -- [~yanxinyi] - this was ported to master as part of the giant index forward-port in PHOENIX-6067. > IndexTool does not set TTL on its log tables correctly > -- > > Key: PHOENIX-5874 > URL: https://issues.apache.org/jira/browse/PHOENIX-5874 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.0.0, 4.14.3 >Reporter: Kadir OZDEMIR >Assignee: Swaroopa Kadam >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5874.4.x.v1.patch, PHOENIX-5874.4.x.v2.patch > > Time Spent: 40m > Remaining Estimate: 0h > > IndexTool does not use the correct API to set 7 day TTL on its log tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6348) java.lang.NoClassDefFoundError: when running with hbase-1.6
[ https://issues.apache.org/jira/browse/PHOENIX-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17274033#comment-17274033 ] Geoffrey Jacoby commented on PHOENIX-6348: -- [~jisaac] Can we just exclude commons-cli 1.2 from the hbase dependencies? > java.lang.NoClassDefFoundError: when running with hbase-1.6 > --- > > Key: PHOENIX-6348 > URL: https://issues.apache.org/jira/browse/PHOENIX-6348 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.16.0 >Reporter: Jacob Isaac >Priority: Blocker > Fix For: 4.16.0 > > > Getting this error, when running with hbase-1.6 > I think this stems from the jar dependency mismatch between phoenix 4.x/4.16 > and hbase1.6 > hbase-1.6 : commons-cli-1.2.jar > (https://github.com/apache/hbase/blob/5ec5a5b115ee36fb28903667c008218abd21b3f5/pom.xml#L1260) > phoenix 4.x : commons-cli-1.4.jar > ([https://github.com/apache/phoenix/blob/44d44029597d032af1be54d5e9a70342c1fe4769/pom.xml#L100)] > > What is the best way to resolve this? Shading? > [~stoty] [~vjasani] > FYI > [~yanxinyi] [~ChinmayKulkarni] [~kadir] > > **Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/commons/cli/DefaultParser > at > org.apache.phoenix.mapreduce.index.IndexTool.parseOptions(IndexTool.java:354) > at org.apache.phoenix.mapreduce.index.IndexTool.run(IndexTool.java:788) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) > at org.apache.phoenix.mapreduce.index.IndexTool.main(IndexTool.java:1201) > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.cli.DefaultParser > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6298) Use timestamp of PENDING_DISABLE_COUNT to calculate elapse time for PENDING_DISABLE state
[
https://issues.apache.org/jira/browse/PHOENIX-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259050#comment-17259050
]
Geoffrey Jacoby commented on PHOENIX-6298:
--
I just want to point out that this only affects the old indexing framework,
because the new one doesn't auto-disable indexes. (It repairs write failures at
read time.)
Since existing indexes have to be manually upgraded to the new framework
(either by dropping/recreating or by using the IndexUpgradeTool), we'll need to
support the old Indexer-style indexes for a little while yet, so still good to
fix bugs / race conditions when we find them.
> Use timestamp of PENDING_DISABLE_COUNT to calculate elapse time for
> PENDING_DISABLE state
> -
>
> Key: PHOENIX-6298
> URL: https://issues.apache.org/jira/browse/PHOENIX-6298
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: Ankit Singhal
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 5.1.0, 4.16.0
>
>
> Instead of taking indexDisableTimestamp to calculate the elapsed time, we
> should be considering the last time we incr/decremented the counter for
> PENDING_DISABLE_COUNT. as if the application writes failures span more than
> the default threshold of 30 seconds, the index will unnecessarily get
> disabled even though the client could have retried and made it active.
> {code}
> long elapsedSinceDisable =
> EnvironmentEdgeManager.currentTimeMillis() - Math.abs(indexDisableTimestamp);
> // on an index write failure, the server side transitions to PENDING_DISABLE,
> then the client
> // retries, and after retries are exhausted, disables the
> index
> if (indexState == PIndexState.PENDING_DISABLE) {
> if (elapsedSinceDisable > pendingDisableThreshold) {
> // too long in PENDING_DISABLE - client didn't
> disable the index, so we do it here
> IndexUtil.updateIndexState(conn,
> indexTableFullName, PIndexState.DISABLE, indexDisableTimestamp);
> }
> continue;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5435) Annotate HBase WALs with Phoenix Metadata
[ https://issues.apache.org/jira/browse/PHOENIX-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250817#comment-17250817 ] Geoffrey Jacoby commented on PHOENIX-5435: -- [~stoty] - yes, switching to default 1.5 was my intention. Been meaning to bring up a discussion about this on the dev list, which I hope to do tomorrow, to evaluate options. The problem with defaulting to the lowest HBase version when doing test runs (either locally or in Jenkins) is that there are now quite a few tests that don't get exercised on earlier versions of HBase. (This is particularly true on the master branch, where many secondary index tests no-op when run against HBase 2.1 or 2.2.) I made the switch as part of this JIRA because WAL annotations are only supported in newer versions of branch-1 (1.5+ I think? and only 2.3 and up when I port to master in the next few days), so the tests would never get executed at default settings. > Annotate HBase WALs with Phoenix Metadata > - > > Key: PHOENIX-5435 > URL: https://issues.apache.org/jira/browse/PHOENIX-5435 > Project: Phoenix > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5435-4.x.patch > > > HBase write-ahead-logs (WALs) drive not only failure recovery, but HBase > replication and some HBase backup frameworks. The WALs contain HBase-level > metadata such as table and region, but lack Phoenix-level metadata. That > means that it's quite difficult to build correct logic that needs to know > about Phoenix-level constructs such as multi-tenancy, views, or indexes. > HBASE-22622 and HBASE-22623 add the capacity for coprocessors to annotate > extra key/value pairs of metadata into the HBase WAL. We should have the > option to annotate the tuple , or > some hashed way to reconstruct that tuple into the WAL. It should have a > feature toggle so operators who don't need it don't bear the slight extra > storage cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6267) View Index PK Fixed Width Field Truncation
[
https://issues.apache.org/jira/browse/PHOENIX-6267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250677#comment-17250677
]
Geoffrey Jacoby commented on PHOENIX-6267:
--
Also, it's weird that we truncate the normal, ASC SEPARATOR_BYTE, but not
DESC_SEPARATOR_BYTE
> View Index PK Fixed Width Field Truncation
> ---
>
> Key: PHOENIX-6267
> URL: https://issues.apache.org/jira/browse/PHOENIX-6267
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0, 4.14.3
>Reporter: Geoffrey Jacoby
>Assignee: Gokcen Iskender
>Priority: Blocker
> Fix For: 5.1.0, 4.16.0
>
>
> 1. Create a table T
> 2. Create a view V that extends the PK of T, where the last field of the
> view's PK is a fixed width column C such as an INTEGER or BIGINT
> 3. Create a view index VI on V that does NOT contain C in its index key
> 4. Select C from the view with a query that runs against the view index
> You will get the following error:
> {code:java}
> java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected length of
> at least 8 bytes, but had 7
> {code}
> [~giskender], [~kadir], [~tkhurana] and I investigated. It appears the
> problem comes from
> IndexMaintainer.buildRowKey:707-713
> {code:java}
> int length = stream.size();
> int minLength = length - maxTrailingNulls;
> byte[] indexRowKey = stream.getBuffer();
> // Remove trailing nulls
> while (length > minLength && indexRowKey[length-1] ==
> QueryConstants.SEPARATOR_BYTE) {
> length--;
> }
> {code}
> The logic to remove trailing separator bytes shouldn't run if the end of the
> index key (i.e the end of the base PK) is a fixed width column. Otherwise,
> we'll truncate whenever the last byte of the fixed width field happens to be
> \x00
>
> This only happens for view indexes (mutitenant or not)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-6058) When maxLookback is not enabled, IndexTool should not do deep verification
[ https://issues.apache.org/jira/browse/PHOENIX-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249245#comment-17249245 ] Geoffrey Jacoby commented on PHOENIX-6058: -- Tests should pass on all supported versions on each branch (1.3, 1.4, 1.5, 1.6 for 4.x and 2.1, 2.2, and 2.3 for master). In some cases if a test just exercises a feature that doesn't exist for a given HBase branch, it should use Assume.assumeTrue(HbaseCompatCapabilities.someVersionSpecificFlag) to be ignored where not supported In other cases, where the presence or absence of a feature changes the correct answer of a test, the test should use an appropriate flag to control its assertions. Looks like the latter is what the master addendum does, so +1 > When maxLookback is not enabled, IndexTool should not do deep verification > -- > > Key: PHOENIX-6058 > URL: https://issues.apache.org/jira/browse/PHOENIX-6058 > Project: Phoenix > Issue Type: Bug >Reporter: Kadir OZDEMIR >Assignee: Kadir OZDEMIR >Priority: Blocker > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6058.4.x.001.patch, PHOENIX-6058.4.x.002.patch, > PHOENIX-6058.master.001.patch > > > IndexTool does two types of verification, verifying the latest row versions, > and verifying all row versions within the maxLookback windows. Verifying just > the latest version can be called shallow verification whereas verifying all > versions can be called deep verification. The maxLookback feature preserves > all row versions including delete markers within the maxLookback window. This > allows IndexTool to compare all versions of data rows with all versions of > the index rows within the window. However, due to a bug, when the maxLookback > feature is not enabled, i.e., the maxLookback window is set to zero, > IndexTool assumes the window size is infinite and thus does deep > verification. Since some versions of the index rows can be removed by > compaction independently if the maxLookback feature is not enabled, IndexTool > may report missing or invalid index rows by doing deep verification. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6058) When maxLookback is not enabled, IndexTool should not do deep verification
[ https://issues.apache.org/jira/browse/PHOENIX-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249227#comment-17249227 ] Geoffrey Jacoby commented on PHOENIX-6058: -- [~kozdemir]- max lookback is supported in master when running against HBase 2.3 (and the future 2.4) > When maxLookback is not enabled, IndexTool should not do deep verification > -- > > Key: PHOENIX-6058 > URL: https://issues.apache.org/jira/browse/PHOENIX-6058 > Project: Phoenix > Issue Type: Bug >Reporter: Kadir OZDEMIR >Assignee: Kadir OZDEMIR >Priority: Blocker > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6058.4.x.001.patch, PHOENIX-6058.4.x.002.patch, > PHOENIX-6058.master.001.patch > > > IndexTool does two types of verification, verifying the latest row versions, > and verifying all row versions within the maxLookback windows. Verifying just > the latest version can be called shallow verification whereas verifying all > versions can be called deep verification. The maxLookback feature preserves > all row versions including delete markers within the maxLookback window. This > allows IndexTool to compare all versions of data rows with all versions of > the index rows within the window. However, due to a bug, when the maxLookback > feature is not enabled, i.e., the maxLookback window is set to zero, > IndexTool assumes the window size is infinite and thus does deep > verification. Since some versions of the index rows can be removed by > compaction independently if the maxLookback feature is not enabled, IndexTool > may report missing or invalid index rows by doing deep verification. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6058) When maxLookback is not enabled, IndexTool should not do deep verification
[ https://issues.apache.org/jira/browse/PHOENIX-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241237#comment-17241237 ] Geoffrey Jacoby commented on PHOENIX-6058: -- [~yanxinyi] We definitely want to get this into 4.16 and 5.1 before they release, because otherwise the new indexes will be particularly hard to use with HBase 1.3, 2.1 and 2.2. > When maxLookback is not enabled, IndexTool should not do deep verification > -- > > Key: PHOENIX-6058 > URL: https://issues.apache.org/jira/browse/PHOENIX-6058 > Project: Phoenix > Issue Type: Bug >Reporter: Kadir OZDEMIR >Assignee: Kadir OZDEMIR >Priority: Blocker > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6058.4.x.001.patch, PHOENIX-6058.4.x.002.patch, > PHOENIX-6058.master.001.patch > > > IndexTool does two types of verification, verifying the latest row versions, > and verifying all row versions within the maxLookback windows. Verifying just > the latest version can be called shallow verification whereas verifying all > versions can be called deep verification. The maxLookback feature preserves > all row versions including delete markers within the maxLookback window. This > allows IndexTool to compare all versions of data rows with all versions of > the index rows within the window. However, due to a bug, when the maxLookback > feature is not enabled, i.e., the maxLookback window is set to zero, > IndexTool assumes the window size is infinite and thus does deep > verification. Since some versions of the index rows can be removed by > compaction independently if the maxLookback feature is not enabled, IndexTool > may report missing or invalid index rows by doing deep verification. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6224) Support Correlated IN Subquery
[
https://issues.apache.org/jira/browse/PHOENIX-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232898#comment-17232898
]
Geoffrey Jacoby commented on PHOENIX-6224:
--
I'm not sure I'd classify this as a Bug, because I thought that we just didn't
support correlated subqueries? Would be a good feature to have, of course, esp.
since we already support joins and in-clause correlated subqueries are often
equivalent to a join.
> Support Correlated IN Subquery
> ---
>
> Key: PHOENIX-6224
> URL: https://issues.apache.org/jira/browse/PHOENIX-6224
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Fix For: 5.1.0, 4.16.0
>
>
> Given following tables :
> {code:java}
> create table item
>(item_id varchar not null primary key,
> name varchar,
> price integer,
> discount1 integer,
> discount2 integer,
> supplier_id varchar,
>description varchar)
>
> create table order
> ( order_id varchar not null primary key,
>customer_id varchar,
>item_id varchar,
>price integer,
>quantity integer,
>date timestamp)
> {code}
> for the correlated in subquery:
> {code:java}
> SELECT item_id, name FROM item i WHERE i.item_id IN
> (SELECT item_id FROM order o where o.price = i.price) ORDER BY name
> {code}
> Phoenix would throw following exception, that is because phoenix only support
> Non-Correlated In Subquery now:
> {code:java}
> org.apache.phoenix.schema.ColumnFamilyNotFoundException: ERROR 1001 (42I01):
> Undefined column family. familyName=I
> at
> org.apache.phoenix.schema.PTableImpl.getColumnFamily(PTableImpl.java:1363)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:527)
> at
> org.apache.phoenix.compile.ExpressionCompiler.resolveColumn(ExpressionCompiler.java:368)
> at
> org.apache.phoenix.compile.WhereCompiler$WhereExpressionCompiler.resolveColumn(WhereCompiler.java:191)
> at
> org.apache.phoenix.compile.WhereCompiler$WhereExpressionCompiler.visit(WhereCompiler.java:177)
> at
> org.apache.phoenix.compile.ExpressionCompiler.visit(ExpressionCompiler.java:1)
> at
> org.apache.phoenix.parse.ColumnParseNode.accept(ColumnParseNode.java:56)
> at
> org.apache.phoenix.parse.CompoundParseNode.acceptChildren(CompoundParseNode.java:64)
> at
> org.apache.phoenix.parse.ComparisonParseNode.accept(ComparisonParseNode.java:45)
> at
> org.apache.phoenix.compile.WhereCompiler.compile(WhereCompiler.java:138)
> at
> org.apache.phoenix.compile.WhereCompiler.compile(WhereCompiler.java:108)
> at
> org.apache.phoenix.compile.QueryCompiler.compileSingleFlatQuery(QueryCompiler.java:629)
> at
> org.apache.phoenix.compile.QueryCompiler.compileSingleQuery(QueryCompiler.java:574)
> at
> org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:203)
> at
> org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157)
> at
> org.apache.phoenix.compile.QueryCompiler.compileSubquery(QueryCompiler.java:563)
> at
> org.apache.phoenix.compile.QueryCompiler.compileJoinQuery(QueryCompiler.java:239)
> at
> org.apache.phoenix.compile.QueryCompiler.compileJoinQuery(QueryCompiler.java:320)
> at
> org.apache.phoenix.compile.QueryCompiler.compileJoinQuery(QueryCompiler.java:252)
> at
> org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:201)
> at
> org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:497)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:1)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.compileQuery(PhoenixStatement.java:1769)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.compileQuery(PhoenixStatement.java:1762)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.optimizeQuery(PhoenixStatement.java:1756)
> {code}
> But other SQL engines like Spark SQL and MySQL all support Correlated In
> Subquery .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-6204) Provide a way to preserve HBase cell timestamps when running UPSERT SELECT statements
[ https://issues.apache.org/jira/browse/PHOENIX-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219358#comment-17219358 ] Geoffrey Jacoby commented on PHOENIX-6204: -- [~ckulkarni] - See also PHOENIX-5991 and discussion on https://github.com/apache/phoenix/pull/823 > Provide a way to preserve HBase cell timestamps when running UPSERT SELECT > statements > - > > Key: PHOENIX-6204 > URL: https://issues.apache.org/jira/browse/PHOENIX-6204 > Project: Phoenix > Issue Type: New Feature >Affects Versions: 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > > Today when we run an UPSERT SELECT statement, the data is upserted with the > current wall clock time rather than using the timestamp of the cells being > read via the SELECT statement. In some cases this is favorable, but in others > it is not. > Providing a way to do an UPSERT SELECT in which upserts use the HBase > timestamp of the cells being read is a useful feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6186) Store table metadata last modified timestamp in PTable / System.Catalog
[ https://issues.apache.org/jira/browse/PHOENIX-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214299#comment-17214299 ] Geoffrey Jacoby commented on PHOENIX-6186: -- [~kadir] suggested to me yesterday that the timestamp in PTable.getTimeStamp() could already function as this. After some digging, I don't think it can be used. PTable.getTimeStamp() doesn't come from a field in System.Catalog itself, but is created at PTable load time by taking the max of each Cell timestamp in a syscat row. While this would probably catch table creation, column addition, and column removal (because the column count will change on the header row), it would also catch changes to internal Phoenix properties like PHOENIX_TTL. The purpose of this JIRA is to identify the last time the metadata "shape" of a table, view, or index has changed in ways that would be relevant to an external application or change detection framework. Changing the existing behavior of PTable.getTimeStamp() to fit this invariant risks breaking other logic in Phoenix, such as the requirement that before a client can ALTER a table or view, its local copy of the metadata must be up to date. I believe, therefore, that this needs to be a new column (SCHEMA_TIMESTAMP? DDL_TIMESTAMP? Open to suggestions on name!), updated on CREATE, or an ALTER ADD/DROP COLUMN. The value would be the _server_ timestamp in MetadataEndpointImpl on the System.Catalog region that hosts the table or view parent row. I don't believe there needs to be any parent/child activity -- i.e, altering a view shouldn't affect the timestamp of its parent table or view, altering an index shouldn't affect the base table, etc. [~ChinmayKulkarni] [~yanxinyi] [~bharathv] [~kadir][~jainankit], fyi. Wdyt? > Store table metadata last modified timestamp in PTable / System.Catalog > --- > > Key: PHOENIX-6186 > URL: https://issues.apache.org/jira/browse/PHOENIX-6186 > Project: Phoenix > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 4.16.0 > > > There are many reasons why it's useful to know when a particular table's > metadata was last modified. It's helpful when solving cache coherency > problems, and also in order to interact with external schema registries which > may have multiple versions of a particular schema and require a timestamp to > resolve ambiguities. > This JIRA will add a last modified timestamp field to System.Catalog, to be > updated both when creating a table/view and also when adding or removing a > column. Changing purely internal Phoenix properties will not update the > timestamp. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6128) Remove unused getAdmin() call inside CQSI.init()
[ https://issues.apache.org/jira/browse/PHOENIX-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201655#comment-17201655 ] Geoffrey Jacoby commented on PHOENIX-6128: -- +1 > Remove unused getAdmin() call inside CQSI.init() > > > Key: PHOENIX-6128 > URL: https://issues.apache.org/jira/browse/PHOENIX-6128 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Richard Antal >Priority: Minor > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6128.4.x.v1.patch, PHOENIX-6128.master.v1.patch > > > A new HBase Admin object is created inside CQSI.init() > [here|https://github.com/apache/phoenix/blob/e3c7b4bdce2524eb4fd1e7eb0ccd3454fcca81ce/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java#L3150], > however this is unused. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6157) Fix dependecy:analyze error introduced by PHOENIX-5909
[ https://issues.apache.org/jira/browse/PHOENIX-6157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201597#comment-17201597 ] Geoffrey Jacoby commented on PHOENIX-6157: -- [~stoty] - thanks for the fix, and sorry for the trouble. 5909 had been sitting around for several months, and I rebased it but didn't think to check if it was introducing any new guava uses. When I checked that it compiled, I used mvn package instead of mvn install so it worked fine for me. Will keep an eye out for this in the future. > Fix dependecy:analyze error introduced by PHOENIX-5909 > -- > > Key: PHOENIX-6157 > URL: https://issues.apache.org/jira/browse/PHOENIX-6157 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 5.1.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Blocker > Fix For: 5.1.0 > > Attachments: PHOENIX-6157.master.00.patch > > Time Spent: 10m > Remaining Estimate: 0h > > PHOENIX-5909 adds an unshaded com.google.common import, which breaks mvn > verify. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6141) Ensure consistency between SYSTEM.CATALOG and SYSTEM.CHILD_LINK
[ https://issues.apache.org/jira/browse/PHOENIX-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197938#comment-17197938 ] Geoffrey Jacoby commented on PHOENIX-6141: -- Agree that this would be very good to get in 4.17, and with [~kozdemir] about using two-phase commit. > Ensure consistency between SYSTEM.CATALOG and SYSTEM.CHILD_LINK > --- > > Key: PHOENIX-6141 > URL: https://issues.apache.org/jira/browse/PHOENIX-6141 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Fix For: 4.17.0 > > > Before 4.15, "CREATE/DROP VIEW" was an atomic operation since we were issuing > batch mutations on just the 1 SYSTEM.CATALOG region. In 4.15 we introduced > SYSTEM.CHILD_LINK to store the parent->child links and so a CREATE VIEW is no > longer atomic since it consists of 2 separate RPCs (1 to SYSTEM.CHILD_LINK > to add the linking row and another to SYSTEM.CATALOG to write metadata for > the new view). > If the second RPC i.e. the RPC to write metadata to SYSTEM.CATALOG fails > after the 1st RPC has already gone through, there will be an inconsistency > between both metadata tables. We will see orphan parent->child linking rows > in SYSTEM.CHILD_LINK in this case. This can cause the following issues: > # ALTER TABLE calls on the base table will fail > # DROP TABLE without CASCADE will fail > # The upgrade path has calls like UpgradeUtil.upgradeTable() which will fail > # Any metadata consistency checks can be thrown off > # Unnecessary extra storage of orphan links > The first 3 issues happen because we wrongly deduce that a base table has > child views due to the orphan linking rows. > This Jira aims at trying to come up with a way to make mutations among > SYSTEM.CATALOG and SYSTEM.CHILD_LINK an atomic transaction. We can use a > 2-phase commit approach like in global indexing or also potentially explore > using a transaction manager. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5872) Close Internal Phoenix Connections that were running during cancel
[ https://issues.apache.org/jira/browse/PHOENIX-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196557#comment-17196557 ] Geoffrey Jacoby commented on PHOENIX-5872: -- [~dbwong] - I see that this patch was committed to 4.x but not master. Was this intentional? Should the JIRA be resolved or are we waiting for a master patch? > Close Internal Phoenix Connections that were running during cancel > -- > > Key: PHOENIX-5872 > URL: https://issues.apache.org/jira/browse/PHOENIX-5872 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.3, 4.x >Reporter: Daniel Wong >Assignee: Daniel Wong >Priority: Major > Fix For: 4.16.0 > > Attachments: PHOENIX-5872.4.x.patch, PHOENIX-5872.master.patch, > PHOENIX-5872v2.4.x.patch, PHOENIX-5872v3.4.x.patch, PHOENIX-5872v4.4.x.patch, > PHOENIX-5872v5.4.x.patch > > Time Spent: 6h 40m > Remaining Estimate: 0h > > 3 part approach: > 1 don't count internal phoenix connections toward the client limit. > 2 count internal phoenix connections toward a newly defined limit > 3 track parent and child relationships between connections to close those > connections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5287) Incorrect results for COUNT(1) on a table with GLOBAL INDEX
[ https://issues.apache.org/jira/browse/PHOENIX-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196455#comment-17196455 ] Geoffrey Jacoby commented on PHOENIX-5287: -- Yes, [~dbwong], this looks like the global index was out of sync on Table A and got back in sync after running a rebuild. Once 5.1 is released, [~vkrava4] should be able to upgrade the indexes on Table A to use the new index framework using the IndexUpgradeTool (which also triggers a one-time rebuild). After that the self-healing capabilities of the new indexing framework should keep things in sync. (The new consistent global indexes automatically detect at read-time if data for a given row is out of sync, and repairs the index row from the base table before returning results to the client.) > Incorrect results for COUNT(1) on a table with GLOBAL INDEX > --- > > Key: PHOENIX-5287 > URL: https://issues.apache.org/jira/browse/PHOENIX-5287 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.0.0 > Environment: Environment and data information: > * Column structure of TABLE_A is fully identical to TABLE_B > * TABLE_A has a GLOBAL INDEX > * TABLE_B has a LOCAL INDEX >Reporter: Vlad Krava >Priority: Blocker > > COUNT(\*\) and COUNT(1) commands display incorrect(outdated) statistics for > table with GLOBAL index. > *Example:* > * Export TABLE_A to SCV file (SELECT * FROM *POMG.TABLE_A*) > * Import CSV file to TABLE_B > * COUNT operation on 'TABLE_A' was constantly returning with an amount of > 218623 (for 2 days without any data modifications!!!) : > ** 0: *jdbc:phoenix:> select count(1) from POMG.TABLE_A*; > *** RESULT: 218623 > * Newly exported table from CSV file (TABLE_B) showed different (higher > amount of records) > ** 0: *jdbc:phoenix:> select count(1) from POMG.TABLE_B*; > *** RESULT: 218683 > * COUNT in Hbase is returning the bigger value than COUNT comparing to > Phoenix table ( 218683 vs 218623) > * Phoenix Statistics for this table was updated few times for the past few > testing days > * I took few attends to define data misalignments by executing diff for > primary keys: > ** select key_1 from *POMG.TABLE_A* where key_1 not in (select key_1 from > *POMG.TABLE_B*) - 0 records selected (_Doesn't make sense considering a fact > that TABLE_A larger than TABLE_B and key_1 is unique PRIMARY KEY_) > ** select key_1 from *POMG.TABLE_B* where key_1 not in (select key_1 from > *POMG.TABLE_A*) - 23 records selected (_Doesn't make sense considering a > fact that TABLE_A larger than TABLE_B and key_1 is unique PRIMARY KEY_) > *Workaround:* > * After executing ALTER INDEX with REBUILD flag COUNT statistics for TABLE_A > become identical to TABLE_B > * Diff selects didn't show any differences between *POMG.TABLE_A* and > *POMG.TABLE_B* > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6067) (5.x) Global Secondary Index Parity with 4.x
[ https://issues.apache.org/jira/browse/PHOENIX-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195643#comment-17195643 ] Geoffrey Jacoby commented on PHOENIX-6067: -- Thanks for the reviews, [~kadir], [~elserj] and [~larsh] > (5.x) Global Secondary Index Parity with 4.x > > > Key: PHOENIX-6067 > URL: https://issues.apache.org/jira/browse/PHOENIX-6067 > Project: Phoenix > Issue Type: Improvement >Reporter: Swaroopa Kadam >Assignee: Geoffrey Jacoby >Priority: Blocker > Fix For: 5.1.0 > > Attachments: PHOENIX-6067.v1.patch, PHOENIX-6067.v2.patch, > PHOENIX-6067.v3.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > A large number of indexing JIRAs were done for Phoenix 4.16 but were > originally unable to be ported to 5.x because of the lack of HBase 2.x > support for PHOENIX-5645, max lookback age. This was eventually fixed in > HBASE-24321 and PHOENIX-5881. Because some JIRAs later than the missing ones > _were_ ported to 5.x, applying them all one-by-one and testing each > intermediate version would be impractical. > This JIRA will import the 4.16 global index implementation into 5.1.0, then > fix HBase API usage to conform to HBase 2.x standards and Phoenix's HBase 2.x > compatibility shim between minor versions. (For example, max lookback age > features aren't supported in 2.1 and 2.2 because they lack HBASE-24321, and > incremental index validation will be deactivated when running against HBase > 2.1, because of the lack of HBASE-22710 to use raw Filters.) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6067) (5.x) Global Secondary Index Parity with 4.x
[ https://issues.apache.org/jira/browse/PHOENIX-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193923#comment-17193923 ] Geoffrey Jacoby commented on PHOENIX-6067: -- [~elserj] - PR is here: https://github.com/apache/phoenix/pull/871/files > (5.x) Global Secondary Index Parity with 4.x > > > Key: PHOENIX-6067 > URL: https://issues.apache.org/jira/browse/PHOENIX-6067 > Project: Phoenix > Issue Type: Improvement >Reporter: Swaroopa Kadam >Assignee: Geoffrey Jacoby >Priority: Blocker > Fix For: 5.1.0 > > Attachments: PHOENIX-6067.v1.patch, PHOENIX-6067.v2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > A large number of indexing JIRAs were done for Phoenix 4.16 but were > originally unable to be ported to 5.x because of the lack of HBase 2.x > support for PHOENIX-5645, max lookback age. This was eventually fixed in > HBASE-24321 and PHOENIX-5881. Because some JIRAs later than the missing ones > _were_ ported to 5.x, applying them all one-by-one and testing each > intermediate version would be impractical. > This JIRA will import the 4.16 global index implementation into 5.1.0, then > fix HBase API usage to conform to HBase 2.x standards and Phoenix's HBase 2.x > compatibility shim between minor versions. (For example, max lookback age > features aren't supported in 2.1 and 2.2 because they lack HBASE-24321, and > incremental index validation will be deactivated when running against HBase > 2.1, because of the lack of HBASE-22710 to use raw Filters.) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6067) (5.x) Global Secondary Index Parity with 4.x
[ https://issues.apache.org/jira/browse/PHOENIX-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193910#comment-17193910 ] Geoffrey Jacoby commented on PHOENIX-6067: -- [~elserj] - I think you just missed my push deactivating IndexToolForNonTxGlobalIndexIT. testEnableOutputLoggingForMaxLookback for 2.1 and 2.2. I've pushed up additional changes deactivating under 2.1 a lot of tests using raw filters in some way (either view index rebuilds or incremental index rebuilds), including the IndexUpgradeToolTest you found. Fixed the whitespace issue in TestUtil.dumpTable. Refactored htableFactory and indexHTable to be disposed of in the superclass close(); thanks for the suggestion. The familyMap potential NPE issues you raise look like valid concerns, but I'd rather tackle those as a separate JIRA because the same logic is in 4.x and would need to be fixed both places. As for a new VerifyType coming from a newer client, I believe best practice is to upgrade Phoenix server first and clients second (this is enforced in the code for minor and major version upgrades), and I don't see a new VerifyType coming in on a patch release. > (5.x) Global Secondary Index Parity with 4.x > > > Key: PHOENIX-6067 > URL: https://issues.apache.org/jira/browse/PHOENIX-6067 > Project: Phoenix > Issue Type: Improvement >Reporter: Swaroopa Kadam >Assignee: Geoffrey Jacoby >Priority: Blocker > Fix For: 5.1.0 > > Attachments: PHOENIX-6067.v1.patch, PHOENIX-6067.v2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > A large number of indexing JIRAs were done for Phoenix 4.16 but were > originally unable to be ported to 5.x because of the lack of HBase 2.x > support for PHOENIX-5645, max lookback age. This was eventually fixed in > HBASE-24321 and PHOENIX-5881. Because some JIRAs later than the missing ones > _were_ ported to 5.x, applying them all one-by-one and testing each > intermediate version would be impractical. > This JIRA will import the 4.16 global index implementation into 5.1.0, then > fix HBase API usage to conform to HBase 2.x standards and Phoenix's HBase 2.x > compatibility shim between minor versions. (For example, max lookback age > features aren't supported in 2.1 and 2.2 because they lack HBASE-24321, and > incremental index validation will be deactivated when running against HBase > 2.1, because of the lack of HBASE-22710 to use raw Filters.) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5712) Got SYSCAT ILLEGAL_DATA exception after created tenant index on view
[
https://issues.apache.org/jira/browse/PHOENIX-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190957#comment-17190957
]
Geoffrey Jacoby commented on PHOENIX-5712:
--
[~larsh] - that approach makes sense, and I could see it working for 4.14 ->
4.16 and 5.0 -> 5.1, but how to manage the fact that 4.15 is already out in the
wild with the current behavior?
> Got SYSCAT ILLEGAL_DATA exception after created tenant index on view
> -
>
> Key: PHOENIX-5712
> URL: https://issues.apache.org/jira/browse/PHOENIX-5712
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.15.0
>Reporter: Xinyi Yan
>Priority: Blocker
> Fix For: 5.1.0, 4.16.0
>
> Attachments: 5712-test.txt, t.txt
>
>
> repo
> //create a multi-tenant table on global connection
> CREATE TABLE A (TENANT_ID CHAR(15) NOT NULL, ID CHAR(3) NOT NULL, NUM BIGINT
> CONSTRAINT PK PRIMARY KEY (TENANT_ID, ID)) MULTI_TENANT = true;
> // create view and index on tenant connection
> CREATE VIEW A_VIEW AS SELECT * FROM A;
> UPSERT INTO A_VIEW (ID, NUM) VALUES ('A', 1);
> CREATE INDEX A_VIEW_INDEX ON A_VIEW (NUM DESC) INCLUDE (ID);
> // qeury data on global connection
> SELECT * RFOM SYSTEM.CATALOG;
> {code:java}
> Error: ERROR 201 (22000): Illegal data. Expected length of at least 8 bytes,
> but had 3 (state=22000,code=201)
> java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected length of at
> least 8 bytes, but had 3
> at
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:559)
> at
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:195)
> at
> org.apache.phoenix.schema.types.PDataType.checkForSufficientLength(PDataType.java:290)
> at
> org.apache.phoenix.schema.types.PLong$LongCodec.decodeLong(PLong.java:256)
> at org.apache.phoenix.schema.types.PLong.toObject(PLong.java:115)
> at org.apache.phoenix.schema.types.PLong.toObject(PLong.java:31)
> at
> org.apache.phoenix.schema.types.PDataType.toObject(PDataType.java:1011)
> at
> org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:75)
> at
> org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:585)
> at sqlline.Rows$Row.(Rows.java:258)
> at sqlline.BufferedRows.nextList(BufferedRows.java:111)
> at sqlline.BufferedRows.(BufferedRows.java:52)
> at sqlline.SqlLine.print(SqlLine.java:1623)
> at sqlline.Commands.execute(Commands.java:982)
> at sqlline.Commands.sql(Commands.java:906)
> at sqlline.SqlLine.dispatch(SqlLine.java:740)
> at sqlline.SqlLine.begin(SqlLine.java:557)
> at sqlline.SqlLine.start(SqlLine.java:270)
> at sqlline.SqlLine.main(SqlLine.java:201)
> {code}
> I tried to drop the view, and I was able to query the data from the SYSCATA.
> I tested on 4.x-HBase1.3 and master branch, all branches have the same
> behavior.
>
> cc [~kadir] [~gjacoby] [~swaroopa]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5712) Got SYSCAT ILLEGAL_DATA exception after created tenant index on view
[
https://issues.apache.org/jira/browse/PHOENIX-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190841#comment-17190841
]
Geoffrey Jacoby commented on PHOENIX-5712:
--
[~larsh][~m2je][~yanxinyi] - do we need a 4.14.4 (or 4.14.3.1) that adds
forward-compatibility to the 4.14 client for the 64-bit view index id feature?
Then users could upgrade their clients to 4.14.4 before upgrading their servers
to 4.15 or the forthcoming 4.16 and be OK. It'd be a pain (both for us to
resurrect an EOLed branch and for operators to need an extra upgrade step), but
it's the cleanest way I can think of to solve the problem.
Also note that there's a good chance that there are similar problems between
5.0 and 5.1, which is the HBase 2 equivalent of a 4.14 -> 4.16 upgrade.
(Haven't checked.)
> Got SYSCAT ILLEGAL_DATA exception after created tenant index on view
> -
>
> Key: PHOENIX-5712
> URL: https://issues.apache.org/jira/browse/PHOENIX-5712
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.15.0
>Reporter: Xinyi Yan
>Priority: Blocker
> Fix For: 5.1.0, 4.16.0
>
> Attachments: 5712-test.txt, t.txt
>
>
> repo
> //create a multi-tenant table on global connection
> CREATE TABLE A (TENANT_ID CHAR(15) NOT NULL, ID CHAR(3) NOT NULL, NUM BIGINT
> CONSTRAINT PK PRIMARY KEY (TENANT_ID, ID)) MULTI_TENANT = true;
> // create view and index on tenant connection
> CREATE VIEW A_VIEW AS SELECT * FROM A;
> UPSERT INTO A_VIEW (ID, NUM) VALUES ('A', 1);
> CREATE INDEX A_VIEW_INDEX ON A_VIEW (NUM DESC) INCLUDE (ID);
> // qeury data on global connection
> SELECT * RFOM SYSTEM.CATALOG;
> {code:java}
> Error: ERROR 201 (22000): Illegal data. Expected length of at least 8 bytes,
> but had 3 (state=22000,code=201)
> java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected length of at
> least 8 bytes, but had 3
> at
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:559)
> at
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:195)
> at
> org.apache.phoenix.schema.types.PDataType.checkForSufficientLength(PDataType.java:290)
> at
> org.apache.phoenix.schema.types.PLong$LongCodec.decodeLong(PLong.java:256)
> at org.apache.phoenix.schema.types.PLong.toObject(PLong.java:115)
> at org.apache.phoenix.schema.types.PLong.toObject(PLong.java:31)
> at
> org.apache.phoenix.schema.types.PDataType.toObject(PDataType.java:1011)
> at
> org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:75)
> at
> org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:585)
> at sqlline.Rows$Row.(Rows.java:258)
> at sqlline.BufferedRows.nextList(BufferedRows.java:111)
> at sqlline.BufferedRows.(BufferedRows.java:52)
> at sqlline.SqlLine.print(SqlLine.java:1623)
> at sqlline.Commands.execute(Commands.java:982)
> at sqlline.Commands.sql(Commands.java:906)
> at sqlline.SqlLine.dispatch(SqlLine.java:740)
> at sqlline.SqlLine.begin(SqlLine.java:557)
> at sqlline.SqlLine.start(SqlLine.java:270)
> at sqlline.SqlLine.main(SqlLine.java:201)
> {code}
> I tried to drop the view, and I was able to query the data from the SYSCATA.
> I tested on 4.x-HBase1.3 and master branch, all branches have the same
> behavior.
>
> cc [~kadir] [~gjacoby] [~swaroopa]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-6067) (5.x) Global Secondary Index Parity with 4.x
[ https://issues.apache.org/jira/browse/PHOENIX-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190822#comment-17190822 ] Geoffrey Jacoby commented on PHOENIX-6067: -- [~larsh] - yes, before starting on this I spoke with [~swaroopa] [~kozdemir] and [~giskender], and we all agreed that this was the best way to go. (I'd started off trying to do one JIRA at a time, but quickly found that doing just one of the large JIRAs, like 5734, was going to be almost as difficult as doing them all at once, and you'd have lots of intermediate states that had never existed in 4.x because the JIRAs were in different order over there. Plus some of the existing 5.x ports of 4.x JIRAs had subtle-but-significant differences whose assumptions weren't correct anymore after 5881.) Thanks for the JIRA hygiene and the list. Most on the list are included in this patch. A few are 4.x specific (5883, 5721), and one (5637) was obsoleted by a later JIRA (5785) that is included in this patch. Looks like there would be a few left that need to come over, but those can better be done separately, imo, so that we can keep this already huge patch just confined to indexing. > (5.x) Global Secondary Index Parity with 4.x > > > Key: PHOENIX-6067 > URL: https://issues.apache.org/jira/browse/PHOENIX-6067 > Project: Phoenix > Issue Type: Improvement >Reporter: Swaroopa Kadam >Assignee: Geoffrey Jacoby >Priority: Blocker > Fix For: 5.1.0 > > Attachments: PHOENIX-6067.v1.patch, PHOENIX-6067.v2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > A large number of indexing JIRAs were done for Phoenix 4.16 but were > originally unable to be ported to 5.x because of the lack of HBase 2.x > support for PHOENIX-5645, max lookback age. This was eventually fixed in > HBASE-24321 and PHOENIX-5881. Because some JIRAs later than the missing ones > _were_ ported to 5.x, applying them all one-by-one and testing each > intermediate version would be impractical. > This JIRA will import the 4.16 global index implementation into 5.1.0, then > fix HBase API usage to conform to HBase 2.x standards and Phoenix's HBase 2.x > compatibility shim between minor versions. (For example, max lookback age > features aren't supported in 2.1 and 2.2 because they lack HBASE-24321, and > incremental index validation will be deactivated when running against HBase > 2.1, because of the lack of HBASE-22710 to use raw Filters.) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6116) Deletes from immutable tables are not honored in local indexes.
[ https://issues.apache.org/jira/browse/PHOENIX-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188014#comment-17188014 ] Geoffrey Jacoby commented on PHOENIX-6116: -- Raising to blocker because whole-row deletes of immutable tables are allowed so it's a data integrity problem. Also surprised there's no test! > Deletes from immutable tables are not honored in local indexes. > --- > > Key: PHOENIX-6116 > URL: https://issues.apache.org/jira/browse/PHOENIX-6116 > Project: Phoenix > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Blocker > Fix For: 5.1.0, 4.16.0 > > > Just noticed that when I looked at PHOENIX-6115. > I'm surprised we do not have a test for that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6112) Coupling of two classes only use logger
[ https://issues.apache.org/jira/browse/PHOENIX-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187881#comment-17187881 ] Geoffrey Jacoby commented on PHOENIX-6112: -- Assuming tests pass, +1. Thanks for spotting that, [~wangchao316]. > Coupling of two classes only use logger > --- > > Key: PHOENIX-6112 > URL: https://issues.apache.org/jira/browse/PHOENIX-6112 > Project: Phoenix > Issue Type: Improvement > Components: core >Affects Versions: 4.x, master >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Minor > Attachments: PHOENIX-6112-4.x.patch, PHOENIX-6112.master.patch, > image-2020-08-28-14-48-34-990.png > > > PhoenixConfigurationUtil use BaseResultIterators.logger for print log. I > think this is inappropriate， Coupling of two classes. geeneral, we print log > for using local class. > !image-2020-08-28-14-48-34-990.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6110) Disable Permission ITs on HBase 2.1
[ https://issues.apache.org/jira/browse/PHOENIX-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186174#comment-17186174 ] Geoffrey Jacoby commented on PHOENIX-6110: -- +1 > Disable Permission ITs on HBase 2.1 > --- > > Key: PHOENIX-6110 > URL: https://issues.apache.org/jira/browse/PHOENIX-6110 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 5.1.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Attachments: PHOENIX-6110.master.v1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The permissions tests have been flakey on Hbase 2.1 ever since we started > supporting it. > As we do not have the problems on 2.2 or 2.3, I am reasonably confident that > the issue is that permission changes are not synchronous on 2.1 under load. > Since 2.1 is EOL, so there no hope for fixing this issue. Since we do not > want drop support support for 2.1 yet, (we may want to do so after 5.1 is > released), we should disable the permission-related flakey tests, so that our > test results stay relevant. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6105) Run precommit tests with HBase 2.2 on master
[ https://issues.apache.org/jira/browse/PHOENIX-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185377#comment-17185377 ] Geoffrey Jacoby commented on PHOENIX-6105: -- Which branch(es) to run on precommit is a tricky problem. After PHOENIX-5881 there are already several tests that check for CompatBaseScannerRegionObserver.isMaxLookbackSupported() and no-op if it's false, so to run the "full" test suite you need to run 2.3. Other tests do the same check and make changes to their logic or expected assertion values based on the results. As I port the rest of the 4.x indexing JIRAs over, there will be more of these. So in order to get "full" validation, you'd have to run under both 2.3, and one of 2.1 or 2.2. But that would probably be prohibitively time-consuming. > Run precommit tests with HBase 2.2 on master > > > Key: PHOENIX-6105 > URL: https://issues.apache.org/jira/browse/PHOENIX-6105 > Project: Phoenix > Issue Type: Task >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > > Since we have flakey tests that only fail with HBase 2.1, and HBase 2.3 > (different sets), set up the Precommit test job to run with HBase 2.2, as > that minimizes the number of false positive test failures on the precommit > job. > The afromentioned flakey tests are probably the results of HBase bugs, though > this warrants further investigation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5943) Consider splitting up SYSTEM.TASK into 2 tables - one for queued jobs and one for task history
[ https://issues.apache.org/jira/browse/PHOENIX-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184719#comment-17184719 ] Geoffrey Jacoby commented on PHOENIX-5943: -- [~ckulkarni] - because this would be a schema change to a system table, it can go in 4.16.0, or it can go in 4.17.0, but I don't think it can go in 4.16.1. > Consider splitting up SYSTEM.TASK into 2 tables - one for queued jobs and one > for task history > -- > > Key: PHOENIX-5943 > URL: https://issues.apache.org/jira/browse/PHOENIX-5943 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Fix For: 5.1.0, 4.16.1 > > > We can split SYSTEM.TASK into 2 tables: > # SYSTEM.TASK_QUEUE: used as an ephemeral task queue with potentially a > smaller TTL which contains tasks that are new and yet to be picked up. > # SYSTEM.TASK_HISTORY: used to store tasks that have already been picked up > and may or may not have succeeded, along with more data about the task (stack > trace, job IDs, etc.). This table can have a longer TTL and can be used for > auditing/debugging purposes. > The advantages of doing this are: > * Makes SYSTEM.TASK_QUEUE more lightweight. > * Less data scanned when picking up new tasks. > * Combined with a PK change, we can introduce a natural priority queue using > the TASK_PRIORITY field. > * Different TTL value on the TASK_HISTORY table so we can maintain past > tasks for a longer period of time. > * Potentially makes this more in-line with other use-cases that want to > maintain a similar kind of task and task history table (for ex: View TTL's > async delete functionality). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5881) Port MaxLookbackAge logic to 5.x
[ https://issues.apache.org/jira/browse/PHOENIX-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183649#comment-17183649 ] Geoffrey Jacoby commented on PHOENIX-5881: -- [~elserj] - it's a port of existing logic in 4.x from PHOENIX-4277. Tephra overloads the HBase timestamps to make them nanoseconds; the logic is basically saying "if the timestamp is super-high, that means Tephra's running this Scan, so don't mess with how HBase treats delete markers." > Port MaxLookbackAge logic to 5.x > > > Key: PHOENIX-5881 > URL: https://issues.apache.org/jira/browse/PHOENIX-5881 > Project: Phoenix > Issue Type: Improvement >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Blocker > Fix For: 5.1.0 > > Attachments: PHOENIX-5881.v1.patch, PHOENIX-5881.v2.patch, > PHOENIX-5881.v3.patch > > Time Spent: 2h 10m > Remaining Estimate: 0h > > PHOENIX-5645 wasn't included in the master (5.x) branch because an HBase 2.x > change prevented the logic from being useful in the case of deletes, since > HBase 2.x no longer allows us to show deleted cells on an SCN query before > the point of deletion. Unfortunately, PHOENIX-5645 wound up requiring a lot > of follow-up work in the IndexTool and IndexScrutinyTool to deal with its > implications, and because of that, the 4.x and 5.x codebases around indexes > have diverged a good bit. > This work item is to get them back in sync, even though the behavior in the > face of deletes will be somewhat different, and so most likely some tests > will have to be changed or Ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6069) We should check that the parent table key is in the region in the MetaDataEndpointImpl.dropTable code
[ https://issues.apache.org/jira/browse/PHOENIX-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183501#comment-17183501 ] Geoffrey Jacoby commented on PHOENIX-6069: -- +1 > We should check that the parent table key is in the region in the > MetaDataEndpointImpl.dropTable code > - > > Key: PHOENIX-6069 > URL: https://issues.apache.org/jira/browse/PHOENIX-6069 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Chinmay Kulkarni >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6069-4.x-v2.patch > > > The check > [here|https://github.com/apache/phoenix/blob/1d844950bb4ec8221873ecd2b094c20f427cd984/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L2231] > should check the parentLockKey instead of the lockKey for the current table > which we've already checked before this. Seems to be a copy-paste error. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6092) Optionally queue DDL requests issued while metadata upgrade is in progress and replay on upgrade failure
[ https://issues.apache.org/jira/browse/PHOENIX-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181471#comment-17181471 ] Geoffrey Jacoby commented on PHOENIX-6092: -- How long do the upgrades usually take? If it's fairly short, is just throwing an exception back to the client saying that DDL is not supported during upgrades sufficient? The WAL would certainly work, just trying to quantify how big a problem this. > Optionally queue DDL requests issued while metadata upgrade is in progress > and replay on upgrade failure > > > Key: PHOENIX-6092 > URL: https://issues.apache.org/jira/browse/PHOENIX-6092 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Chinmay Kulkarni >Priority: Critical > Fix For: 5.1.0, 4.16.0 > > > Currently, if a metadata upgrade is in progress (either triggered by an > explicit "EXECUTE UPGRADE" command or by a new client with autoUpgrade > enabled), in-flight DDLs will generally go through and work as expected. > However, if the upgrade happens to fail, we restore the snapshot of > SYSTEM.CATALOG (and with > [PHOENIX-6086|https://issues.apache.org/jira/browse/PHOENIX-6086] even other > SYSTEM tables) to represent its state before the upgrade started. Due to > this, any DDLs issued after the upgrade began are lost. > There are upgrade steps that need to iterate over each table/index/view in > the cluster and multiple steps that need full table scans on SYSTEM.CATALOG > and so this time window where we could potentially lose client DDLs is not > negligible (could be to the order of minutes). > This Jira is to discuss ways to tackle this problem. Perhaps we can introduce > a configuration which when enabled could use some sort of write-ahead log to > store DDLs issued while the upgrade is in progress and replay those DDLs in > case we need to restore SYSTEM tables from their snapshot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5860) Throw exception which region is closing or splitting when delete data
[
https://issues.apache.org/jira/browse/PHOENIX-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181284#comment-17181284
]
Geoffrey Jacoby commented on PHOENIX-5860:
--
[~wangchao316] - the HadoopQA test run shows an NPE coming from the new test
you wrote. Could you please take a look?
https://ci-hadoop.apache.org/job/PreCommit-PHOENIX-Build/24//testReport/
> Throw exception which region is closing or splitting when delete data
> -
>
> Key: PHOENIX-5860
> URL: https://issues.apache.org/jira/browse/PHOENIX-5860
> Project: Phoenix
> Issue Type: Bug
> Components: core
>Affects Versions: 4.13.1
>Reporter: Chao Wang
>Assignee: Chao Wang
>Priority: Blocker
> Fix For: 4.x
>
> Attachments: PHOENIX-5860-4.x.patch,
> PHOENIX-5860.4.13.x-HBASE.1.3.x.002.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently delete data is UngroupedAggregateRegionObserver class on server
> side, this class check if isRegionClosingOrSplitting is true. when
> isRegionClosingOrSplitting is true, will throw new IOException("Temporarily
> unable to write from scan because region is closing or splitting").
> when region online , which initialize phoenix CP that
> isRegionClosingOrSplitting is false.before region split, region change
> isRegionClosingOrSplitting to true.but if region split failed,split will roll
> back where not change isRegionClosingOrSplitting to false. after that all
> write opration will always throw exception which is Temporarily unable to
> write from scan because region is closing or splitting。
> so we should change isRegionClosingOrSplitting to false when region
> preRollBackSplit in UngroupedAggregateRegionObserver class。
> A simple test where a data table split failed, then roll back success.but
> delete data always throw exception.
> # create data table
> # bulkload data for this table
> # alter hbase-server code, which region split will throw exception , then
> rollback.
> # use hbase shell , split region
> # view regionserver log, where region split failed, and then rollback
> success.
> # user phoenix sqlline.py for delete data, which will throw exption
> Caused by: java.io.IOException: Temporarily unable to write from scan
> because region is closing or splitting Caused by: java.io.IOException:
> Temporarily unable to write from scan because region is closing or splitting
> at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostScannerOpen(UngroupedAggregateRegionObserver.java:516)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:245)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:293)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2881)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3082)
> ... 5 more
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:548)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
> at
> org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64)
> at
> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
> at
> org.apache.phoenix.compile.DeleteCompiler$2.execute(DeleteCompiler.java:498)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:303)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:295)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:293)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeUpdate(PhoenixPreparedStatement.java:200)
> at
> com.huawei.mds.apps.ecidRepeatProcess.EcidProcessCommon$$anonfun$40$$anonfun$apply$19.apply(EcidProcessCommon.scala:2253)
> at
> com.huawei.mds.apps.ecidRepeatProcess.EcidProcessCommon$$anonfun$40$$anonfun$apply$19.apply(EcidProcessCommon.scala:2249)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
> at
> com.huawei.mds.apps.ecidRepeatProcess.EcidProcessCommon$$anonfun$40.apply(EcidProcessCommon.scala:2249)
>
[jira] [Commented] (PHOENIX-6016) Support HBase 2.3.0
[ https://issues.apache.org/jira/browse/PHOENIX-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173206#comment-17173206 ] Geoffrey Jacoby commented on PHOENIX-6016: -- [~stoty] [~elserj], is there a trick to getting tests working with HBase 2.3? I see in the PR that you were able to get a mvn verify run to pass, but I can't. I'm trying to run them by setting -Dhbase.profile=2.3 and getting the Hadoop error in PHOENIX-5993. Switching to Hadoop 2.10 to work around PHOENIX-5993 results in a compilation error. > Support HBase 2.3.0 > --- > > Key: PHOENIX-6016 > URL: https://issues.apache.org/jira/browse/PHOENIX-6016 > Project: Phoenix > Issue Type: Improvement >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 5.1.0 > > Attachments: PHOENIX-6016.master.v1.patch, > PHOENIX-6016.master.v2.patch, PHOENIX-6016.master.v3.patch, > PHOENIX-6016.master.v4.patch, PHOENIX-6016.master.v5.patch > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Hbase 2.3.0 has been released. > Make sure Phoenix works with it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6058) When maxLookback is not enabled, IndexTool should not do deep verification
[ https://issues.apache.org/jira/browse/PHOENIX-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170336#comment-17170336 ] Geoffrey Jacoby commented on PHOENIX-6058: -- Thanks for filing this, [~kozdemir]. Marking this as a blocker for 5.1.0, since max lookback will not be available in HBase 2.1 and 2.2-compatible versions of it. > When maxLookback is not enabled, IndexTool should not do deep verification > -- > > Key: PHOENIX-6058 > URL: https://issues.apache.org/jira/browse/PHOENIX-6058 > Project: Phoenix > Issue Type: Bug >Reporter: Kadir OZDEMIR >Assignee: Kadir OZDEMIR >Priority: Blocker > Fix For: 5.1.0, 4.16.0 > > > IndexTool does two types of verification, verifying the latest row versions, > and verifying all row versions within the maxLookback windows. Verifying just > the latest version can be called shallow verification whereas verifying all > versions can be called deep verification. The maxLookback feature preserves > all row versions including delete markers within the maxLookback window. This > allows IndexTool to compare all versions of data rows with all versions of > the index rows within the window. However, due to a bug, when the maxLookback > feature is not enabled, i.e., the maxLookback window is set to zero, > IndexTool assumes the window size is infinite and thus does deep > verification. Since some versions of the index rows can be removed by > compaction independently if the maxLookback feature is not enabled, IndexTool > may report missing or invalid index rows by doing deep verification. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (PHOENIX-6055) Improve error reporting for index validation when there are "Not matching index rows"
[ https://issues.apache.org/jira/browse/PHOENIX-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated PHOENIX-6055: - Comment: was deleted (was: [~giskender] - Some operators may have legal or contractual restrictions on emitting production data into their logs, which this would be likely to do. ) > Improve error reporting for index validation when there are "Not matching > index rows" > - > > Key: PHOENIX-6055 > URL: https://issues.apache.org/jira/browse/PHOENIX-6055 > Project: Phoenix > Issue Type: Improvement >Reporter: Gokcen Iskender >Priority: Major > > In the IndexRebuildRegionScanner, in the following line we return the > timestamp of the latest expected mutation rather than the expected mutation > itself. > > byte[] dataKey = indexMaintainer.buildDataRowKey(new > ImmutableBytesWritable(indexRow.getRow()), viewConstants); > String errorMsg = "Not matching index row"; > logToIndexToolOutputTable(dataKey, indexRow.getRow(), > getTimestamp(expectedMutationList.get(0)), 0L, errorMsg, isBeforeRebuild, > INVALID_ROW); > > Instead of expectedMutationList.get(0), expected should be used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6055) Improve error reporting for index validation when there are "Not matching index rows"
[ https://issues.apache.org/jira/browse/PHOENIX-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168250#comment-17168250 ] Geoffrey Jacoby commented on PHOENIX-6055: -- [~giskender] - Some operators may have legal or contractual restrictions on emitting production data into their logs, which this would be likely to do. > Improve error reporting for index validation when there are "Not matching > index rows" > - > > Key: PHOENIX-6055 > URL: https://issues.apache.org/jira/browse/PHOENIX-6055 > Project: Phoenix > Issue Type: Improvement >Reporter: Gokcen Iskender >Priority: Major > > In the IndexRebuildRegionScanner, in the following line we return the > timestamp of the latest expected mutation rather than the expected mutation > itself. > > byte[] dataKey = indexMaintainer.buildDataRowKey(new > ImmutableBytesWritable(indexRow.getRow()), viewConstants); > String errorMsg = "Not matching index row"; > logToIndexToolOutputTable(dataKey, indexRow.getRow(), > getTimestamp(expectedMutationList.get(0)), 0L, errorMsg, isBeforeRebuild, > INVALID_ROW); > > Instead of expectedMutationList.get(0), expected should be used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6013) RVC Offset does not handle coerced literal nulls properly.
[ https://issues.apache.org/jira/browse/PHOENIX-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159384#comment-17159384 ] Geoffrey Jacoby commented on PHOENIX-6013: -- [~dbwong] - when you're happy with the state of your PR, could you please also post a patch to the JIRA and mark as Submit Patch so we can get a test run? > RVC Offset does not handle coerced literal nulls properly. > -- > > Key: PHOENIX-6013 > URL: https://issues.apache.org/jira/browse/PHOENIX-6013 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0 >Reporter: Daniel Wong >Assignee: Daniel Wong >Priority: Major > Fix For: 4.16.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > As part of query rewrite paths the offset may go through the A=null gets > rewritten to A IS NULL, the code sanity checks against an equality comparison > op which is not the case on this rewrite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6011) ServerCacheClient throw NullPointerException
[ https://issues.apache.org/jira/browse/PHOENIX-6011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159371#comment-17159371 ] Geoffrey Jacoby commented on PHOENIX-6011: -- [~wangchao316] - I'm having some trouble applying your patch using "git am" or "git apply" to either master or 4.x. 1. The paths in the patch seem to be one directory too far up (/phoenix/phoenix-core rather than /phoenix-core) and that's confusing git. Oddly Jenkins seems to have been fine with it. 2. Could you please reword the commit message to something like "PHOENIX-6011 - ServerCacheClient throws NullPointerException". Thanks! > ServerCacheClient throw NullPointerException > > > Key: PHOENIX-6011 > URL: https://issues.apache.org/jira/browse/PHOENIX-6011 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.1 >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Major > Attachments: 1.png, PHOENIX-6011-v1.patch, PHOENIX-6011-v2.patch > > > Throw NullPointerException in my product Environment now, As shown in the > following image. > !1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6011) ServerCacheClient throw NullPointerException
[ https://issues.apache.org/jira/browse/PHOENIX-6011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159364#comment-17159364 ] Geoffrey Jacoby commented on PHOENIX-6011: -- +1, thanks [~wangchao316] > ServerCacheClient throw NullPointerException > > > Key: PHOENIX-6011 > URL: https://issues.apache.org/jira/browse/PHOENIX-6011 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.1 >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Major > Attachments: 1.png, PHOENIX-6011-v1.patch, PHOENIX-6011-v2.patch > > > Throw NullPointerException in my product Environment now, As shown in the > following image. > !1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6000) Client side DELETEs should use local indexes for filtering
[ https://issues.apache.org/jira/browse/PHOENIX-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159361#comment-17159361 ] Geoffrey Jacoby commented on PHOENIX-6000: -- +1, thanks [~larsh] > Client side DELETEs should use local indexes for filtering > -- > > Key: PHOENIX-6000 > URL: https://issues.apache.org/jira/browse/PHOENIX-6000 > Project: Phoenix > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: 6000-4.x-HBase-1.5-v2.txt, 6000-4.x-HBase-1.5.txt, > 6000-4.x-v3.txt, 6000-4.x-v4.txt > > > I just noticed that client side DELETEs do not use local indexes for > filtering if they do not cover all keys of all other indexes. > Unless I am missing something this is not necessary, for local indexes we > have the data available in the region and can fill back the uncovered column > (just like we do for SELECTs) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6011) ServerCacheClient throw NullPointerException
[ https://issues.apache.org/jira/browse/PHOENIX-6011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158540#comment-17158540 ] Geoffrey Jacoby commented on PHOENIX-6011: -- Thanks for the patch, [~wangchao316]. Could you please add an Apache 2.0 License to the new test file? Otherwise that looks good. > ServerCacheClient throw NullPointerException > > > Key: PHOENIX-6011 > URL: https://issues.apache.org/jira/browse/PHOENIX-6011 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.1 >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Major > Attachments: 1.png, PHOENIX-6011-v1.patch > > > Throw NullPointerException in my product Environment now, As shown in the > following image. > !1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5997) Phoenix Explain Plan for Deletes does not clearly differentiate between server side and client side paths.
[ https://issues.apache.org/jira/browse/PHOENIX-5997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158533#comment-17158533 ] Geoffrey Jacoby commented on PHOENIX-5997: -- [~dbwong] - could you please make a patch for master as well when you get a minute, so we can keep them (mostly) in sync? > Phoenix Explain Plan for Deletes does not clearly differentiate between > server side and client side paths. > -- > > Key: PHOENIX-5997 > URL: https://issues.apache.org/jira/browse/PHOENIX-5997 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0, 4.16.0 >Reporter: Daniel Wong >Assignee: Daniel Wong >Priority: Minor > Fix For: 4.16.0 > > Attachments: PHOENIX-5997-v2-4.x.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Phoenix Explain Plan for Deletes does not clearly differentiate between > server side and client side paths. > Currently a server side and client side delete explain plans look like: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6000) Client side DELETEs should use local indexes for filtering
[ https://issues.apache.org/jira/browse/PHOENIX-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158530#comment-17158530 ] Geoffrey Jacoby commented on PHOENIX-6000: -- [~larsh] - It looks like your commit ran against 4.x-HBase-1.5, which is a dead branch. All 4.x development is now done in the "4.x" branch. Also, I think your test will break after the explain plan changes in [~dbwong]'s PHOENIX-5997, which were recently committed. > Client side DELETEs should use local indexes for filtering > -- > > Key: PHOENIX-6000 > URL: https://issues.apache.org/jira/browse/PHOENIX-6000 > Project: Phoenix > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: 6000-4.x-HBase-1.5-v2.txt, 6000-4.x-HBase-1.5.txt > > > I just noticed that client side DELETEs do not use local indexes for > filtering if they do not cover all keys of all other indexes. > Unless I am missing something this is not necessary, for local indexes we > have the data available in the region and can fill back the uncovered column > (just like we do for SELECTs) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-6000) Client side DELETEs should use local indexes for filtering
[ https://issues.apache.org/jira/browse/PHOENIX-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155705#comment-17155705 ] Geoffrey Jacoby commented on PHOENIX-6000: -- [~larsh] - I can't think of any reason why an uncovered local index would be bad for a DELETE given the region locality inherent in locals. Curious why you're removing the check for allowServerMutations, however. Also please see the conversation [~dbwong] and I are having over on PHOENIX-5990. In general I'm unsure why the SELECT portion of DELETEs are getting their own bespoke query planning logic (so that some hints work and others don't, for example) rather than just re-using the select logic. > Client side DELETEs should use local indexes for filtering > -- > > Key: PHOENIX-6000 > URL: https://issues.apache.org/jira/browse/PHOENIX-6000 > Project: Phoenix > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Major > Attachments: 6000-4.x-HBase-1.5.txt > > > I just noticed that client side DELETEs do not use local indexes for > filtering if they do not cover all keys of all other indexes. > Unless I am missing something this is not necessary, for local indexes we > have the data available in the region and can fill back the uncovered column > (just like we do for SELECTs) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5990) Delete statements don't honor index hints
[ https://issues.apache.org/jira/browse/PHOENIX-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155090#comment-17155090 ] Geoffrey Jacoby commented on PHOENIX-5990: -- [~dbwong] - is there a reason why the query plan generation (including index hints) should be different for the SELECT portion of a delete from a standalone SELECT? You know the client-side logic better than I do, but my expectation had been that they'd be the same, and I was surprised when they weren't. > Delete statements don't honor index hints > - > > Key: PHOENIX-5990 > URL: https://issues.apache.org/jira/browse/PHOENIX-5990 > Project: Phoenix > Issue Type: Bug >Reporter: Geoffrey Jacoby >Priority: Major > > While investigating a perf issue with a delete statement, we noticed that the > query optimizer wasn't using an index with a better (though not complete) > match for the WHERE clause than the base table PK was. Even using a query > hint to specify the index did not seem to change the EXPLAIN plan. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5973) IndexToolForNonTxGlobalIndexIT - Stabilize and speed up
[ https://issues.apache.org/jira/browse/PHOENIX-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149135#comment-17149135 ] Geoffrey Jacoby commented on PHOENIX-5973: -- [~RichardAntal] - Thanks! Yes, working on a port to master. Hope to get a master patch up sometime this week. The indexing code is significantly different between 4.x and master, because HBase 2.x tightened up its coproc interfaces in a way that blocks us from forward-porting PHOENIX-5645 until HBase 2.3 comes out. So whenever we forward port, lots of subtle tweaks to the code and tests are necessary to account for no protection from major compaction during index verification. (This is why in general the indexing code in the master branch lags 4.x, whereas in most non-indexing patches 4.x and master get committed at about the same time.) > IndexToolForNonTxGlobalIndexIT - Stabilize and speed up > --- > > Key: PHOENIX-5973 > URL: https://issues.apache.org/jira/browse/PHOENIX-5973 > Project: Phoenix > Issue Type: Test >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Attachments: PHOENIX-5973-4.x.v1.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > The many IndexTool test suites have clearly had some bad performance > regressions recently, which is a part of why the overall Phoenix pre commit > builds are timing out. We should investigate why and fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5935) Select with non primary keys and PHOENIX_ROW_TIMESTAMP() in where clause fails
[ https://issues.apache.org/jira/browse/PHOENIX-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146646#comment-17146646 ] Geoffrey Jacoby commented on PHOENIX-5935: -- The one test failure appears to be a minicluster startup issue. > Select with non primary keys and PHOENIX_ROW_TIMESTAMP() in where clause fails > -- > > Key: PHOENIX-5935 > URL: https://issues.apache.org/jira/browse/PHOENIX-5935 > Project: Phoenix > Issue Type: Bug >Reporter: Jacob Isaac >Assignee: Jacob Isaac >Priority: Major > Fix For: 4.x > > Attachments: PHOENIX-5935.4.x.002.patch > > > This fails when COLUMN_ENCODED_BYTES != NON_ENCODED_QUALIFIERS > Steps to reproduce:- > CREATE TABLE IF NOT EXISTS N01 (PK1 INTEGER NOT NULL, PK2 DATE NOT NULL, > KV1 VARCHAR, KV2 VARCHAR CONSTRAINT PK PRIMARY KEY(PK1, PK2)) > COLUMN_ENCODED_BYTES = 1,IMMUTABLE_STORAGE_SCHEME = ONE_CELL_PER_COLUMN > > SELECT COUNT(*) FROM N01 WHERE PHOENIX_ROW_TIMESTAMP() > PK2 AND KV1 = > 'KV1_1'"; > > Fails with the following exception - > Caused by: java.util.NoSuchElementExceptionCaused by: > java.util.NoSuchElementException at > org.apache.phoenix.filter.MultiEncodedCQKeyValueComparisonFilter$FilteredKeyValueHolder.getCellAtIndex(MultiEncodedCQKeyValueComparisonFilter.java:151) > at > org.apache.phoenix.filter.MultiEncodedCQKeyValueComparisonFilter$EncodedCQIncrementalResultTuple.getValue(MultiEncodedCQKeyValueComparisonFilter.java:311) > at > org.apache.phoenix.expression.function.PhoenixRowTimestampFunction.evaluate(PhoenixRowTimestampFunction.java:98) > at > org.apache.phoenix.expression.ComparisonExpression.evaluate(ComparisonExpression.java:330) > at > org.apache.phoenix.expression.AndOrExpression.evaluate(AndOrExpression.java:75) > at > org.apache.phoenix.filter.BooleanExpressionFilter.evaluate(BooleanExpressionFilter.java:93) > at > org.apache.phoenix.filter.MultiEncodedCQKeyValueComparisonFilter.filterKeyValue(MultiEncodedCQKeyValueComparisonFilter.java:233) > at > org.apache.hadoop.hbase.regionserver.querymatcher.UserScanQueryMatcher.matchColumn(UserScanQueryMatcher.java:122) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5962) Stabilize builds
[ https://issues.apache.org/jira/browse/PHOENIX-5962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136812#comment-17136812 ] Geoffrey Jacoby commented on PHOENIX-5962: -- I have no objection to putting a timeout on that IndexVerificationOutputRepositoryIT test, but it's puzzling that it would need one -- it's just creating an HBase (not Phoenix) table if it doesn't exist, writing 2 rows, then reading them back with narrow range scans. If it's consistently timing out, more investigation is needed as to why. Maybe a minicluster issue? > Stabilize builds > > > Key: PHOENIX-5962 > URL: https://issues.apache.org/jira/browse/PHOENIX-5962 > Project: Phoenix > Issue Type: Task >Reporter: Richard Antal >Assignee: Richard Antal >Priority: Major > Attachments: PHOENIX-5962.master.v1.patch, > PHOENIX-5962.master.v2.patch > > > Last 2 build was aborted because Build timed out (after 360 minutes). > Search the possible cause of the problem and try to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5712) Got SYSCAT ILLEGAL_DATA exception after created tenant index on view
[
https://issues.apache.org/jira/browse/PHOENIX-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136225#comment-17136225
]
Geoffrey Jacoby commented on PHOENIX-5712:
--
Just hunting for reasons it might be using the wrong codec.
> Got SYSCAT ILLEGAL_DATA exception after created tenant index on view
> -
>
> Key: PHOENIX-5712
> URL: https://issues.apache.org/jira/browse/PHOENIX-5712
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.15.0
>Reporter: Xinyi Yan
>Priority: Major
> Fix For: 4.16.0
>
>
> repo
> //create a multi-tenant table on global connection
> CREATE TABLE A (TENANT_ID CHAR(15) NOT NULL, ID CHAR(3) NOT NULL, NUM BIGINT
> CONSTRAINT PK PRIMARY KEY (TENANT_ID, ID)) MULTI_TENANT = true;
> // create view and index on tenant connection
> CREATE VIEW A_VIEW AS SELECT * FROM A;
> UPSERT INTO A_VIEW (ID, NUM) VALUES ('A', 1);
> CREATE INDEX A_VIEW_INDEX ON A_VIEW (NUM DESC) INCLUDE (ID);
> // qeury data on global connection
> SELECT * RFOM SYSTEM.CATALOG;
> {code:java}
> Error: ERROR 201 (22000): Illegal data. Expected length of at least 8 bytes,
> but had 3 (state=22000,code=201)
> java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected length of at
> least 8 bytes, but had 3
> at
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:559)
> at
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:195)
> at
> org.apache.phoenix.schema.types.PDataType.checkForSufficientLength(PDataType.java:290)
> at
> org.apache.phoenix.schema.types.PLong$LongCodec.decodeLong(PLong.java:256)
> at org.apache.phoenix.schema.types.PLong.toObject(PLong.java:115)
> at org.apache.phoenix.schema.types.PLong.toObject(PLong.java:31)
> at
> org.apache.phoenix.schema.types.PDataType.toObject(PDataType.java:1011)
> at
> org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:75)
> at
> org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:585)
> at sqlline.Rows$Row.(Rows.java:258)
> at sqlline.BufferedRows.nextList(BufferedRows.java:111)
> at sqlline.BufferedRows.(BufferedRows.java:52)
> at sqlline.SqlLine.print(SqlLine.java:1623)
> at sqlline.Commands.execute(Commands.java:982)
> at sqlline.Commands.sql(Commands.java:906)
> at sqlline.SqlLine.dispatch(SqlLine.java:740)
> at sqlline.SqlLine.begin(SqlLine.java:557)
> at sqlline.SqlLine.start(SqlLine.java:270)
> at sqlline.SqlLine.main(SqlLine.java:201)
> {code}
> I tried to drop the view, and I was able to query the data from the SYSCATA.
> I tested on 4.x-HBase1.3 and master branch, all branches have the same
> behavior.
>
> cc [~kadir] [~gjacoby] [~swaroopa]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5712) Got SYSCAT ILLEGAL_DATA exception after created tenant index on view
[
https://issues.apache.org/jira/browse/PHOENIX-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136219#comment-17136219
]
Geoffrey Jacoby commented on PHOENIX-5712:
--
As you say, [~ckulkarni], sounds like a lingering issue with PHOENIX-3547,
which switched the default size of a view index id from a Short to a Long. I
made a change at about the same time to change the way view index ids were
generated (so that views owned by different tenants didn't clash), but that
doesn't sound related, at least at first glance.
Is the column encoding different between the data table and indexes maybe?
[~m2je] and [~larsh] , fyi as well.
> Got SYSCAT ILLEGAL_DATA exception after created tenant index on view
> -
>
> Key: PHOENIX-5712
> URL: https://issues.apache.org/jira/browse/PHOENIX-5712
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.15.0
>Reporter: Xinyi Yan
>Priority: Major
> Fix For: 4.16.0
>
>
> repo
> //create a multi-tenant table on global connection
> CREATE TABLE A (TENANT_ID CHAR(15) NOT NULL, ID CHAR(3) NOT NULL, NUM BIGINT
> CONSTRAINT PK PRIMARY KEY (TENANT_ID, ID)) MULTI_TENANT = true;
> // create view and index on tenant connection
> CREATE VIEW A_VIEW AS SELECT * FROM A;
> UPSERT INTO A_VIEW (ID, NUM) VALUES ('A', 1);
> CREATE INDEX A_VIEW_INDEX ON A_VIEW (NUM DESC) INCLUDE (ID);
> // qeury data on global connection
> SELECT * RFOM SYSTEM.CATALOG;
> {code:java}
> Error: ERROR 201 (22000): Illegal data. Expected length of at least 8 bytes,
> but had 3 (state=22000,code=201)
> java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected length of at
> least 8 bytes, but had 3
> at
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:559)
> at
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:195)
> at
> org.apache.phoenix.schema.types.PDataType.checkForSufficientLength(PDataType.java:290)
> at
> org.apache.phoenix.schema.types.PLong$LongCodec.decodeLong(PLong.java:256)
> at org.apache.phoenix.schema.types.PLong.toObject(PLong.java:115)
> at org.apache.phoenix.schema.types.PLong.toObject(PLong.java:31)
> at
> org.apache.phoenix.schema.types.PDataType.toObject(PDataType.java:1011)
> at
> org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:75)
> at
> org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:585)
> at sqlline.Rows$Row.(Rows.java:258)
> at sqlline.BufferedRows.nextList(BufferedRows.java:111)
> at sqlline.BufferedRows.(BufferedRows.java:52)
> at sqlline.SqlLine.print(SqlLine.java:1623)
> at sqlline.Commands.execute(Commands.java:982)
> at sqlline.Commands.sql(Commands.java:906)
> at sqlline.SqlLine.dispatch(SqlLine.java:740)
> at sqlline.SqlLine.begin(SqlLine.java:557)
> at sqlline.SqlLine.start(SqlLine.java:270)
> at sqlline.SqlLine.main(SqlLine.java:201)
> {code}
> I tried to drop the view, and I was able to query the data from the SYSCATA.
> I tested on 4.x-HBase1.3 and master branch, all branches have the same
> behavior.
>
> cc [~kadir] [~gjacoby] [~swaroopa]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5951) IndexTool output logging for past-max-lookback rows should be configurable
[
https://issues.apache.org/jira/browse/PHOENIX-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134423#comment-17134423
]
Geoffrey Jacoby commented on PHOENIX-5951:
--
HadoopQA seems to have crashed midway through the run (in SplitSystemCatalog
tests) but the following were the results up to that point. None seem related
to this patch, and many are common flappers. The OrphanViewToolIT failures seem
to be happening consistently on several runs and are new; I will file a JIRA
for those.
{code:java}
17:40:24 [ERROR] Failures:
17:40:24 [ERROR] OrphanViewToolIT.testDeleteBaseTableRows:279
17:40:24 [ERROR] OrphanViewToolIT.testDeleteBaseTableRows:279
17:40:24 [ERROR]
OrphanViewToolIT.testDeleteChildParentLinkRows:402->verifyOrphanFileLineCounts:255->verifyLineCount:209
17:40:24 [ERROR]
OrphanViewToolIT.testDeleteChildParentLinkRows:402->verifyOrphanFileLineCounts:255->verifyLineCount:209
17:40:24 [ERROR]
OrphanViewToolIT.testDeleteChildViewRows:315->verifyOrphanFileLineCounts:255->verifyLineCount:209
17:40:24 [ERROR]
OrphanViewToolIT.testDeleteChildViewRows:315->verifyOrphanFileLineCounts:255->verifyLineCount:209
17:40:24 [ERROR]
OrphanViewToolIT.testDeleteGrandchildViewRows:344->verifyOrphanFileLineCounts:256->verifyLineCount:209
17:40:24 [ERROR]
OrphanViewToolIT.testDeleteGrandchildViewRows:344->verifyOrphanFileLineCounts:255->verifyLineCount:209
17:40:24 [ERROR]
OrphanViewToolIT.testDeleteParentChildLinkRows:374->verifyOrphanFileLineCounts:255->verifyLineCount:209
17:40:24 [ERROR]
OrphanViewToolIT.testDeleteParentChildLinkRows:374->verifyOrphanFileLineCounts:255->verifyLineCount:209
17:40:24 [ERROR]
OrphanViewToolIT.testDeletePhysicalTableLinks:429->verifyLineCount:209
17:40:24 [ERROR]
OrphanViewToolIT.testDeletePhysicalTableLinks:424->verifyCountQuery:218
17:40:24 [ERROR]
LocalImmutableNonTxIndexIT>BaseIndexIT.testCreateIndexAfterUpsertStarted:263->BaseIndexIT.testCreateIndexAfterUpsertStarted:338
expected:<4> but was:<3>
17:40:24 [ERROR]
LocalImmutableNonTxIndexIT>BaseIndexIT.testCreateIndexAfterUpsertStarted:263->BaseIndexIT.testCreateIndexAfterUpsertStarted:338
expected:<4> but was:<3>
17:40:24 [ERROR] Errors:
17:40:24 [ERROR]
OrphanViewToolIT.testCreateTableAndViews:231->verifyOrphanFileLineCounts:255->verifyLineCount:203
» FileNotFound
17:40:24 [ERROR]
MutableIndexIT>ParallelStatsDisabledIT.doSetup:60->BaseTest.setUpTestDriver:516->BaseTest.setUpTestDriver:521->BaseTest.checkClusterInitialized:435->BaseTest.setUpTestCluster:449->BaseTest.initMiniCluster:550
» Runtime
17:40:24 [ERROR]
HashJoinGlobalIndexIT>ParallelStatsDisabledIT.doSetup:60->BaseTest.setUpTestDriver:516->BaseTest.setUpTestDriver:521->BaseTest.checkClusterInitialized:435->BaseTest.setUpTestCluster:449->BaseTest.initMiniCluster:550
» Runtime
17:40:24 [ERROR]
HashJoinNoIndexIT>ParallelStatsDisabledIT.doSetup:60->BaseTest.setUpTestDriver:516->BaseTest.setUpTestDriver:521->BaseTest.checkClusterInitialized:435->BaseTest.setUpTestCluster:449->BaseTest.initMiniCluster:550
» Runtime
17:40:24 [ERROR]
SortMergeJoinLocalIndexIT>ParallelStatsDisabledIT.doSetup:60->BaseTest.setUpTestDriver:516->BaseTest.setUpTestDriver:521->BaseTest.checkClusterInitialized:435->BaseTest.setUpTestCluster:449->BaseTest.initMiniCluster:550
» Runtime
17:40:24 [ERROR]
WhereOptimizerForJoinFiltersIT>ParallelStatsDisabledIT.doSetup:60->BaseTest.setUpTestDriver:516->BaseTest.setUpTestDriver:521->BaseTest.checkClusterInitialized:435->BaseTest.setUpTestCluster:449->BaseTest.initMiniCluster:550
» Runtime
17:40:24 [ERROR]
ChunkedResultIteratorIT>ParallelStatsDisabledIT.doSetup:60->BaseTest.setUpTestDriver:516->BaseTest.setUpTestDriver:521->BaseTest.checkClusterInitialized:435->BaseTest.setUpTestCluster:449->BaseTest.initMiniCluster:550
» Runtime
17:40:24 [ERROR]
ParameterizedTransactionIT>ParallelStatsDisabledIT.doSetup:60->BaseTest.setUpTestDriver:516->BaseTest.setUpTestDriver:521->BaseTest.checkClusterInitialized:435->BaseTest.setUpTestCluster:449->BaseTest.initMiniCluster:550
» Runtime
17:40:24 [INFO]
17:40:24 [ERROR] Tests run: 3659, Failures: 14, Errors: 8, Skipped: 3
{code}
> IndexTool output logging for past-max-lookback rows should be configurable
> --
>
> Key: PHOENIX-5951
> URL: https://issues.apache.org/jira/browse/PHOENIX-5951
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5951-4.x.v1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> On a global mutable table with lots of writes, it can be possible for there
> to be many rows flagged by index verification as invalid that are older than
> the max lookback age. These are likely due
[jira] [Commented] (PHOENIX-5875) Optional logging for IndexTool verification
[ https://issues.apache.org/jira/browse/PHOENIX-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134420#comment-17134420 ] Geoffrey Jacoby commented on PHOENIX-5875: -- The test run seems to have crashed but had gotten most of the way done before it did (it was in the splittable syscat tests), including having already run the index tests. The release audit flag is for a test not related to this patch. > Optional logging for IndexTool verification > --- > > Key: PHOENIX-5875 > URL: https://issues.apache.org/jira/browse/PHOENIX-5875 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.14.3 >Reporter: Kadir OZDEMIR >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5875-4.x.v1.patch, PHOENIX-5875-4.x.v2.patch, > PHOENIX-5875-4.x.v3.patch, PHOENIX-5875.v1.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > IndexTool logs information about each row that has a verification failure. > The amount of logging can be excessive for large index tables. Also for some > runs, we may want to just do incremental repair, i.e., repair only the rows > that are broken, using -v BOTH or -v BEFORE, but not interested in the > details of how they are broken. Thus, IndexTool should have an option to > disable logging, something like, -dl (--disable-logging) BEFORE | AFTER | > BOTH where BEFORE, AFTER and BOTH indicate at the which phase (before or > after index repair or a both phases ) logging is disabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5942) ParameterizedIndexUpgradeIT is too slow
[ https://issues.apache.org/jira/browse/PHOENIX-5942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133573#comment-17133573 ] Geoffrey Jacoby commented on PHOENIX-5942: -- Note: the Hudson error is because of a minicluster spin-up error on AlterIndexIT, and not related to this JIRA. > ParameterizedIndexUpgradeIT is too slow > --- > > Key: PHOENIX-5942 > URL: https://issues.apache.org/jira/browse/PHOENIX-5942 > Project: Phoenix > Issue Type: Bug >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Minor > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5942-4.x.v1.patch > > Time Spent: 1h > Remaining Estimate: 0h > > The test creates the same tables and views for each test case, drops them > between, and recreates them. In addition each test case is doing index > upgrades / downgrades which requires disabling tables and rebuilding them. > Plus there are a lot of test cases, because of the cross-product of the > parameters. > According to [~achouhan], this is causing some builds to fail with timeouts, > and taking 20-30 min for me locally the last time I needed to add a test > case. Looks like it's time to refactor this test so it goes faster. > I suspect just having each test case create its own test-specific tables (so > no more dropping) would help, but that may not be enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5950) View With Where Clause On A Table With Composite Key Should Be Able To Optimize Queries
[
https://issues.apache.org/jira/browse/PHOENIX-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132819#comment-17132819
]
Geoffrey Jacoby commented on PHOENIX-5950:
--
Thanks for researching this [~m2je]. I've definitely also seen poor performance
on IndexScrutiny runs of views and your proposed solution makes sense.
> View With Where Clause On A Table With Composite Key Should Be Able To
> Optimize Queries
>
>
> Key: PHOENIX-5950
> URL: https://issues.apache.org/jira/browse/PHOENIX-5950
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.14.3
>Reporter: Mehdi Salarkia
>Assignee: Mehdi Salarkia
>Priority: Major
>
> For a table with a composite primary
> {code:java}
> CREATE TABLE MY_TABLE (K1 INTEGER NOT NULL, K2 VARCHAR NOT NULL,K3 INTEGER
> NOT NULL, V1 DECIMAL, CONSTRAINT pk PRIMARY KEY (K1, K2, K3))
> {code}
> when a view is created that has some (and not all) of primary key columns
> {code:java}
> CREATE VIEW MY_VIEW(v2 VARCHAR, V3 VARCHAR ) AS SELECT * FROM MY_TABLE WHERE
> K2 = 'A'
> {code}
> if you run a query on the view without providing all the primary columns
> {code:java}
> EXPLAIN SELECT K1, K2, K3, V1 FROM MY_VIEW WHERE (K1,K3) IN ((1,2),(3,4));
> ++-++--+
> | PLAN
>| EST_BYTES_READ | EST_ROWS_READ | EST_INFO_TS |
> ++-++--+
> | CLIENT 1-CHUNK PARALLEL 1-WAY ROUND ROBIN SKIP SCAN ON 2 KEYS OVER MY_TABLE
> [1,'A'] - [3,'A'] | null| null | null |
> | SERVER FILTER BY (K1, K3) IN
> ([128,0,0,1,128,0,0,2],[128,0,0,3,128,0,0,4]) | null
> | null | null |
> ++-++--+
> 2 rows selected (0.047 seconds)
> {code}
> the query generated is scan rather than a point look up
> same query on the parent table (with all the keys) looks like this
> {code:java}
> EXPLAIN SELECT K1, K2, K3, V1 FROM MY_TABLE WHERE (K1,K2,K3) IN
> ((1,'A',2),(3,'A',4));
> +--+-++--+
> | PLAN
> | EST_BYTES_READ | EST_ROWS_READ | EST_INFO_TS |
> +--+-++--+
> | CLIENT 1-CHUNK 2 ROWS 268 BYTES PARALLEL 1-WAY ROUND ROBIN POINT LOOKUP ON
> 2 KEYS OVER MY_TABLE | 268 | 2 | 0|
> +--+-++--+
> 1 row selected (0.025 seconds)
> {code}
> The issue is view condition is always added as `AND` to user provided where
> clause and in this case query optimizer is failing to optimize this query to
> a point look up.
>
> -- *[Affected
> Use Case]*
>
> The impact of this issue is most visible when you try to run IndexScrutiny
> tool on a view with an index which generates queries like:
> {code:java}
> SELECT /*+ NO_INDEX */ CAST("K1" AS INTEGER),CAST("K2" AS VARCHAR),CAST("V1"
> AS DECIMAL),CAST("0"."V2" AS VARCHAR),CAST("0"."V3" AS VARCHAR) FROM MY_VIEW
> WHERE ("K1","K3") IN ((?,?),(?,?));
> {code}
> and has very poor performance and causes performance degradation.
> -- *[POSSIBLE
> WORKAROUND]*
>
> One possible workaround is to provide all the pk (including the view pk
> columns)
> {code:java}
> EXPLAIN SELECT K1, K2, K3, V1, V2, V3 FROM MY_VIEW WHERE (K1,K2,K3) IN
> ((1,'A',2),(3,'A',4));
> +--+-++--+
> | PLAN
> | EST_BYTES_READ | EST_ROWS_READ | EST_INFO_TS |
>
[jira] [Commented] (PHOENIX-5935) Select with non primary keys and PHOENIX_ROW_TIMESTAMP() in where clause fails
[ https://issues.apache.org/jira/browse/PHOENIX-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124600#comment-17124600 ] Geoffrey Jacoby commented on PHOENIX-5935: -- [~jisaac] What happens if a row has Cells with different timestamps, particularly if you select multiple Cells? > Select with non primary keys and PHOENIX_ROW_TIMESTAMP() in where clause fails > -- > > Key: PHOENIX-5935 > URL: https://issues.apache.org/jira/browse/PHOENIX-5935 > Project: Phoenix > Issue Type: Bug >Reporter: Jacob Isaac >Assignee: Jacob Isaac >Priority: Major > Attachments: PHOENIX-5935.4.x.001.patch > > > This fails when COLUMN_ENCODED_BYTES != NON_ENCODED_QUALIFIERS > Steps to reproduce:- > CREATE TABLE IF NOT EXISTS N01 (PK1 INTEGER NOT NULL, PK2 DATE NOT NULL, > KV1 VARCHAR, KV2 VARCHAR CONSTRAINT PK PRIMARY KEY(PK1, PK2)) > COLUMN_ENCODED_BYTES = 1,IMMUTABLE_STORAGE_SCHEME = ONE_CELL_PER_COLUMN > > SELECT COUNT(*) FROM N01 WHERE PHOENIX_ROW_TIMESTAMP() > PK2 AND KV1 = > 'KV1_1'"; > > Fails with the following exception - > Caused by: java.util.NoSuchElementExceptionCaused by: > java.util.NoSuchElementException at > org.apache.phoenix.filter.MultiEncodedCQKeyValueComparisonFilter$FilteredKeyValueHolder.getCellAtIndex(MultiEncodedCQKeyValueComparisonFilter.java:151) > at > org.apache.phoenix.filter.MultiEncodedCQKeyValueComparisonFilter$EncodedCQIncrementalResultTuple.getValue(MultiEncodedCQKeyValueComparisonFilter.java:311) > at > org.apache.phoenix.expression.function.PhoenixRowTimestampFunction.evaluate(PhoenixRowTimestampFunction.java:98) > at > org.apache.phoenix.expression.ComparisonExpression.evaluate(ComparisonExpression.java:330) > at > org.apache.phoenix.expression.AndOrExpression.evaluate(AndOrExpression.java:75) > at > org.apache.phoenix.filter.BooleanExpressionFilter.evaluate(BooleanExpressionFilter.java:93) > at > org.apache.phoenix.filter.MultiEncodedCQKeyValueComparisonFilter.filterKeyValue(MultiEncodedCQKeyValueComparisonFilter.java:233) > at > org.apache.hadoop.hbase.regionserver.querymatcher.UserScanQueryMatcher.matchColumn(UserScanQueryMatcher.java:122) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5931) PhoenixIndexFailurePolicy throws NPE if cause of IOE is null
[ https://issues.apache.org/jira/browse/PHOENIX-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121392#comment-17121392 ] Geoffrey Jacoby commented on PHOENIX-5931: -- +1, good find. > PhoenixIndexFailurePolicy throws NPE if cause of IOE is null > > > Key: PHOENIX-5931 > URL: https://issues.apache.org/jira/browse/PHOENIX-5931 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.16.0 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Minor > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5931-4.x.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
