[
https://issues.apache.org/jira/browse/PHOENIX-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14997553#comment-14997553
]
Don Brinn edited comment on PHOENIX-2381 at 11/9/15 10:44 PM:
--------------------------------------------------------------
I did a bit more investigating and found an area where I though the problem
might be. However, the results of my investigation make me think this is not
actually the problem. I am including my notes here just in case this is useful
or so others can avoid doing the same thing:
At line 84 of
{quote}phoenix-core\src\main\java\org\apache\phoenix\coprocessor\HashJoinRegionScanner.java{quote}
(as of the Phoenix 4.6.0 release source code) there is this line of code where
"cache" is retrieved using the tenantId:
{quote}TenantCache cache = GlobalCache.getTenantCache(env, tenantId);{quote}
And then at line 93 there is this block of code that looks in the cache for the
joinId, throwing an exception if it cannot be found:
{quote}HashCache hashCache = (HashCache)cache.getServerCache(joinId);
if (hashCache == null)
throw new DoNotRetryIOException("Could not find hash cache for
joinId: "
+ Bytes.toString(joinId.get(),
joinId.getOffset(), joinId.getLength())
+ ". The cache might have expired and have been
removed.");{quote}
I updated it to the following, to include the tenantId in the exception's error
message:
{quote}HashCache hashCache = (HashCache)cache.getServerCache(joinId);
if (hashCache == null)
throw new DoNotRetryIOException("Could not find hash cache for
joinId: "
+ Bytes.toString(joinId.get(),
joinId.getOffset(), joinId.getLength())
+ ", tenantId: " + tenantId
+ ". The cache might have expired and have been
removed.");{quote}
Now when I reproduce the problem using the views/tables/data that I attached
earlier, attaching to phoenix with tenantId=15, and using the statement
{quote}select PRODUCT_NAME, "E"."V" from INVENTORY INNER JOIN PRODUCT_IDS ON
(INVENTORY.ID = PRODUCT_ID);{quote}, I get this error message:
{quote}java.lang.RuntimeException:
org.apache.phoenix.exception.PhoenixIOException:
org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache for
joinId: ����, tenantId: 00 00 00 0f. The cache might have expired and have been
removed.
at
org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:96)
at
org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:212)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1931)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3178)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
at sqlline.SqlLine.print(SqlLine.java:1653)
at sqlline.Commands.execute(Commands.java:833)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292){quote}
That tenantId, "00 00 00 0f" is correct for the tenantId of 15 that I was using
for my connection to Phoenix. I did this investigation because I thought maybe
the {quote}tenantId{quote} variable might be null at that point, but it is not.
was (Author: dbrinn):
I did a bit more investigating and found an area where I though the problem
might be. However, the results of my investigation make me think this is not
actually the problem. I am including my notes here just in case this is useful
or so others can avoid doing the same thing:
At line 84 of
{quote}phoenix-core\src\main\java\org\apache\phoenix\coprocessor\HashJoinRegionScanner.java{quote}
(as of the Phoenix 4.6.0 release source code) there is this line of code where
"cache" is retrieved using the tenantId:
{quote}TenantCache cache = GlobalCache.getTenantCache(env, tenantId);{quote}
And then at line 93 there is this block of code that looks in the cache for the
joinId, throwing an exception if it cannot be found:
{quote}{monospaced}HashCache hashCache =
(HashCache)cache.getServerCache(joinId);
if (hashCache == null)
throw new DoNotRetryIOException("Could not find hash cache for
joinId: "
+ Bytes.toString(joinId.get(),
joinId.getOffset(), joinId.getLength())
+ ". The cache might have expired and have been
removed.");{monospaced}{quote}
I updated it to the following, to include the tenantId in the exception's error
message:
{quote}{monospaced}HashCache hashCache =
(HashCache)cache.getServerCache(joinId);
if (hashCache == null)
throw new DoNotRetryIOException("Could not find hash cache for
joinId: "
+ Bytes.toString(joinId.get(),
joinId.getOffset(), joinId.getLength())
+ ", tenantId: " + tenantId
+ ". The cache might have expired and have been
removed.");{quote}{monospaced}
Now when I reproduce the problem using the views/tables/data that I attached
earlier, attaching to phoenix with tenantId=15, and using the statement
{quote}select PRODUCT_NAME, "E"."V" from INVENTORY INNER JOIN PRODUCT_IDS ON
(INVENTORY.ID = PRODUCT_ID);{quote}, I get this error message:
{quote}{monospaced}java.lang.RuntimeException:
org.apache.phoenix.exception.PhoenixIOException:
org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache for
joinId: ����, tenantId: 00 00 00 0f. The cache might have expired and have been
removed.
at
org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:96)
at
org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:212)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1931)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3178)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
at sqlline.SqlLine.print(SqlLine.java:1653)
at sqlline.Commands.execute(Commands.java:833)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292){quote}{monospaced}
That tenantId, "00 00 00 0f" is correct for the tenantId of 15 that I was using
for my connection to Phoenix. I did this investigation because I thought maybe
the {quote}tenantId{quote} variable might be null at that point, but it is not.
> Inner Join with any table or view with Multi_Tenant=true causes "could not
> find hash cache for joinId" error
> ------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-2381
> URL: https://issues.apache.org/jira/browse/PHOENIX-2381
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.6.0
> Environment: This is with Phoenix version 4.6.0 and HBase version
> 0.98.4.2.2.6.0-2800-hadoop2.
> Reporter: Don Brinn
> Assignee: Maryann Xue
> Labels: join, joins, multi-tenant
> Attachments: Inner_Join_Cache-Tables_and_Views_and_Data.zip,
> tmp-2381.patch
>
>
> I am seeing the following error when doing an INNER JOIN of a view with
> MULTI_TENANT=true with any other table or view:
> java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache for
> joinId: Ys�0��%�. The cache might have expired and have been removed.
> at
> org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:95)
> at
> org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:212)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1931)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3178)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
>
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
> at sqlline.SqlLine.print(SqlLine.java:1653)
> at sqlline.Commands.execute(Commands.java:833)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
>
> This is with Phoenix version 4.6.0 and HBase version
> 0.98.4.2.2.6.0-2800-hadoop2.
>
> This seems very strongly related to the MULTI_TENANT=true property on a view
> or table. I see the error whenever the view has MULTI_TENANT=true and I have
> a tenant-specific connection to Phoenix. I do not see the problem if the
> MULTI_TENANT=true property is not set on the view or if I do not have a
> tenant-specific connection to Phoenix.
>
> Here is an example SQL statement that has this error when the view INVENTORY
> has the MULTI_TENANT=true property and I have a tenant-specific connection,
> but that succeeds in other cases. (The view PRODUCT_IDS is not Multi-Tenant.)
> SELECT * FROM INVENTORY INNER JOIN PRODUCT_IDS ON (PRODUCT_ID = INVENTORY.ID)
>
> Note: "INNER JOIN" fails under these conditions, as does "LEFT OUTER JOIN".
> However, "RIGHT OUTER JOIN" and "FULL OUTER JOIN" do work. Also, if I tell
> Phoenix to use a Sort Join for the Inner Join or Left Outer Join then it does
> work, e.g. SELECT /\*+ USE_SORT_MERGE_JOIN\*/ * FROM INVENTORY INNER JOIN
> PRODUCT_IDS ON (PRODUCT_ID = INVENTORY.ID); works.
>
> This seems to be the same problem that was discussed previously in this
> mailing list:
> https://mail-archives.apache.org/mod_mbox/phoenix-user/201507.mbox/%3ccaotkwx5xfbwkjf--0k-zj91tfdqwfq6rmuqw0r_lojcnj1a...@mail.gmail.com%3E
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
