Multi-Tenancy and shared records

2019-09-02 Thread Simon Mottram
Hi I'm working on a project where we have a combination of very sparse data columns with added headaches of multi-tenancy. Hbase looks great for the back end but I need to check that we can support the customer's multi-tenancy requirements. There are 2 that I'm struggling to find a definitive

Re: Multi-Tenancy and shared records

2019-09-03 Thread Simon Mottram
on for > what > you're trying to accomplish. > > On 9/2/19 8:34 PM, Simon Mottram wrote: > > Hi > > > > I'm working on a project where we have a combination of very sparse > > data columns with added headaches of multi-tenancy. Hbase looks > > grea

Re: Multi-Tenancy and shared records

2019-09-04 Thread Simon Mottram
p > mutation.setACL(perms); > } > htable.batch(mutationList); > } > conn.rollback(); > > > On Tue, Sep 3, 2019 at 3:19 PM Simon Mottram < > simon.mott...@cucumber.co.nz> wrote: > > Hi Josh > > > > Thought

Issue using ANY ARRAY feature

2019-11-17 Thread Simon Mottram
Phoenix Version: 4.14.2-HBase-1.4 HBase Version: AWS EMR Release label:emr-5.24.1 Hadoop distribution:Amazon Applications:Phoenix 4.14.1, Hue 4.4.0, HBase 1.4.9 Having an issue where ANY(ARRAY[]) stops the query returning any results when used in a 'AND' conjunction e.g (fielda = 'a') AND

Re: Issue using ANY ARRAY feature

2019-12-08 Thread Simon Mottram
: org.apache.phoenix phoenix-queryserver-client 4.14.1-HBase-1.4 This required support help from AWS as this driver version is not mentioned on the official Apache Phoenix download page Regards Simon On Sun, 2019-11-17 at 21:03 +, Simon Mottram wrote

Re: Multi-Tenancy and shared records

2019-10-16 Thread Simon Mottram
for your original problem, I’m not sure if Apache Ranger > supports row-level yet in their HBase plugin but you can certainly > add that functionality to make what you’re talking about a LOT easier > to maintain. > > Good luck, > > Billy Watson > > On Tue, Oct 1

Re: Multi-Tenancy and shared records

2019-10-15 Thread Simon Mottram
of the data means > that I will be constantly adding new columns, not sure if Phoenix > would > have a problem with that. > Phoenix supports dynamic columns so you should not have a problem > with that. > > Regards, > Ankit Singhal > > On Wed, Sep 4, 2019 at 6:24 PM Simon M

Re: Reverse engineer a phoneix table definition

2020-04-11 Thread Simon Mottram
Best I can offer is "SELECT * FROM SYSTEM.CATALOG where table_name = '" + tableName + "' and table_schem = '" +schemaName + "'" S From: Mich Talebzadeh Sent: Sunday, 12 April 2020 1:36 AM To: user Subject: Reverse engineer a phoneix table definition Hi, I

Doing an array intersection check when querying Hbase

2020-04-05 Thread Simon Mottram
Hi I'm looking for some pointers on how to address this requirement: I have records that have an array field, this array could have 20 entries at the very outside: e.g ['GROUP1', 'GROUP2', 'GROUP3'] A user can be a member of many groups again, comparatively small numbers, think 20 or so

Advice on primary row key construction

2020-05-11 Thread Simon Mottram
Hiya, hope someone has some tips to spare on this subject I have a use case where a column is an array and we would like to be able to filter records based on this array. We know that the number of elements in the array would be small, 10 absolute max. Is there a way we can encode this in a

Unable to use some functions on DECIMAL columns

2020-07-29 Thread Simon Mottram
If you try the queries marked with BUG below, you get an exception SQL Error [0]: Error -1 (0) : Error while executing SQL "ESELECT PERCENTILE_DISC (0.5) WITHIN GROUP (ORDER BY DECIMALCOLUMN ASC) FROM TEST.TESTEXCEPTIONS": Remote driver error: ArrayIndexOutOfBoundsException: (null

Re: Unable to use some functions on DECIMAL columns

2020-08-03 Thread Simon Mottram
n.java:90) at org.apache.phoenix.expression.function.DistinctValueWithCountAggregateFunction.(DistinctValueWithCountAggregateFunction.java:36) at org.apache.phoenix.expression.function.StddevPopFunction.(StddevPopFunction.java:51) ... 44 more ] ____ From: Simon Mottram Sent: 04 August 2020 1:12 PM To: user@phoenix.apache.org Su

Re: Unable to use some functions on DECIMAL columns

2020-08-03 Thread Simon Mottram
nt or if it affects both the thin-client and thick-client is important. On 7/31/20 9:10 PM, Simon Mottram wrote: > Hi Josh > > Thanks very much for the reply, I did share the error at the top of the > email > > SQL Error [0]: Error -1 (0) : Error while executing SQL "ES

Re: Unable to use some functions on DECIMAL columns

2020-07-31 Thread Simon Mottram
try to fix the issues you're seeing ;) On 7/29/20 10:18 PM, Simon Mottram wrote: > > If you try the queries marked with BUG below, you get an exception > > SQL Error [0]: Error -1 (0) : Error while executing SQL "ESELECT > PERCENTILE_DISC (0.5) WITHIN GROUP (OR

Re: Status of Apache Kafka Plugin?

2020-11-15 Thread Simon Mottram
Accessing kafka broker when it is running inside an image causes people a lot of problems, sounds like this may be your problem Basically, it's all about configuring two zookeeper listeners, one for inside the docker network and an external one for your PC This is the best blog I have found

Re: Setting hbase.rpc.timeout for Phoenix queries on AWS EMR

2021-05-12 Thread Simon Mottram
11:45 PM To: user@phoenix.apache.org Cc: Simon Mottram Subject: Re: Setting hbase.rpc.timeout for Phoenix queries on AWS EMR The client XML file is the hbase-site.xml that is on the classpath of the Phoenix client application. You have to add it to the client classpath, and keep the hbase

Setting hbase.rpc.timeout for Phoenix queries on AWS EMR

2021-05-11 Thread Simon Mottram
Has anyone had experience of this? We have the issue that queries that run longer than a minute are cut off. I presume this is the hbase.rpc.timeout setting. Our admin used the following to configure EMR, but it seems to have had no effect. The posts I can find mention setting this timeout

Re: [Discuss] Phoenix Tech Talks

2021-02-25 Thread Simon Mottram
This is great news. It's not a great time to watch live here in NZ but I will be following with great interest Many thanks S From: Kadir Ozdemir Sent: 26 February 2021 8:09 AM To: d...@phoenix.apache.org Cc: user Subject: Re: [Discuss] Phoenix Tech Talks Hi

Advice wanted on supporting a tag feature for searching an HBase Table via Phoenix

2021-02-21 Thread Simon Mottram
The requirement is to be able to search from a list of tags, each record can have a possible large number of tags. There would be more than one tag field. An example might 3 different hashtag fields. They do have to be different; we can't have just one tag cloud. The data size is large so we

Re: Advice wanted on supporting a tag feature for searching an HBase Table via Phoenix

2021-02-22 Thread Simon Mottram
:P). It's a fast rowkey, prefix scan to pull all the tags for the "data record". Just don't forget that hbase won't split a single row across multiple Regions. That's the important part in designing this table. On 2/21/21 11:51 PM, Simon Mottram wrote: > The requirement is to be abl

Re: Major problem for us with Phoenix joins with certain aggregations

2021-10-11 Thread Simon Mottram
.@apache.org>> wrote: No worries. Thanks for confirming! On 10/10/21 1:43 PM, Simon Mottram wrote: > Hi > > Thanks for the reply, I posted here by mistake and wasn't sure how to delete. > It's indeed a problem with phoenix > > Sorry

Re: Urgent problem with Phoenix function LAST_VALUE and joins

2021-09-28 Thread Simon Mottram
, 2021 at 9:23 PM Simon Mottram mailto:simon.mott...@cucumber.co.nz>> wrote: Hi Got my fingers crossed that there's a work around for this as this really is a big problem for us We are using: Amazon EMR Release label:emr-6.1.0 Hadoop distribution:Amazon Applications:Hbase 2.2.5, Hive

Re: Urgent problem with Phoenix function LAST_VALUE and joins

2021-09-28 Thread Simon Mottram
unning HBase 2.2, which I thought wasn't supported until Phoenix 5.1?) Geoffrey On Tue, Sep 28, 2021 at 4:07 AM Simon Mottram mailto:simon.mott...@cucumber.co.nz>> wrote: Hi thanks for the quick response I'm going to appear thick now, brace yourself. By client do you mean the queryse

Re: Urgent problem with Phoenix function LAST_VALUE and joins

2021-09-29 Thread Simon Mottram
cluster. On Wed, Sep 29, 2021 at 12:36 AM Simon Mottram mailto:simon.mott...@cucumber.co.nz>> wrote: Ok, should I bug this in your JIRA? I will try and setup a more reproducible series of steps Cheers S From: Kadir Ozdemir mailto:kozde...@salesforce.com>&

Re: Urgent problem with Phoenix function LAST_VALUE and joins

2021-09-30 Thread Simon Mottram
To: user@phoenix.apache.org Subject: Re: Urgent problem with Phoenix function LAST_VALUE and joins git clone https://github.com/apache/phoenix mvn clean package bin/phoenix_sandbox.py use the host and port displayed to start sqlline.py On Thu, Sep 30, 2021 at 4:21 AM Simon Mottram mailto:

Urgent problem with Phoenix function LAST_VALUE and joins

2021-09-27 Thread Simon Mottram
Hi Got my fingers crossed that there's a work around for this as this really is a big problem for us We are using: Amazon EMR Release label:emr-6.1.0 Hadoop distribution:Amazon Applications:Hbase 2.2.5, Hive 3.1.2, Phoenix 5.0.0, Pig 0.17.0 Thin Client version:

PERCENTILE_DISC has problems with DECIMAL columns that may have nulls in

2022-03-15 Thread Simon Mottram
Just checking here for workarounds, mistakes on my part before I create a JIRA bug -- Tested with both DBeaver and local new build of Phoenix -- Phoenix sandbox. HBase 2.4.9 Phoenix 5.2.0-SNAPSHOT pulled from master 13/3/2022 -- DBeaver driver version - phoenix-5.0.0-HBase-2.0-thin-client.jar

Certain Aggregates break if a GROUP BY clause results in single row with a NULL column value

2022-03-15 Thread Simon Mottram
Sorry for bug spam, just want to get feedback (if any) before I JIRA If you just create the first 10 rows in the scripts below, most functions work fine. If you create records that are in just one group, that contain any nulls, then exceptions are thrown: So try with first 10, confirm stuff

Re: Bugs with multiple LAST_VALUE aggregation on multiple columns

2022-03-16 Thread Simon Mottram
repro. If you can repro the problem with sandbox+sqlline, then please open the JIRA. If the behaviour is different with the latest PQS + thin client, then please also note that. This applied to all three problems. On Wed, Mar 16, 2022 at 1:24 AM Simon Mottram mailto:simon.mott...@cucumbe

Bugs with multiple LAST_VALUE aggregation on multiple columns

2022-03-15 Thread Simon Mottram
Before I submit a JIRA, can someone point out if I made a mistake. Pretty sure the exceptions are a bug, but... -- Tested with both DBeaver and local new build of Phoenix -- Phoenix sandbox. HBase 2.4.9 Phoenix 5.2.0-SNAPSHOT pulled from master 13/3/2022 -- DBeaver driver version -

Re: Bugs with multiple LAST_VALUE aggregation on multiple columns

2022-03-17 Thread Simon Mottram
r 17, 2022 at 12:54 AM Simon Mottram mailto:simon.mott...@cucumber.co.nz>> wrote: Hi Thanks for the quick response, we can ignore Dbeaver from now on I was using the PQS version that was included in the latest Phoenix HEAD (5.2.0.SNAPSHOT) I've built the latest queryserver 6.0.1-SNAPSHOT