Re: [DISCUSS] Design for a "query log"

2018-03-02 Thread Artem Ervits
+1 Is idea here to collect information on all types of queries even against system tables? It would be nice to keep counts of how many times a query was executed over time. In my mssql days we also had ability to check when was the last time update stats ran and all kinds of impact information fro

array_remove array_replace?

2018-03-02 Thread Xavier Jodoin
Hi, Does anyone tried to implement array_remove and array_replace function? Thanks Xavier Jodoin

Re: [DISCUSS] Design for a "query log"

2018-03-02 Thread Nick Dimiduk
I'm a big fan of this idea. There was a brief discussion on the topic over on PHOENIX-2715. My first concern is that the collected information is huge -- easily far larger than the user data for a busy cluster. For instance, a couple 10's of GB stored user data, guideposts set to default 100mb, en

[jira] [Commented] (PHOENIX-4607) Allow PhoenixInputFormat to use tenant-specific connections

2018-03-02 Thread Thomas D'Silva (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383902#comment-16383902 ] Thomas D'Silva commented on PHOENIX-4607: - +1 LGTM > Allow PhoenixInputFormat t

Re: [DISCUSS] Design for a "query log"

2018-03-02 Thread Andrew Purtell
Agree with Nick's points but let me augment with an additional suggestion: Tunable/configurable threshold for sampling. In many cases it's sufficient to sample e.g. 1% of queries to get sufficient coverage and this would prune 99% of actual load from the query log. Also let me underline that compl

[jira] [Updated] (PHOENIX-4130) Avoid server retries for mutable indexes

2018-03-02 Thread Vincent Poon (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated PHOENIX-4130: -- Attachment: PHOENIX-4130_5.x-HBase-2.0.v2.patch > Avoid server retries for mutable indexes > -

[jira] [Commented] (PHOENIX-4130) Avoid server retries for mutable indexes

2018-03-02 Thread Vincent Poon (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384021#comment-16384021 ] Vincent Poon commented on PHOENIX-4130: --- Thanks [~rajeshbabu] , attached a v2 that

[jira] [Commented] (PHOENIX-4632) Phoenix does not use local index if there is a field in where clause not included in index

2018-03-02 Thread James Taylor (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384084#comment-16384084 ] James Taylor commented on PHOENIX-4632: --- [~rajeshbabu] - would you have any spare

[jira] [Commented] (PHOENIX-4635) HBase Connection leak in org.apache.phoenix.hive.mapreduce.PhoenixInputFormat

2018-03-02 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384122#comment-16384122 ] Hudson commented on PHOENIX-4635: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBas

Re: pre-commit Jenkins build isn't running - is this related to mast moved to HBase 1.4?

2018-03-02 Thread Thomas D'Silva
The precommit build is running but not posting the results to the jira. It looks like hadoopqa is unable to login. Does anyone know how to fix this? export USER=hudson + USER=hudson + /home/jenkins/tools/jiracli/latest/jira -s https://issues.apache.org/jira -a addcomment -u hadoopqa -p 4hadoopqa

[jira] [Commented] (PHOENIX-4635) HBase Connection leak in org.apache.phoenix.hive.mapreduce.PhoenixInputFormat

2018-03-02 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384201#comment-16384201 ] Hudson commented on PHOENIX-4635: - SUCCESS: Integrated in Jenkins build Phoenix-4.x-HBas

Re: [DISCUSS] Design for a "query log"

2018-03-02 Thread Josh Elser
My gut reaction would be to avoid storing queries over the system tables as they would have more noise than value, but I think this is something that could be entertained! On 3/2/18 8:02 AM, Artem Ervits wrote: +1 Is idea here to collect information on all types of queries even against system

[jira] [Commented] (PHOENIX-2715) Query Log

2018-03-02 Thread Josh Elser (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384223#comment-16384223 ] Josh Elser commented on PHOENIX-2715: - Nick kindly pointed me to this one. There's a

[jira] [Commented] (PHOENIX-4635) HBase Connection leak in org.apache.phoenix.hive.mapreduce.PhoenixInputFormat

2018-03-02 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384249#comment-16384249 ] Hudson commented on PHOENIX-4635: - ABORTED: Integrated in Jenkins build PreCommit-PHOENI

[jira] [Commented] (PHOENIX-4607) Allow PhoenixInputFormat to use tenant-specific connections

2018-03-02 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384250#comment-16384250 ] Hudson commented on PHOENIX-4607: - ABORTED: Integrated in Jenkins build PreCommit-PHOENI

Re: [DISCUSS] Design for a "query log"

2018-03-02 Thread Josh Elser
Thanks Nick and Andrew! These are great points. * A TTL out of the box is a must. That's such a good suggestion * Sensitivity of data being stored is also a tricky-serious issue to consider. We'll want to lock the table down and be able to state very clearly what data may show up in it. * I lik

[jira] [Commented] (PHOENIX-4418) UPPER() and LOWER() functions should be locale-aware

2018-03-02 Thread Thomas D'Silva (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384262#comment-16384262 ] Thomas D'Silva commented on PHOENIX-4418: - [~shehzaadn], can you also update our

[jira] [Assigned] (PHOENIX-4624) COLLATION_KEY function cannot handle null values

2018-03-02 Thread Shehzaad Nakhoda (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shehzaad Nakhoda reassigned PHOENIX-4624: - Assignee: Shehzaad Nakhoda > COLLATION_KEY function cannot handle null values >

Re: [DISCUSS] Design for a "query log"

2018-03-02 Thread Samarth Jain
A couple more points which I think you alluded to, Josh, but I would still like to call out: 1) Writing of these query logs to a phoenix table should be best effort i.e. a query definitely shouldn't fail because we encountered an issue while writing its log 2) Writing of query logs should happen in

Re: [DISCUSS] Design for a "query log"

2018-03-02 Thread Josh Elser
Yup, totally agree. Will also try to make these more clear in the doc. We want to queue up the work to "log" and do it in the background. If that queue gets too full, we drop it right off that cliff :) Thanks, Samarth! On 3/2/18 5:47 PM, Samarth Jain wrote: A couple more points which I think

[jira] [Commented] (PHOENIX-4607) Allow PhoenixInputFormat to use tenant-specific connections

2018-03-02 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384288#comment-16384288 ] Hudson commented on PHOENIX-4607: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBas

[jira] [Commented] (PHOENIX-4539) Unit tests failures on move of master to HBase 1.4

2018-03-02 Thread Thomas D'Silva (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384322#comment-16384322 ] Thomas D'Silva commented on PHOENIX-4539: - The pom.xml still uses the 1.3 module

[jira] [Commented] (PHOENIX-4607) Allow PhoenixInputFormat to use tenant-specific connections

2018-03-02 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384332#comment-16384332 ] Hudson commented on PHOENIX-4607: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBas

[jira] [Commented] (PHOENIX-4130) Avoid server retries for mutable indexes

2018-03-02 Thread Vincent Poon (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384362#comment-16384362 ] Vincent Poon commented on PHOENIX-4130: --- [~rajeshbabu] Actually, MutableIndexFailu

[jira] [Updated] (PHOENIX-4304) Refactoring to avoid using deprecated HTableDescriptor, HColumnDescriptor, HRegionInfo

2018-03-02 Thread James Taylor (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-4304: -- Fix Version/s: (was: 4.14.0) 5.0.0 > Refactoring to avoid using depreca

[jira] [Commented] (PHOENIX-4304) Refactoring to avoid using deprecated HTableDescriptor, HColumnDescriptor, HRegionInfo

2018-03-02 Thread James Taylor (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384388#comment-16384388 ] James Taylor commented on PHOENIX-4304: --- FYI, I changed fixVersion from 4.14.0 to

[jira] [Commented] (PHOENIX-4633) Handle the creation of SYSTEM.CATALOG correctly for users that don't have CREATE access on SYSTEM.CATALOG

2018-03-02 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384407#comment-16384407 ] Hudson commented on PHOENIX-4633: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBas

[jira] [Commented] (PHOENIX-4633) Handle the creation of SYSTEM.CATALOG correctly for users that don't have CREATE access on SYSTEM.CATALOG

2018-03-02 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384425#comment-16384425 ] Hudson commented on PHOENIX-4633: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBas

[jira] [Commented] (PHOENIX-4418) UPPER() and LOWER() functions should be locale-aware

2018-03-02 Thread Thomas D'Silva (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384463#comment-16384463 ] Thomas D'Silva commented on PHOENIX-4418: - [~shehzaadn] There were some test fai

[jira] [Comment Edited] (PHOENIX-4418) UPPER() and LOWER() functions should be locale-aware

2018-03-02 Thread Thomas D'Silva (JIRA)
[ https://issues.apache.org/jira/browse/PHOENIX-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384463#comment-16384463 ] Thomas D'Silva edited comment on PHOENIX-4418 at 3/3/18 4:11 AM: -