[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats

2017-11-21 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261691#comment-16261691
 ] 

ASF GitHub Bot commented on HIVE-11294:
---

Github user alanfgates closed the pull request at:

https://github.com/apache/hive/pull/46


> Use HBase to cache aggregated stats
> ---
>
> Key: HIVE-11294
> URL: https://issues.apache.org/jira/browse/HIVE-11294
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: hbase-metastore-branch
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Fix For: hbase-metastore-branch, 2.0.0
>
> Attachments: HIVE-11294.2.patch, HIVE-11294.patch
>
>
> Currently stats are cached only in the memory of the client.  Given that 
> HBase can easily manage the scale of caching aggregated stats we should be 
> using it to do so.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats

2015-07-24 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640711#comment-14640711
 ] 

Alan Gates commented on HIVE-11294:
---

Good question, I don't know.  I'll ask the HBase community if they recommend 
moving it, and if so we can file a ticket to do that.

> Use HBase to cache aggregated stats
> ---
>
> Key: HIVE-11294
> URL: https://issues.apache.org/jira/browse/HIVE-11294
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: hbase-metastore-branch
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: hbase-metastore-branch
>
> Attachments: HIVE-11294.2.patch, HIVE-11294.patch
>
>
> Currently stats are cached only in the memory of the client.  Given that 
> HBase can easily manage the scale of caching aggregated stats we should be 
> using it to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats

2015-07-23 Thread Thejas M Nair (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640019#comment-14640019
 ] 

Thejas M Nair commented on HIVE-11294:
--

[~alangates] Should the bloom column and stats column in this new table be in 
different column families ? It seems like two column families would be better, 
as the two are not being accessed simultaneously - The filter invalidation 
looks only at bloom filter information, while calls to get stats would only 
look at the stats column.


> Use HBase to cache aggregated stats
> ---
>
> Key: HIVE-11294
> URL: https://issues.apache.org/jira/browse/HIVE-11294
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: hbase-metastore-branch
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: hbase-metastore-branch
>
> Attachments: HIVE-11294.2.patch, HIVE-11294.patch
>
>
> Currently stats are cached only in the memory of the client.  Given that 
> HBase can easily manage the scale of caching aggregated stats we should be 
> using it to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats

2015-07-22 Thread Lefty Leverenz (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638169#comment-14638169
 ] 

Lefty Leverenz commented on HIVE-11294:
---

Doc note:  This creates four configuration parameters, so they will need to be 
documented in the wiki after hbase-metastore-branch gets merged to trunk.  In 
the meantime, I'm linking this issue to HIVE-9752 (Documentation for HBase 
metastore).

> Use HBase to cache aggregated stats
> ---
>
> Key: HIVE-11294
> URL: https://issues.apache.org/jira/browse/HIVE-11294
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: hbase-metastore-branch
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: hbase-metastore-branch
>
> Attachments: HIVE-11294.2.patch, HIVE-11294.patch
>
>
> Currently stats are cached only in the memory of the client.  Given that 
> HBase can easily manage the scale of caching aggregated stats we should be 
> using it to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats

2015-07-21 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635866#comment-14635866
 ] 

Alan Gates commented on HIVE-11294:
---

Incorporated most of the feedback.  Responses a couple of the comments where I 
disagreed.

> Use HBase to cache aggregated stats
> ---
>
> Key: HIVE-11294
> URL: https://issues.apache.org/jira/browse/HIVE-11294
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: hbase-metastore-branch
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: hbase-metastore-branch
>
> Attachments: HIVE-11294.patch
>
>
> Currently stats are cached only in the memory of the client.  Given that 
> HBase can easily manage the scale of caching aggregated stats we should be 
> using it to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats

2015-07-20 Thread Thejas M Nair (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634527#comment-14634527
 ] 

Thejas M Nair commented on HIVE-11294:
--

Reviewed most of it. Adding some comments - 
https://github.com/apache/hive/pull/46 
You can also use this link to view -  
https://github.com/alanfgates/hive/commit/738e901a5c9a69342f29fffbfc53cb53ff192694#diff-0bb77b0faac26bc52b50eec03ed9a997

> Use HBase to cache aggregated stats
> ---
>
> Key: HIVE-11294
> URL: https://issues.apache.org/jira/browse/HIVE-11294
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: hbase-metastore-branch
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: hbase-metastore-branch
>
> Attachments: HIVE-11294.patch
>
>
> Currently stats are cached only in the memory of the client.  Given that 
> HBase can easily manage the scale of caching aggregated stats we should be 
> using it to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats

2015-07-17 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632048#comment-14632048
 ] 

ASF GitHub Bot commented on HIVE-11294:
---

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/46

Hbase metastore

HIVE-11294 Cache aggregated stats in hbase

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hbase-metastore

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/46.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #46


commit 58ca965ca58925e4846e14cf37d64638edb3e353
Author: Alan Gates 
Date:   2015-01-23T19:05:19Z

Creating branch for hbase metastore development.


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1654331 
13f79535-47bb-0310-9956-ffa450edef68

commit e81704eeaf061e7152c63379b828debdfdfe9a8e
Author: Alan Gates 
Date:   2015-02-04T20:00:49Z

HIVE-9453 Initial patch [hbase-metastore branch] (Alan Gates, reviewed by 
Thejas Nair)


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1657394 
13f79535-47bb-0310-9956-ffa450edef68

commit 120047c308a011f17f94258779e4c47fcff191a5
Author: Alan Gates 
Date:   2015-02-10T21:41:19Z

HIVE-9578 Add support for getDatabases and alterDatabase calls 
[hbase-metastore branch]


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1658821 
13f79535-47bb-0310-9956-ffa450edef68

commit c7dcf4a85d5cef4151d7b72115b4a1027af488ac
Author: Alan Gates 
Date:   2015-02-10T22:47:53Z

HIVE-9579 Support all get tables [hbase-metastore branch]


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1658829 
13f79535-47bb-0310-9956-ffa450edef68

commit cd82601f6e5c4aea5db72b74629b6aada23cf7c2
Author: Alan Gates 
Date:   2015-02-19T18:51:39Z

HIVE-9606 Need a tool to export metadata from RDBMS based metastore into 
HBase [hbase-metastore branch] (Alan Gates, reviewed by Thejas Nair)


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1660974 
13f79535-47bb-0310-9956-ffa450edef68

commit 21ba0a93dcadcf8726ae7a18e81f758921552697
Author: Alan Gates 
Date:   2015-03-02T18:17:01Z

HIVE-9641 Fill out remaining partition functions in HBaseStore (Alan Gates 
reviewed by Vaibhav Gumashta)


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1663386 
13f79535-47bb-0310-9956-ffa450edef68

commit bb0010caa8734af34ecac7b5b054afd95b49b0e0
Author: Alan Gates 
Date:   2015-03-05T03:46:28Z

HIVE-9677 Implement privileges call in HBaseStore (Alan Gates)


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1664220 
13f79535-47bb-0310-9956-ffa450edef68

commit 408d467bffa0923714b8262427f1d1d72ba9932d
Author: Alan Gates 
Date:   2015-03-05T04:09:31Z

HIVE-9763 Remove M* classes from RawStore interface (Alan Gates)


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1664226 
13f79535-47bb-0310-9956-ffa450edef68

commit be575b84a8d5b4764550499c60bdccb7b0d1f82f
Author: Alan Gates 
Date:   2015-03-05T04:35:53Z

HIVE-9802 Refactor HBaseReadWrite to allow different implementations 
underneath, also includes code from HIVE-9783 Move serialization of objects in 
HBase to protocol buffers (Alan Gates)


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1664227 
13f79535-47bb-0310-9956-ffa450edef68

commit 646468fcdfd612524961751ba3575a26a1fb3fcb
Author: Alan Gates 
Date:   2015-03-12T19:41:56Z

HIVE-9885 HBase metastore protobufs need to check hasX() functions for 
values that are not auto set in serialization [hbase-metastore branch] (Alan 
Gates)


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1666280 
13f79535-47bb-0310-9956-ffa450edef68

commit d268b81bcd2fbaae6fdbfe2f980f644ce5e3d92d
Author: Alan Gates 
Date:   2015-03-12T20:01:30Z

HIVE-9874 Partition storage descriptors being set from table sd without 
copying [hbase-metastore branch] (Alan Gates)


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1666285 
13f79535-47bb-0310-9956-ffa450edef68

commit 24f224867b70bacc91a877242aee1b0b84efd127
Author: Alan Gates 
Date:   2015-03-17T15:50:31Z

HIVE-9942 Implement functions methods in HBaseStore (Alan Gates)


git-svn-id: 
https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1667338 
13f79535-47bb-0310-9956-ffa450edef68

commit 35ac8b0257614523064c1b3647a91e123b78e999
Author: Alan Gates 
Date:   2015-03-18T17:41:50Z

HIVE-10008 Ne

[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats

2015-07-17 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631945#comment-14631945
 ] 

Hive QA commented on HIVE-11294:




{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745874/HIVE-11294.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4641/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4641/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4641/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4641/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   7fe23aa..8646c12  master -> origin/master
   dddffe1..4221699  spark  -> origin/spark
+ git reset --hard HEAD
HEAD is now at 7fe23aa HIVE-11282: CBO (Calcite Return Path): Inferring Hive 
type char/varchar of length zero which is not allowed (Jesus Camacho Rodriguez, 
reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 8646c12 HIVE-11285 : ObjectInspector for partition columns in 
FetchOperator in SMBJoin causes exception (Pengcheng Xiong via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12745874 - PreCommit-HIVE-TRUNK-Build

> Use HBase to cache aggregated stats
> ---
>
> Key: HIVE-11294
> URL: https://issues.apache.org/jira/browse/HIVE-11294
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: hbase-metastore-branch
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: hbase-metastore-branch
>
> Attachments: HIVE-11294.patch
>
>
> Currently stats are cached only in the memory of the client.  Given that 
> HBase can easily manage the scale of caching aggregated stats we should be 
> using it to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)