[jira] [Commented] (HIVE-18609) Results cache invalidation based on table updates
[ https://issues.apache.org/jira/browse/HIVE-18609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16433413#comment-16433413 ] Gopal V commented on HIVE-18609: LGTM - +1 tests pending HIVE-18948 is just a performance improvement for a cache-hit (i.e query is in cache, the tables haven't changed) & all other cases early exit on first-miss. > Results cache invalidation based on table updates > - > > Key: HIVE-18609 > URL: https://issues.apache.org/jira/browse/HIVE-18609 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.0 >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18609.1.patch, HIVE-18609.2.patch, > HIVE-18609.3.patch > > > Look into using the materialized view invalidation mechanisms to > automatically invalidate queries in the results cache if the underlying > tables used in the cached queries have been modified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18609) Results cache invalidation based on table updates
[ https://issues.apache.org/jira/browse/HIVE-18609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16433395#comment-16433395 ] Jason Dere commented on HIVE-18609: --- RB at https://reviews.apache.org/r/66368/ > Results cache invalidation based on table updates > - > > Key: HIVE-18609 > URL: https://issues.apache.org/jira/browse/HIVE-18609 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18609.1.patch, HIVE-18609.2.patch, > HIVE-18609.3.patch > > > Look into using the materialized view invalidation mechanisms to > automatically invalidate queries in the results cache if the underlying > tables used in the cached queries have been modified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18609) Results cache invalidation based on table updates
[ https://issues.apache.org/jira/browse/HIVE-18609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16433394#comment-16433394 ] Jason Dere commented on HIVE-18609: --- Rebasing with master again. > Results cache invalidation based on table updates > - > > Key: HIVE-18609 > URL: https://issues.apache.org/jira/browse/HIVE-18609 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18609.1.patch, HIVE-18609.2.patch, > HIVE-18609.3.patch > > > Look into using the materialized view invalidation mechanisms to > automatically invalidate queries in the results cache if the underlying > tables used in the cached queries have been modified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18609) Results cache invalidation based on table updates
[
https://issues.apache.org/jira/browse/HIVE-18609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16433371#comment-16433371
]
Hive QA commented on HIVE-18609:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918291/HIVE-18609.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10129/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10129/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10129/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-04-11 04:03:01.475
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-10129/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-04-11 04:03:01.478
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f6dbcac HIVE-19138: Results cache: allow queries waiting on
pending cache entries to check cache again if pending query fails (Jason Dere,
reviewed by Deepak Jaiswal)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f6dbcac HIVE-19138: Results cache: allow queries waiting on
pending cache entries to check cache again if pending query fails (Jason Dere,
reviewed by Deepak Jaiswal)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-04-11 04:03:04.937
+ rm -rf ../yetus_PreCommit-HIVE-Build-10129
+ mkdir ../yetus_PreCommit-HIVE-Build-10129
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-10129
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10129/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not
exist in index
error: a/itests/src/test/resources/testconfiguration.properties: does not exist
in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/Driver.java: does not exist in
index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
does not exist in index
error: a/ql/src/test/queries/clientpositive/results_cache_1.q: does not exist
in index
error: a/ql/src/test/queries/clientpositive/results_cache_2.q: does not exist
in index
error: a/ql/src/test/queries/clientpositive/results_cache_capacity.q: does not
exist in index
error: a/ql/src/test/queries/clientpositive/results_cache_empty_result.q: does
not exist in index
error: a/ql/src/test/queries/clientpositive/results_cache_lifetime.q: does not
exist in index
error: a/ql/src/test/queries/clientpositive/results_cache_quoted_identifiers.q:
does not exist in index
error: a/ql/src/test/queries/clientpositive/results_cache_temptable.q: does not
exist in index
error: a/ql/src/test/queries/clientpositive/results_cache_with_masking.q: does
not exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java:519
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java'
with conflicts.
Going to apply patch with: git apply -p1
/data/hiveptest/working/scratch/build.patch:603: trailing whitespace.
Map 1
/data/hiveptest/working/scratch/build.patch:619: trailing whitespace.
sort order:
/data/hiveptest/working/scratch/build.patch:624:
[jira] [Commented] (HIVE-18609) Results cache invalidation based on table updates
[ https://issues.apache.org/jira/browse/HIVE-18609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431552#comment-16431552 ] Jason Dere commented on HIVE-18609: --- Rebased with master. [~jcamachorodriguez] [~gopalv] can you take a look? > Results cache invalidation based on table updates > - > > Key: HIVE-18609 > URL: https://issues.apache.org/jira/browse/HIVE-18609 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18609.1.patch, HIVE-18609.2.patch > > > Look into using the materialized view invalidation mechanisms to > automatically invalidate queries in the results cache if the underlying > tables used in the cached queries have been modified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18609) Results cache invalidation based on table updates
[ https://issues.apache.org/jira/browse/HIVE-18609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419565#comment-16419565 ] Jason Dere commented on HIVE-18609: --- For the background thread to periodically check for invalidated queries in the cache, I will move that to a separate Jira. > Results cache invalidation based on table updates > - > > Key: HIVE-18609 > URL: https://issues.apache.org/jira/browse/HIVE-18609 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18609.1.patch > > > Look into using the materialized view invalidation mechanisms to > automatically invalidate queries in the results cache if the underlying > tables used in the cached queries have been modified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18609) Results cache invalidation based on table updates
[ https://issues.apache.org/jira/browse/HIVE-18609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419563#comment-16419563 ] Jason Dere commented on HIVE-18609: --- In-progress patch to check validity of cached queries on transactional tables by comparing the transactional WriteIDs. One part is waiting on HIVE-18948, though that part could be changed later once HIVE-18948 is in. > Results cache invalidation based on table updates > - > > Key: HIVE-18609 > URL: https://issues.apache.org/jira/browse/HIVE-18609 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18609.1.patch > > > Look into using the materialized view invalidation mechanisms to > automatically invalidate queries in the results cache if the underlying > tables used in the cached queries have been modified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18609) Results cache invalidation based on table updates
[ https://issues.apache.org/jira/browse/HIVE-18609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418075#comment-16418075 ] Jason Dere commented on HIVE-18609: --- Proposed changes: - At query time, the validity of the cached result will be checked by comparing the transactional WriteIDs of the tables used in the cached query to the WriteIDs of the current query, similar to HIVE-19027. This approach only guarantees freshness if all of the tables used in the query are transactional tables. As a result, the results cache will be disabled by default for queries involving non-transactional tables. - The above approach will only invalidate entries in the cache if that query looked up again, which could potentially leave a lot of undetected invalid entries in the cache taking up space which could cause other entries to be evicted. To remove these entries in a more timely fashion, have a background thread to periodically check the notification events for updates to the tables used in the results cache. > Results cache invalidation based on table updates > - > > Key: HIVE-18609 > URL: https://issues.apache.org/jira/browse/HIVE-18609 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Priority: Major > > Look into using the materialized view invalidation mechanisms to > automatically invalidate queries in the results cache if the underlying > tables used in the cached queries have been modified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
