> On Jan. 29, 2018, 6:26 p.m., Jesús Camacho Rodríguez wrote: > > ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java > > Lines 174 (patched) > > <https://reviews.apache.org/r/65304/diff/3/?file=1946568#file1946568line174> > > > > What happens if execution fails? Will the results still be cleaned > > properly? > > Jason Dere wrote: > This is being called in Driver.closeInProcess()/Driver.close(), so my > impression was this should be the case. > > Jesús Camacho Rodríguez wrote: > Makes sense. What if HS2 fails or closes? Should we add it to the > shutdown hook so it cleans it even if readers number is greater than zero? > (Not sure about the order in which things happen in this case, just want to > make sure we do not leave any residual data/dirs). > > Jason Dere wrote: > The QueryResultsCache initialization calls deleteOnExit() for the the > results cache directory, so all subdirectories of this directory > (representing cached results) should be deleted on shutdown regardless of the > number of readers. > > One thing to improve here might be a way to cleanup any previous Hive > instances where the process exited abnormally and may not have been able to > run the shutdown hook. I'll open another item for that.
Sounds good, please do that. - Jesús ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65304/#review196440 ----------------------------------------------------------- On Feb. 2, 2018, 2:18 a.m., Jason Dere wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65304/ > ----------------------------------------------------------- > > (Updated Feb. 2, 2018, 2:18 a.m.) > > > Review request for hive, Ashutosh Chauhan, Gopal V, and Jesús Camacho > Rodríguez. > > > Bugs: HIVE-18513 > https://issues.apache.org/jira/browse/HIVE-18513 > > > Repository: hive-git > > > Description > ------- > > - For queries that result in MR/Tez/Spark jobs on the cluster, save the > temporary query results to a cache directory where they can be re-used. > - Add QueryResultsCache to manage cached results. Currently cache > invalidation is time-based, update-based cache invalidation needs to be added > later. > - Driver/SemanticAnalyzer/Calcite planner changes to lookup queries in the > cache and use in place of the query plan. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99 > common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 2767bca > data/conf/hive-site.xml 01f83d1 > data/conf/llap/hive-site.xml cdda875 > data/conf/perf-reg/spark/hive-site.xml 497a61f > data/conf/perf-reg/tez/hive-site.xml 012369f > data/conf/rlist/hive-site.xml 9de00e5 > data/conf/spark/local/hive-site.xml fd0e6a0 > data/conf/spark/standalone/hive-site.xml 1e5bd65 > data/conf/spark/yarn-client/hive-site.xml a9a788b > data/conf/tez/hive-site.xml 4519678 > itests/hive-blobstore/src/test/resources/hive-site.xml 038db0d > itests/src/test/resources/testconfiguration.properties d86ff58 > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 4432aca > ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b0 > ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7348faa > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java > f0dd167 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java > 372cfad > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 85a1f34 > ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java ae2ec3d > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java dbf9363 > ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 7243dc7 > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 3946d4a > ql/src/test/queries/clientpositive/results_cache_1.q PRE-CREATION > ql/src/test/queries/clientpositive/results_cache_2.q PRE-CREATION > ql/src/test/queries/clientpositive/results_cache_capacity.q PRE-CREATION > ql/src/test/queries/clientpositive/results_cache_lifetime.q PRE-CREATION > ql/src/test/queries/clientpositive/results_cache_temptable.q PRE-CREATION > ql/src/test/queries/clientpositive/results_cache_with_masking.q > PRE-CREATION > ql/src/test/results/clientpositive/llap/results_cache_1.q.out PRE-CREATION > ql/src/test/results/clientpositive/results_cache_1.q.out PRE-CREATION > ql/src/test/results/clientpositive/results_cache_2.q.out PRE-CREATION > ql/src/test/results/clientpositive/results_cache_capacity.q.out > PRE-CREATION > ql/src/test/results/clientpositive/results_cache_lifetime.q.out > PRE-CREATION > ql/src/test/results/clientpositive/results_cache_temptable.q.out > PRE-CREATION > ql/src/test/results/clientpositive/results_cache_with_masking.q.out > PRE-CREATION > service/src/java/org/apache/hive/service/server/HiveServer2.java 2a528cd > > > Diff: https://reviews.apache.org/r/65304/diff/6/ > > > Testing > ------- > > qfile tests added. > > > Thanks, > > Jason Dere > >
