[
https://issues.apache.org/jira/browse/HIVE-25976?focusedWorklogId=785080&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785080
]
ASF GitHub Bot logged work on HIVE-25976:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Jun/22 12:56
Start Date: 27/Jun/22 12:56
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3289:
URL: https://github.com/apache/hive/pull/3289#discussion_r907347860
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java:
##########
@@ -152,28 +152,32 @@ private FetchTask optimize(ParseContext pctx, String
alias, TableScanOperator so
}
private boolean checkThreshold(FetchData data, int limit, ParseContext pctx)
throws Exception {
- if (limit > 0) {
- if (data.hasOnlyPruningFilter()) {
- /* partitioned table + query has only pruning filters */
- return true;
- } else if (data.isPartitioned() == false && data.isFiltered() == false) {
- /* unpartitioned table + no filters */
- return true;
+ boolean cachingEnabled = HiveConf.getBoolVar(pctx.getConf(),
HiveConf.ConfVars.HIVEFETCHTASKCACHING);
+ if (!cachingEnabled) {
+ /* if caching is enabled we apply the treshold in all cases */
Review Comment:
The `cachingEnabled` variable name and the comment suggest that true means
caching is enabled but the `if` predicate is negated. Does is means that this
branch should run when caching is disabled?
##########
ql/src/test/results/clientpositive/llap/nonmr_fetch_threshold.q.out:
##########
@@ -115,12 +115,15 @@ Plan optimized by CBO.
Stage-0
Fetch Operator
limit:10
- Select Operator [SEL_2]
- Output:["_col0","_col1","_col2","_col3"]
- Limit [LIM_3]
- Number of rows:10
- TableScan [TS_0]
- Output:["key","value"]
+ Stage-1
+ Map 1 vectorized, llap
+ File Output Operator [FS_8]
Review Comment:
It seems that this plan has a Map phase. Does this expected?
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java:
##########
@@ -152,28 +152,32 @@ private FetchTask optimize(ParseContext pctx, String
alias, TableScanOperator so
}
private boolean checkThreshold(FetchData data, int limit, ParseContext pctx)
throws Exception {
- if (limit > 0) {
- if (data.hasOnlyPruningFilter()) {
- /* partitioned table + query has only pruning filters */
- return true;
- } else if (data.isPartitioned() == false && data.isFiltered() == false) {
- /* unpartitioned table + no filters */
- return true;
+ boolean cachingEnabled = HiveConf.getBoolVar(pctx.getConf(),
HiveConf.ConfVars.HIVEFETCHTASKCACHING);
+ if (!cachingEnabled) {
+ /* if caching is enabled we apply the treshold in all cases */
+ if (limit > 0) {
+ if (data.hasOnlyPruningFilter()) {
+ /* partitioned table + query has only pruning filters */
Review Comment:
nit: I thinks we should use `//` for one line comments.
Issue Time Tracking
-------------------
Worklog Id: (was: 785080)
Time Spent: 1h 20m (was: 1h 10m)
> Cleaner may remove files being accessed from a fetch-task-converted reader
> --------------------------------------------------------------------------
>
> Key: HIVE-25976
> URL: https://issues.apache.org/jira/browse/HIVE-25976
> Project: Hive
> Issue Type: Bug
> Reporter: Zoltan Haindrich
> Assignee: László Végh
> Priority: Major
> Labels: pull-request-available
> Attachments: fetch_task_conv_compactor_test.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> in a nutshell the following happens:
> * query is compiled in fetch-task-converted mode
> * no real execution happens....but the locks are released
> * the HS2 is communicating with the client and uses the fetch-task to get the
> rows - which in this case will directly read files from the table's
> directory....
> * client sleeps between reads - so there is ample time for other events...
> * cleaner wakes up and removes some files....
> * in the next read the fetch-task encounters a read error...
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
