[
https://issues.apache.org/jira/browse/HIVE-25976?focusedWorklogId=784083&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784083
]
ASF GitHub Bot logged work on HIVE-25976:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Jun/22 08:53
Start Date: 23/Jun/22 08:53
Worklog Time Spent: 10m
Work Description: pvary commented on code in PR #3289:
URL: https://github.com/apache/hive/pull/3289#discussion_r904754762
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java:
##########
@@ -134,39 +121,18 @@ public void setMaxRows(int maxRows) {
this.maxRows = maxRows;
}
- public boolean fetch(List res) throws IOException {
- sink.reset(res);
- int rowsRet = work.getLeastNumRows();
- if (rowsRet <= 0) {
- rowsRet = work.getLimit() >= 0 ? Math.min(work.getLimit() - totalRows,
maxRows) : maxRows;
- }
- try {
- if (rowsRet <= 0 || work.getLimit() == totalRows) {
- fetch.clearFetchContext();
+ public boolean fetch(List res) {
+ if (cachingEnabled) {
+ if (currentRow >= fetchedData.size()) {
return false;
}
- boolean fetched = false;
- while (sink.getNumRows() < rowsRet) {
- if (!fetch.pushRow()) {
- if (work.getLeastNumRows() > 0) {
- throw new HiveException("leastNumRows check failed");
- }
-
- // Closing the operator can sometimes yield more rows (HIVE-11892)
- fetch.closeOperator();
-
- return fetched;
- }
- fetched = true;
- }
- return true;
- } catch (IOException e) {
- throw e;
- } catch (Exception e) {
- throw new IOException(e);
- } finally {
- totalRows += sink.getNumRows();
+ int toIndex = Math.min(fetchedData.size(), currentRow + maxRows);
+ res.addAll(fetchedData.subList(currentRow, toIndex));
+ currentRow = toIndex;
+ } else {
+ executeInner(res);
Review Comment:
Wasn't this planned as:
```
return executeInner(res);
```
?
Issue Time Tracking
-------------------
Worklog Id: (was: 784083)
Time Spent: 50m (was: 40m)
> Cleaner may remove files being accessed from a fetch-task-converted reader
> --------------------------------------------------------------------------
>
> Key: HIVE-25976
> URL: https://issues.apache.org/jira/browse/HIVE-25976
> Project: Hive
> Issue Type: Bug
> Reporter: Zoltan Haindrich
> Assignee: László Végh
> Priority: Major
> Labels: pull-request-available
> Attachments: fetch_task_conv_compactor_test.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> in a nutshell the following happens:
> * query is compiled in fetch-task-converted mode
> * no real execution happens....but the locks are released
> * the HS2 is communicating with the client and uses the fetch-task to get the
> rows - which in this case will directly read files from the table's
> directory....
> * client sleeps between reads - so there is ample time for other events...
> * cleaner wakes up and removes some files....
> * in the next read the fetch-task encounters a read error...
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
