Lars Volker has posted comments on this change. Change subject: IMPALA-4223: Handle truncated file read from HDFS cache ......................................................................
Patch Set 1: (1 comment) > (1 comment) > > > (1 comment) > > > > > (1 comment) > > > > > > Can we test this by having the test load metadata and then > > truncate > > > a cached file? > > > > We can try to do this in a custom cluster test. It needs to > follow > > the steps outlined here: > > https://issues.cloudera.org/browse/IMPALA-4223?focusedCommentId=212776&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-212776 > > > > They require changes to the system limits to allow for larger > > cached files, and to the data nodes to increase the file cache as > > well, so they might be rather disruptive. Once these are changed > we > > can write a custom cluster test to download, truncate, and upload > > files and run queries over them, checking that the correct log > > messages appear. > > > > Should we break this out into several Jiras / changes? The limits > > should be change in impala-setup, too. The datanode settings > change > > will be required to integrate this into fuzz testing, too. > > > > I will also have a look at the scanner fuzz test and see if it is > > easy to inject these error there. > > Yes, if it's a good amount of work to set up that kind of test, > let's save it for another step. It's probably better to spend time > with getting the fuzzer working rather than this one particular > test case. I Opened IMPALA-4394 and IMPALA-4395 to track both adding tests for this change and adding Hdfs caching to the fuzz tests. http://gerrit.cloudera.org:8080/#/c/4828/1/be/src/runtime/disk-io-mgr-scan-range.cc File be/src/runtime/disk-io-mgr-scan-range.cc: PS1, Line 448: disk > Okay. I'm fine either leaving this one as "disk" or changing both. Thanks. I'll leave it to keep any tooling that searches for this working. -- To view, visit http://gerrit.cloudera.org:8080/4828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id1e1fdb0211819c5938956abb13b512350a46f1a Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Lars Volker <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-HasComments: Yes
