[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for 
some scratch failures.
..


Patch Set 3: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for 
some scratch failures.
..


IMPALA-3642: Adding backend addresses to error statuses for some scratch 
failures.

Adds GetBackendAddress() (which is host:port) to error messages stemming
from SCRATCH_LIMIT_EXCEEDED, SCRATCH_READ_TRUNCATED, and
SCRATCH_ALLOCATION_FAILED messages.

Testing:

* Unit tests assert the string is updated for SCRATCH_LIMIT_EXCEEDED
  and SCRATCH_ALLOCATION_FAILED. SCRATCH_READ_TRUNCATED doesn't
  have an existing test, and I didn't add a new one.

* Manually testing a query that spills after "chmod 000 /tmp/impala-scratch":
$ chmod 000 /tmp/impala-scratch
$ impala-shell
[dev:21000] > set mem_limit=100m;
MEM_LIMIT set to 100m
[dev:21000] > select count(*) from tpch_parquet.lineitem join 
tpch_parquet.orders on l_orderkey = o_orderkey;
Query: select count(*) from tpch_parquet.lineitem join tpch_parquet.orders 
on l_orderkey = o_orderkey
Query submitted at: 2017-09-11 11:07:06 (Coordinator: http://dev:25000)
Query progress can be monitored at: 
http://dev:25000/query_plan?query_id=5c48ff8f4103c194:1b40a6c
WARNINGS: Could not create files in any configured scratch directories 
(--scratch_dirs=/tmp/impala-scratch) on backend 'dev:22002'. See logs for 
previous errors that may have prevented creating or writing scratch files.
Opening 
'/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c_08e8d63b-169d-4571-a0fe-c48fa08d73e6'
 for write failed with errno=13 description=Error(13): Permission denied
Opening 
'/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c_08e8d63b-169d-4571-a0fe-c48fa08d73e6'
 for write failed with errno=13 description=Error(13): Permission denied
Opening 
'/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c_08e8d63b-169d-4571-a0fe-c48fa08d73e6'
 for write failed with errno=13 description=Error(13): Permission denied

Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Reviewed-on: http://gerrit.cloudera.org:8080/7816
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins
---
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/tmp-file-mgr-test.cc
M be/src/runtime/tmp-file-mgr.cc
M common/thrift/generate_error_codes.py
4 files changed, 13 insertions(+), 9 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Tim Armstrong: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for 
some scratch failures.
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1212/

-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for 
some scratch failures.
..


Patch Set 3: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for 
some scratch failures.
..


Patch Set 2: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has uploaded a new patch set (#2).

Change subject: IMPALA-3642: Adding backend addresses to error statuses for 
some scratch failures.
..

IMPALA-3642: Adding backend addresses to error statuses for some scratch 
failures.

Adds GetBackendAddress() (which is host:port) to error messages stemming
from SCRATCH_LIMIT_EXCEEDED, SCRATCH_READ_TRUNCATED, and
SCRATCH_ALLOCATION_FAILED messages.

Testing:

* Unit tests assert the string is updated for SCRATCH_LIMIT_EXCEEDED
  and SCRATCH_ALLOCATION_FAILED. SCRATCH_READ_TRUNCATED doesn't
  have an existing test, and I didn't add a new one.

* Manually testing a query that spills after "chmod 000 /tmp/impala-scratch":
$ chmod 000 /tmp/impala-scratch
$ impala-shell
[dev:21000] > set mem_limit=100m;
MEM_LIMIT set to 100m
[dev:21000] > select count(*) from tpch_parquet.lineitem join 
tpch_parquet.orders on l_orderkey = o_orderkey;
Query: select count(*) from tpch_parquet.lineitem join tpch_parquet.orders 
on l_orderkey = o_orderkey
Query submitted at: 2017-09-11 11:07:06 (Coordinator: http://dev:25000)
Query progress can be monitored at: 
http://dev:25000/query_plan?query_id=5c48ff8f4103c194:1b40a6c
WARNINGS: Could not create files in any configured scratch directories 
(--scratch_dirs=/tmp/impala-scratch) on backend 'dev:22002'. See logs for 
previous errors that may have prevented creating or writing scratch files.
Opening 
'/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c_08e8d63b-169d-4571-a0fe-c48fa08d73e6'
 for write failed with errno=13 description=Error(13): Permission denied
Opening 
'/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c_08e8d63b-169d-4571-a0fe-c48fa08d73e6'
 for write failed with errno=13 description=Error(13): Permission denied
Opening 
'/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c_08e8d63b-169d-4571-a0fe-c48fa08d73e6'
 for write failed with errno=13 description=Error(13): Permission denied

Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
---
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/tmp-file-mgr-test.cc
M be/src/runtime/tmp-file-mgr.cc
M common/thrift/generate_error_codes.py
4 files changed, 13 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/7816/2
-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong