[
https://issues.apache.org/jira/browse/IMPALA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238358#comment-17238358
]
Tim Armstrong commented on IMPALA-10216:
----------------------------------------
So from the failed assertion we have:
{noformat}
TmpFilePaths(pages[NO_ERROR_QUERY])
/tmp/buffer-pool-test.1/impala-scratch/0000000000000000:0000000000000000_3083829c-75fc-4c44-812a-bb5a4c067c88
/tmp/buffer-pool-test.1/impala-scratch/0000000000000000:0000000000000000_3083829c-75fc-4c44-812a-bb5a4c067c88
DumpScratchDir(error_dir)
/tmp/buffer-pool-test.0/impala-scratch:
0000000000000000:0000000000000000_b049d991-adb2-4845-991f-2dc287c0c90a
0000000000000000:0000000000000000_20081b32-bef6-4909-8bff-4fd61ec8ca0d
{noformat}
buffer-pool-test.INFO for reference (it looks normal)
{noformat}
I1123 23:09:20.500730 9357 tmp-file-mgr.cc:254] Using scratch directory
/tmp/buffer-pool-test.0/impala-scratch on disk 0 limit: 8589934592.00 GB
I1123 23:09:20.506762 9357 tmp-file-mgr.cc:254] Using scratch directory
/tmp/buffer-pool-test.1/impala-scratch on disk 0 limit: 8589934592.00 GB
I1123 23:09:20.545521 9357 buffer-pool-test.cc:1727] Manager 0 Block 0 backed
by file
/tmp/buffer-pool-test.0/impala-scratch/0000000000000000:0000000000000000_b049d991-adb2-4845-991f-2dc287c0c90a
I1123 23:09:20.545562 9357 buffer-pool-test.cc:1727] Manager 0 Block 1 backed
by file
/tmp/buffer-pool-test.1/impala-scratch/0000000000000000:0000000000000000_cd33af91-4e27-412a-ae45-f16061b5ef56
I1123 23:09:20.556377 9357 buffer-pool-test.cc:1727] Manager 1 Block 0 backed
by file
/tmp/buffer-pool-test.1/impala-scratch/0000000000000000:0000000000000000_3083829c-75fc-4c44-812a-bb5a4c067c88
I1123 23:09:20.556414 9357 buffer-pool-test.cc:1727] Manager 1 Block 1 backed
by file
/tmp/buffer-pool-test.0/impala-scratch/0000000000000000:0000000000000000_20081b32-bef6-4909-8bff-4fd61ec8ca0d
I1123 23:09:20.558903 9357 buffer-pool-test.cc:336] Injected fault by removing
file permissions
/tmp/buffer-pool-test.0/impala-scratch/0000000000000000:0000000000000000_b049d991-adb2-4845-991f-2dc287c0c90a
E1123 23:09:20.559283 451 tmp-file-mgr.cc:359] Error for temporary file
'/tmp/buffer-pool-test.0/impala-scratch/0000000000000000:0000000000000000_b049d991-adb2-4845-991f-2dc287c0c90a':
Disk I/O error on
impala-ec2-centos74-m5-4xlarge-ondemand-1898.vpc.cloudera.com:27000: open()
failed for
/tmp/buffer-pool-test.0/impala-scratch/0000000000000000:0000000000000000_b049d991-adb2-4845-991f-2dc287c0c90a.
Access denied for the process' user errno=13
I1123 23:09:20.563634 9357 buffer-pool-test.cc:1781] Newly created page backed
by file
/tmp/buffer-pool-test.1/impala-scratch/0000000000000000:0000000000000000_cd33af91-4e27-412a-ae45-f16061b5ef56
I1123 23:09:20.563666 9357 buffer-pool-test.cc:1781] Newly created page backed
by file
/tmp/buffer-pool-test.1/impala-scratch/0000000000000000:0000000000000000_cd33af91-4e27-412a-ae45-f16061b5ef56
I1123 23:09:20.606637 9357 krpc-data-stream-mgr.cc:436] Waiting for
data-stream-mgr maintenance thread...
I1123 23:09:20.606671 9357 krpc-data-stream-mgr.cc:438] Waiting for
deserialization thread pool...
{noformat}
It's weird that both pages got backed by the same temp dir, because the
allocation should be round-robin. II looked at the code and I do think there's
some actual non-determinism here because of the way Pin() runs asynchronously
until the actual buffer is requested - it may not actually complete for one of
these pages. I think if one thread went down this code path on one page but not
the other, the allocation may not be round robin -
https://github.com/apache/impala/blob/master/be/src/runtime/bufferpool/buffer-pool.cc#L207
> BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
> ----------------------------------------------------------------------
>
> Key: IMPALA-10216
> URL: https://issues.apache.org/jira/browse/IMPALA-10216
> Project: IMPALA
> Issue Type: Bug
> Reporter: Sahil Takiar
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: flaky
> Attachments: LastTest.log, LastTestsFailed.log,
> buffer-pool-test.ERROR, buffer-pool-test.INFO, buffer-pool-test.WARNING,
> buffer-pool-test.xml, impala-cdpd-master-staging-core-tsan_111.tar.gz
>
>
> Only seen this once so far:
> {code}
> BufferPoolTest.WriteErrorBlacklistCompression
> Error Message
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> Stacktrace
> Impala/be/src/runtime/bufferpool/buffer-pool-test.cc:1764
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
