[
https://issues.apache.org/jira/browse/IMPALA-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073736#comment-17073736
]
ASF subversion and git services commented on IMPALA-7138:
---------------------------------------------------------
Commit 7abbde3c5d18217a84979454fffad1887e10816a in impala's branch
refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7abbde3 ]
IMPALA-7138: detect devicemapper volumes correctly
This is a more general fix for device detection that
disables the digit-stripping heuristic if a device
with the same name as a partition already exists.
E.g. for device-mapper devices, /proc/partitions might
include a partition called "dm-0" that maps directly
to the block device "dm-0" (i.e. /sys/block/dm-0).
Before this fix, we looked for a device called "dm-",
which did not exist.
Testing:
Created a dummy device-mapper device with the following command:
sudo dmsetup create 1gb-zero --table '0 1953125 zero'
Then confirmed that the device name in the logs was correct
dm-0 (rotational=true)
Other devices - sda, sda1, sda2, sdb, sdb1, etc on my system
were still detected the same as before.
Change-Id: I9d231bcf9105db8d4ab03586eab74e0644337a6f
Reviewed-on: http://gerrit.cloudera.org:8080/15631
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Fix detection and handling of Device Mapper volumes
> ---------------------------------------------------
>
> Key: IMPALA-7138
> URL: https://issues.apache.org/jira/browse/IMPALA-7138
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.0, Impala 2.12.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: resource-management
>
> I've heard a couple of instances of Impala doing the wrong thing when
> multiple scratch directories are present on a single DeviceMapper logical
> volume. E.g. in the following setup, we only use one directory as a disk.
> {noformat}
> >> mount list
> /dev/mapper/xx01-lv_data01 on /data01 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx02-lv_data02 on /data02 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx03-lv_data03 on /data03 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx04-lv_data04 on /data04 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx05-lv_data05 on /data05 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx06-lv_data06 on /data06 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx07-lv_data07 on /data07 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx08-lv_data08 on /data08 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx09-lv_data09 on /data09 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx10-lv_data10 on /data10 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx11-lv_data11 on /data11 type ext4 (rw,relatime,data=ordered)
> /dev/mapper/xx12-lv_data12 on /data12 type ext4 (rw,relatime,data=ordered)
> Scratch dirs are:
> /data01/impala/impalad,/data02/impala/impalad,/data03/impala/impalad,/data04/impala/impalad,/data05/impala/impalad,/data06/impala/impalad,/data07/impala/impalad,/data08/impala/impalad,/data09/impala/impalad,/data10/impala/impalad,/data11/impala/impalad,/data12/impala/impalad
> ~~~ tmp-file-mgr.cc:122] Using scratch directory
> /data01/impala/impalad/impala-scratch on disk13.
> {noformat}
> A workaround is to set --allow_multiple_scratch_dirs_per_device=true
> There are a few questions here:
> # Does the deduplication logic even make sense to have?
> # Do we need to fix how we treat Device mapper volumes?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
