[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-26 Thread Joe McDonnell (Code Review) 
Joe McDonnell has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/12557 )

Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..

IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

DiskInfo::GetDeviceNames() assumes a device name pattern where
the last digits can be removed to get the main disk device
(i.e. sda3 -> sda). NVME devices have a naming pattern of
nvme{device_id}n{namespace_id}p{partition_id} where the main
disk device removes the p{partition_id} part. This difference
in naming prevents DiskInfo::GetDeviceNames() from looking
up the associated information in /sys/block to find whether
it is a rotational device (and thus use an appropriate
number of DiskIo threads).

This adds a condition to detect an NVME device name and
handle it correctly.

Testing:
 - Added unit test for the function used for detection
 - Tested this on an AWS m5-4xlarge, which is a Nitro instance
   type that exposes the main disk as an NVME device.

Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Reviewed-on: http://gerrit.cloudera.org:8080/12557
Tested-by: Impala Public Jenkins 
Reviewed-by: Philip Zeyliger 
---
M be/src/util/disk-info.cc
M be/src/util/disk-info.h
M be/src/util/sys-info-test.cc
3 files changed, 79 insertions(+), 3 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Philip Zeyliger: Looks good to me, approved

--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zoram Thanga

[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-26 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12557 )

Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Tue, 26 Feb 2019 23:30:51 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-26 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12557 )

Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..


Patch Set 3: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Tue, 26 Feb 2019 23:27:48 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-26 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12557 )

Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3833/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Tue, 26 Feb 2019 19:09:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-22 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12557 )

Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2221/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Sat, 23 Feb 2019 01:19:25 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-22 Thread Joe McDonnell (Code Review) 
Hello Philip Zeyliger, Zoram Thanga, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12557

to look at the new patch set (#2).

Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..

IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

DiskInfo::GetDeviceNames() assumes a device name pattern where
the last digits can be removed to get the main disk device
(i.e. sda3 -> sda). NVME devices have a naming pattern of
nvme{device_id}n{namespace_id}p{partition_id} where the main
disk device removes the p{partition_id} part. This difference
in naming prevents DiskInfo::GetDeviceNames() from looking
up the associated information in /sys/block to find whether
it is a rotational device (and thus use an appropriate
number of DiskIo threads).

This adds a condition to detect an NVME device name and
handle it correctly.

Testing:
 - Added unit test for the function used for detection
 - Tested this on an AWS m5-4xlarge, which is a Nitro instance
   type that exposes the main disk as an NVME device.

Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
---
M be/src/util/disk-info.cc
M be/src/util/disk-info.h
M be/src/util/sys-info-test.cc
3 files changed, 79 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/12557/2
--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zoram Thanga

[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-22 Thread Joe McDonnell (Code Review) 
Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12557 )

Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..


Patch Set 1:

> Could you extract the function and add a unit test that the regexp
 > does what it's supposed to?
 >
 > Thanks!

Done


--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Sat, 23 Feb 2019 00:50:28 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-22 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12557 )

Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..


Patch Set 1:

Could you extract the function and add a unit test that the regexp does what 
it's supposed to?

Thanks!


--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Fri, 22 Feb 2019 21:15:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-22 Thread Zoram Thanga (Code Review) 
Zoram Thanga has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12557 )

Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..


Patch Set 1: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Fri, 22 Feb 2019 19:39:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-22 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12557 )

Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2217/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 22 Feb 2019 19:29:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

2019-02-22 Thread Joe McDonnell (Code Review) 
Joe McDonnell has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/12557


Change subject: IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
..

IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks

DiskInfo::GetDeviceNames() assumes a device name pattern where
the last digits can be removed to get the main disk device
(i.e. sda3 -> sda). NVME devices have a naming pattern of
nvme{device_id}n{namespace_id}p{partition_id} where the main
disk device removes the p{partition_id} part. This difference
in naming prevents DiskInfo::GetDeviceNames() from looking
up the associated information in /sys/block to find whether
it is a rotational device (and thus use an appropriate
number of DiskIo threads).

This adds a condition to detect an NVME device name and
handle it correctly.

Tested this on an AWS m5-4xlarge, which is a Nitro instance
type that exposes the main disk as an NVME device.

Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
---
M be/src/util/disk-info.cc
1 file changed, 16 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/12557/1
--
To view, visit http://gerrit.cloudera.org:8080/12557
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Gerrit-Change-Number: 12557
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell