[jira] [Commented] (KUDU-1334) Support pid_max > 16 bits in the mini cluster
[
https://issues.apache.org/jira/browse/KUDU-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065165#comment-17065165
]
ASF subversion and git services commented on KUDU-1334:
---
Commit ee86f568cb405f8636ae56f3b69f8009f8f481da in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=ee86f56 ]
[util] GetBindIpForDaemon() consumes PID wider than 18 bit
This patch updates the logic of GetBindIpForDaemon() to accept process
identifiers wider than 18 bits. Prior to this patch, maximum allowable
PID was 262143, and feeding in a higher PID lead to a crash. The
limitation was due to the mapping scheme of (PID, index) --> loopback
IP address: it mapped 18-bit PID and 6-bit index into the 24-bit space
of 127.0.0.0/8 loopback addresses.
With this patch, GetBindIpForDaemon() stops imposing the limit on the
maximum PID. Now the uniqueness of the generated IP address is not
guaranteed in general if OS generates PIDs wider than 18 bits, but it
still holds across all Kudu test processes if both of the following
conditions are true:
* PIDs are monotonically increasing with a delta D which is much
smaller than 262143 for newly spawned processes (usually D is 1)
* no more than 262144/D processes are spawned between the earliest
still running process and the latest running one while executing
Kudu tests involving external mini-cluster
The former item is not true for hardened Linux distros where PID
assignment is randomized, but it's unlikely that anybody is interested
running Kudu tests on such systems.
Essentially, the new scheme uses PID % 262144 as input for the PID part,
keeping it in the 18-bit space even if the original PID address space is
wider (it might be up to 22 bits wide for 64-bit Linux distros).
This patch helps running Kudu tests on Docker containers where the host
OS is configured to have PIDs higher than 262143 and updating maximum
PID is not an option since /proc/sys/kernel/pid_max is mounted
read-only.
This patch relates to the following JIRA issues:
KUDU-1334
KUDU-3044
Change-Id: Idf2162a2a976a0ce997f02a321f230e7f5878328
Reviewed-on: http://gerrit.cloudera.org:8080/15519
Reviewed-by: Adar Dembo
Tested-by: Kudu Jenkins
> Support pid_max > 16 bits in the mini cluster
> -
>
> Key: KUDU-1334
> URL: https://issues.apache.org/jira/browse/KUDU-1334
> Project: Kudu
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.5.0
>Reporter: Jean-Daniel Cryans
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.6.0
>
>
> Pretty much anybody running on newer machines/platforms will hit this while
> running the unit tests:
> {noformat}
> I0216 11:27:23.617383 110702 external_mini_cluster.cc:582] Started
> /home/stack/apache-kudu-incubating-0.7.0/build/rc/bin/kudu-master as pid
> 110706
> F0216 11:27:23.617473 110702 external_mini_cluster.cc:258] Check failed: p
> <= MathLimits::kMax (110702 vs. 65535) Cannot run on systems with
> >16-bit pid
> *** Check failure stack trace: ***
> {noformat}
> Having this limitation was fine but now it's something everybody hits.
> The workaround is running this:
> {noformat}
> echo "32768" > /proc/sys/kernel/pid_max
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-1334) Support pid_max > 16 bits in the mini cluster
[
https://issues.apache.org/jira/browse/KUDU-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884240#comment-16884240
]
Alexey Serbin commented on KUDU-1334:
-
Ah, I see. I haven't looked at IPv6 yet. Probably, using some sort of
workaround you mentioned for loopback interfaces should be fine. However, I
wouldn't be surprised if it were necessary to manually add/configure IPv6
addresses for loopbacks if using one of those approaches.
If you are constrained with the limitation of 18 bits for maximum PID, another
possible solution might be not using {{UNIQUE_LOOPBACK}} mode, but that would
limit the concurrency while running the tests to avoid conflicts (you can see
how it's done in case of macOS). That's not the end of the world, but if you
hit the necessity to configure 'local' IPv6 addresses for network interfaces
anyways, instead of adding/removing addresses (under super-user creds) you
might consider switching to other than UNIQUE_LOOPBACK modes. Ideally, it
would be nice to have UNIQUE_LOOPBACK mode working in IPv6 case on Linux as
well, but at this point I don't know whether it's possible at all.
Anyway, thank you for your work on KUDU-1457 !
> Support pid_max > 16 bits in the mini cluster
> -
>
> Key: KUDU-1334
> URL: https://issues.apache.org/jira/browse/KUDU-1334
> Project: Kudu
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.5.0
>Reporter: Jean-Daniel Cryans
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.6.0
>
>
> Pretty much anybody running on newer machines/platforms will hit this while
> running the unit tests:
> {noformat}
> I0216 11:27:23.617383 110702 external_mini_cluster.cc:582] Started
> /home/stack/apache-kudu-incubating-0.7.0/build/rc/bin/kudu-master as pid
> 110706
> F0216 11:27:23.617473 110702 external_mini_cluster.cc:258] Check failed: p
> <= MathLimits::kMax (110702 vs. 65535) Cannot run on systems with
> >16-bit pid
> *** Check failure stack trace: ***
> {noformat}
> Having this limitation was fine but now it's something everybody hits.
> The workaround is running this:
> {noformat}
> echo "32768" > /proc/sys/kernel/pid_max
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KUDU-1334) Support pid_max > 16 bits in the mini cluster
[
https://issues.apache.org/jira/browse/KUDU-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884208#comment-16884208
]
Ritwik Yadav commented on KUDU-1334:
I am using CentOS 7. And of course, putting a limit on it would solve that for
me. I believe that was the resolution before as well.
Thanks, I wasn't aware of the limitation on BSD. I am working on the task
KUDU-1457 to add IPv6 compatibility. I was using existing unit tests to test my
patch and was trying to figure out why tests were failing on master which
brought me here. I will try to use mapped addresses and see if it works.
> Support pid_max > 16 bits in the mini cluster
> -
>
> Key: KUDU-1334
> URL: https://issues.apache.org/jira/browse/KUDU-1334
> Project: Kudu
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.5.0
>Reporter: Jean-Daniel Cryans
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.6.0
>
>
> Pretty much anybody running on newer machines/platforms will hit this while
> running the unit tests:
> {noformat}
> I0216 11:27:23.617383 110702 external_mini_cluster.cc:582] Started
> /home/stack/apache-kudu-incubating-0.7.0/build/rc/bin/kudu-master as pid
> 110706
> F0216 11:27:23.617473 110702 external_mini_cluster.cc:258] Check failed: p
> <= MathLimits::kMax (110702 vs. 65535) Cannot run on systems with
> >16-bit pid
> *** Check failure stack trace: ***
> {noformat}
> Having this limitation was fine but now it's something everybody hits.
> The workaround is running this:
> {noformat}
> echo "32768" > /proc/sys/kernel/pid_max
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KUDU-1334) Support pid_max > 16 bits in the mini cluster
[
https://issues.apache.org/jira/browse/KUDU-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884200#comment-16884200
]
Alexey Serbin commented on KUDU-1334:
-
As for IPv6 network addresses, that's a separate task to properly support that
in Kudu.
Even with IPv4, the provision of 127.0.0.0/8 addresses being automatically
available on loopback network interfaces is a special provision of linux
kernerls, AFAIK: that's not available on BSD systems. So, most likely with
IPv6 on loopback interfaces for Linux kernels there might be some options
available, yep: it would be necessary to evaluate various options.
> Support pid_max > 16 bits in the mini cluster
> -
>
> Key: KUDU-1334
> URL: https://issues.apache.org/jira/browse/KUDU-1334
> Project: Kudu
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.5.0
>Reporter: Jean-Daniel Cryans
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.6.0
>
>
> Pretty much anybody running on newer machines/platforms will hit this while
> running the unit tests:
> {noformat}
> I0216 11:27:23.617383 110702 external_mini_cluster.cc:582] Started
> /home/stack/apache-kudu-incubating-0.7.0/build/rc/bin/kudu-master as pid
> 110706
> F0216 11:27:23.617473 110702 external_mini_cluster.cc:258] Check failed: p
> <= MathLimits::kMax (110702 vs. 65535) Cannot run on systems with
> >16-bit pid
> *** Check failure stack trace: ***
> {noformat}
> Having this limitation was fine but now it's something everybody hits.
> The workaround is running this:
> {noformat}
> echo "32768" > /proc/sys/kernel/pid_max
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KUDU-1334) Support pid_max > 16 bits in the mini cluster
[
https://issues.apache.org/jira/browse/KUDU-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884191#comment-16884191
]
Alexey Serbin commented on KUDU-1334:
-
[~ibp73]
Yeah, these days 64-bit Linux distros have a limit of 2^22 as the maximum PID,
even if pid_t is 32 bit. Also, by default most Linux distros usually limit
that by some lower value. I'm curious what's your distro, if that's Linux?
If that's a linux machine, it's easy to limit the pid_max:
{noformat}
echo 262144 > /proc/sys/kernel/pid_max
{noformat}
Would it work for you?
> Support pid_max > 16 bits in the mini cluster
> -
>
> Key: KUDU-1334
> URL: https://issues.apache.org/jira/browse/KUDU-1334
> Project: Kudu
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.5.0
>Reporter: Jean-Daniel Cryans
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.6.0
>
>
> Pretty much anybody running on newer machines/platforms will hit this while
> running the unit tests:
> {noformat}
> I0216 11:27:23.617383 110702 external_mini_cluster.cc:582] Started
> /home/stack/apache-kudu-incubating-0.7.0/build/rc/bin/kudu-master as pid
> 110706
> F0216 11:27:23.617473 110702 external_mini_cluster.cc:258] Check failed: p
> <= MathLimits::kMax (110702 vs. 65535) Cannot run on systems with
> >16-bit pid
> *** Check failure stack trace: ***
> {noformat}
> Having this limitation was fine but now it's something everybody hits.
> The workaround is running this:
> {noformat}
> echo "32768" > /proc/sys/kernel/pid_max
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KUDU-1334) Support pid_max > 16 bits in the mini cluster
[
https://issues.apache.org/jira/browse/KUDU-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884186#comment-16884186
]
Ritwik Yadav commented on KUDU-1334:
Talking about IPv6, the loopback address is simply ::1/128 in CIDR notation.
Therefore, there is only one possibility. However, with IPv4 loopback addresses
were denoted by 127.0.0.1/8 so we had 2^24 different choices.
There is a possible workaround that IPv4 addresses could be used as IPv6 as
detailed here:
[https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.hale001/ipv6d0031001726.htm]
> Support pid_max > 16 bits in the mini cluster
> -
>
> Key: KUDU-1334
> URL: https://issues.apache.org/jira/browse/KUDU-1334
> Project: Kudu
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.5.0
>Reporter: Jean-Daniel Cryans
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.6.0
>
>
> Pretty much anybody running on newer machines/platforms will hit this while
> running the unit tests:
> {noformat}
> I0216 11:27:23.617383 110702 external_mini_cluster.cc:582] Started
> /home/stack/apache-kudu-incubating-0.7.0/build/rc/bin/kudu-master as pid
> 110706
> F0216 11:27:23.617473 110702 external_mini_cluster.cc:258] Check failed: p
> <= MathLimits::kMax (110702 vs. 65535) Cannot run on systems with
> >16-bit pid
> *** Check failure stack trace: ***
> {noformat}
> Having this limitation was fine but now it's something everybody hits.
> The workaround is running this:
> {noformat}
> echo "32768" > /proc/sys/kernel/pid_max
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KUDU-1334) Support pid_max > 16 bits in the mini cluster
[
https://issues.apache.org/jira/browse/KUDU-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884178#comment-16884178
]
Ritwik Yadav commented on KUDU-1334:
[~aserbin]
Yeah. I have a lot of tests failing like this and in the code we are assuming
that the pid would fit into 18 bits which is not true for machines these days.
Right?
{code:java}
F0712 14:21:56.171644 1771822 net_util.cc:481] Check failed: pid < 1 <<
kPidBits (1771822 vs. 262144) PID 1771822 is more than 18 bits wide
*** Check failure stack trace: ***
*** Aborted at 1562966516 (unix time) try "date -d @1562966516" if you are
using GNU date ***
PC: @ 0x7f74d09b2207 __GI_raise
*** SIGABRT (@0x8bb27001b092e) received by PID 1771822 (TID 0x7f74d3031180)
from PID 1771822; stack trace: ***
@ 0x7f74d27a95d0 (unknown)
@ 0x7f74d09b2207 __GI_raise
@ 0x7f74d09b38f8 __GI_abort
@ 0xb3fed9 google::logging_fail()
@ 0xb416ad google::LogMessage::Fail()
@ 0xb43603 google::LogMessage::SendToLog()
@ 0xb41209 google::LogMessage::Flush()
@ 0xb43f8f google::LogMessageFatal::~LogMessageFatal()
@ 0x24e1d2c kudu::GetBindIpForDaemon()
@ 0xb9992d kudu::cluster::MiniCluster::ReserveDaemonSocket()
@ 0xb94898 kudu::cluster::InternalMiniCluster::StartMasters()
@ 0xb975dc kudu::cluster::InternalMiniCluster::Start()
@ 0xb3f7fb kudu::RpcLineItemDAOTest::SetUp()
@ 0x2209a69 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x21fb820 testing::Test::Run()
@ 0x21fb93d testing::TestInfo::Run()
@ 0x21fba55 testing::TestCase::Run()
@ 0x21fbd18 testing::internal::UnitTestImpl::RunAllTests()
@ 0x21fbfb9 testing::UnitTest::Run()
@ 0xae316d main
@ 0x7f74d099e3d5 __libc_start_main
@ 0xb3a016 (unknown)
[1] 1771822 abort (core dumped) ./build/release/bin/rpc_line_item_dao-test{code}
> Support pid_max > 16 bits in the mini cluster
> -
>
> Key: KUDU-1334
> URL: https://issues.apache.org/jira/browse/KUDU-1334
> Project: Kudu
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.5.0
>Reporter: Jean-Daniel Cryans
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.6.0
>
>
> Pretty much anybody running on newer machines/platforms will hit this while
> running the unit tests:
> {noformat}
> I0216 11:27:23.617383 110702 external_mini_cluster.cc:582] Started
> /home/stack/apache-kudu-incubating-0.7.0/build/rc/bin/kudu-master as pid
> 110706
> F0216 11:27:23.617473 110702 external_mini_cluster.cc:258] Check failed: p
> <= MathLimits::kMax (110702 vs. 65535) Cannot run on systems with
> >16-bit pid
> *** Check failure stack trace: ***
> {noformat}
> Having this limitation was fine but now it's something everybody hits.
> The workaround is running this:
> {noformat}
> echo "32768" > /proc/sys/kernel/pid_max
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KUDU-1334) Support pid_max > 16 bits in the mini cluster
[
https://issues.apache.org/jira/browse/KUDU-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884162#comment-16884162
]
Alexey Serbin commented on KUDU-1334:
-
The idea behind the bit manipulations in
https://github.com/apache/kudu/blob/c1d16856bd4b3bb4b6242fa098d600b6ef321e6a/src/kudu/util/net/net_util.cc#L472
can be seen clearer if looking at the usage of the utility method
https://github.com/apache/kudu/blob/c1d16856bd4b3bb4b6242fa098d600b6ef321e6a/src/kudu/mini-cluster/mini_cluster.cc#L37
in {{internal_mini_cluster.cc}} and {{external_mini_cluster.cc}}, assuming
it's all about {{UNIQUE_LOOPBACK}} bind mode.
Basically, all you need is a set of unique loopback IP addresses to bind Kudu
masters and tservers, so multiple tests that use internal/external mini cluster
can run in parallel at the same machine without port conflicts. Current
implementation allows for running about 2^18 (262144) Kudu processes at one
machine that uses IPv4 addresses for its network interfaces, that's about 4K
Kudu mini-clusters at once (each mini-cluster in that scheme can run at most 62
indexed daemons, meaning master and tservers). Those bits manipulations is
just a simple mapping of a process identifier (PID) and an index of a daemon
into a 32-bit number (IPv4 address).
I think that in case of using IPv6 addresses the restrictions for maximum
allowable PID is less strict since the target address space is much larger.
> Support pid_max > 16 bits in the mini cluster
> -
>
> Key: KUDU-1334
> URL: https://issues.apache.org/jira/browse/KUDU-1334
> Project: Kudu
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.5.0
>Reporter: Jean-Daniel Cryans
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.6.0
>
>
> Pretty much anybody running on newer machines/platforms will hit this while
> running the unit tests:
> {noformat}
> I0216 11:27:23.617383 110702 external_mini_cluster.cc:582] Started
> /home/stack/apache-kudu-incubating-0.7.0/build/rc/bin/kudu-master as pid
> 110706
> F0216 11:27:23.617473 110702 external_mini_cluster.cc:258] Check failed: p
> <= MathLimits::kMax (110702 vs. 65535) Cannot run on systems with
> >16-bit pid
> *** Check failure stack trace: ***
> {noformat}
> Having this limitation was fine but now it's something everybody hits.
> The workaround is running this:
> {noformat}
> echo "32768" > /proc/sys/kernel/pid_max
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KUDU-1334) Support pid_max > 16 bits in the mini cluster
[
https://issues.apache.org/jira/browse/KUDU-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884148#comment-16884148
]
Alexey Serbin commented on KUDU-1334:
-
I'm curious how you are hitting the limit of 18 bits of PID? Does you OS uses
process identifiers that do not fit into 18 bit number?
> Support pid_max > 16 bits in the mini cluster
> -
>
> Key: KUDU-1334
> URL: https://issues.apache.org/jira/browse/KUDU-1334
> Project: Kudu
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.5.0
>Reporter: Jean-Daniel Cryans
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.6.0
>
>
> Pretty much anybody running on newer machines/platforms will hit this while
> running the unit tests:
> {noformat}
> I0216 11:27:23.617383 110702 external_mini_cluster.cc:582] Started
> /home/stack/apache-kudu-incubating-0.7.0/build/rc/bin/kudu-master as pid
> 110706
> F0216 11:27:23.617473 110702 external_mini_cluster.cc:258] Check failed: p
> <= MathLimits::kMax (110702 vs. 65535) Cannot run on systems with
> >16-bit pid
> *** Check failure stack trace: ***
> {noformat}
> Having this limitation was fine but now it's something everybody hits.
> The workaround is running this:
> {noformat}
> echo "32768" > /proc/sys/kernel/pid_max
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KUDU-1334) Support pid_max > 16 bits in the mini cluster
[
https://issues.apache.org/jira/browse/KUDU-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884040#comment-16884040
]
Ritwik Yadav commented on KUDU-1334:
IPv4:
I am hitting the limit with 18 bits of PID. I guess we need a more scalable
solution than expanding the number of bits of allowable PIDs. I would like to
understand the exact problem here. I presume we have multiple processes running
tests on a single host. Each process needs a bunch of unique IPv4 addresses.
The problem boils down to constructing an injection from (pid_t, 6 bit value)
to a 24 bit value. Now, pid_t today would have 32 bits for most machines
running Kudu. However, for commonly used implementations we would contiguous
pid unless we hit the limit. I think a hash function with a check on
availability of that address etc. should be a durable solution.
The underlying assumption is the number of processes on machines would grow
faster than the number of unit tests requiring IP addresses on Kudu. What is
the degree of parallelism on tests we run and how can it be controlled.
IPv6:
As for IPv6 support, we can possibly use IPv6 mapped IPv4 addresses. This
document
[https://tools.ietf.org/id/draft-smith-v6ops-larger-ipv6-loopback-prefix-04.html]
mentions some drawbacks of it but I guess they shouldn't affect us. I am happy
to give this a shot. Posting here for visibility.
> Support pid_max > 16 bits in the mini cluster
> -
>
> Key: KUDU-1334
> URL: https://issues.apache.org/jira/browse/KUDU-1334
> Project: Kudu
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.5.0
>Reporter: Jean-Daniel Cryans
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.6.0
>
>
> Pretty much anybody running on newer machines/platforms will hit this while
> running the unit tests:
> {noformat}
> I0216 11:27:23.617383 110702 external_mini_cluster.cc:582] Started
> /home/stack/apache-kudu-incubating-0.7.0/build/rc/bin/kudu-master as pid
> 110706
> F0216 11:27:23.617473 110702 external_mini_cluster.cc:258] Check failed: p
> <= MathLimits::kMax (110702 vs. 65535) Cannot run on systems with
> >16-bit pid
> *** Check failure stack trace: ***
> {noformat}
> Having this limitation was fine but now it's something everybody hits.
> The workaround is running this:
> {noformat}
> echo "32768" > /proc/sys/kernel/pid_max
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
