Hi, On Wed, 2020-02-12 at 12:29 -0500, Bob Vandette wrote: > I applied the delegation change that you recommended and now all container > tests pass. > > Here’s the change that I applied. > > echo +cpu +cpuset +io +memory +pids > /sys/fs/cgroup/cgroup.subtree_control > echo +cpu +cpuset +io +memory +pids > > /sys/fs/cgroup/user.slice/cgroup.subtree_control > echo +cpu +cpuset +io +memory +pids > > /sys/fs/cgroup/user.slice/user-23603.slice/cgroup.subtree_control > > Although all the tests now pass, the values for cpusets and cpusets.mems > still do not show up on the host > or container unless limits are specified. That’s the only remaining minor > difference from cgroupv1 that I can see. > > testing container APIs > Directory "JTwork" not found: creating > Passed: containers/cgroup/PlainRead.java > Passed: containers/docker/DockerBasicTest.java > Passed: containers/docker/TestCPUAwareness.java > Passed: containers/docker/TestCPUSets.java > Passed: containers/docker/TestJcmdWithSideCar.java > Passed: containers/docker/TestJFREvents.java > Passed: containers/docker/TestJFRNetworkEvents.java > Passed: containers/docker/TestMemoryAwareness.java > Passed: containers/docker/TestMisc.java > Test results: passed: 9 > Results written to /export/users/bobv/jdk15/build/jtreg/JTwork > testing jdk.internal.platform APIs > Passed: jdk/internal/platform/cgroup/TestCgroupMetrics.java > Passed: jdk/internal/platform/cgroup/TestCgroupSubsystemController.java > Passed: jdk/internal/platform/docker/TestDockerCpuMetrics.java > Passed: jdk/internal/platform/docker/TestDockerMemoryMetrics.java > Passed: jdk/internal/platform/docker/TestSystemMetrics.java > Test results: passed: 5 > Results written to /export/users/bobv/jdk15/build/jtreg/JTwork > testing -XshowSettings:system launcher option > Passed: tools/launcher/Settings.java > Test results: passed: 1 > Results written to /export/users/bobv/jdk15/build/jtreg/JTwork
Excellent, thanks! Mandy: Any more comments? Thanks in advance! Cheers, Severin > Bob. > > > On Feb 12, 2020, at 10:13 AM, Bob Vandette <bob.vande...@oracle.com > > > wrote: > > > > > > > On Feb 12, 2020, at 6:22 AM, Severin Gehwolf <sgehw...@redhat.com > > > > wrote: > > > > > > Hi Bob, > > > > > > On Tue, 2020-02-11 at 16:59 -0500, Bob Vandette wrote: > > > > I applied your patch to the latest JDK 15 sources and ran the > > > > container > > > > tests on Oracle Linux 8.1 with podman/cgroupv2 enabled. There > > > > were some issues. > > > > I’m not sure if its my setup or not. > > > > > > Looking over them, they appear to be setup issues. > > > > > > Getting the setup working right was tricky for me as well. > > > > > > Aside: > > > podman has a --runtime switch which you could point to a cgroups > > > v2 > > > capable crun binary for example. > > > https://bugs.openjdk.java.net/browse/JDK-8230305 has some > > > examples of > > > how to use it. > > > > > > I also needed some extra setup to get the controllers' delegation > > > to > > > work: > > > https://urldefense.com/v3/__https://scrivano.org/2019/02/26/resources-management-with-rootless-containers/__;!!GqivPVa7Brio!Ot7tv-QJgGntRyaimzIRrRh7rqvnzMu9AoOaBWINWNnnpR9LUVNNvGUe9GXp0Ri0Wg$ > > > > > > > > > > I also ran the same build on Ubuntu with docker/cgroupv1. I > > > > didn't see any failures on > > > > the cgroupv1 system. > > > > > > Great, thanks! > > > > > > > Here are some notes: > > > > > > > > The podman version on OL 8.1 doesn't yes support cgroupv2. I > > > > built the latest from sources for this test. > > > > > > > > The docker tests take a very long time under podman! Longer > > > > than > > > > the cgroupv1 run. > > > > > > Possibly. I haven't done any fair comparison of the two. > > > > It’s possible that it’s trying to access several container hubs. I > > may be getting some timeouts > > since I haven’t figured out how to specify proxies for > > podman. I’ve set a local http_proxy env variable > > but maybe that’s not working. > > > > > > cpusets and cpusets.mems are blank on host and if none are > > > > specified > > > > on podman/docker run command. On cgroupv1 they are host values > > > > if not > > > > specified for container. Effective cpusets and cpusets.mems > > > > are set properly > > > > in a container. > > > > > > Yes, there seems to be slight differences to cgroup v1. You can > > > only > > > set cpusets on containers if the host has the controller enabled, > > > though: > > > > > > # cat /sys/fs/cgroup/cgroup.subtree_control > > > cpuset cpu io memory pids > > > # podman run --memory=300M --cpuset-cpus=0,1 -ti -v `pwd`:/mnt > > > fedora:30 /bin/bash > > > [root@5d4a4e593a24 /]# cat /sys/fs/cgroup$(cat /proc/self/cgroup > > > | cut -d':' -f3)/cpuset.cpus > > > 0-1 > > > [root@5d4a4e593a24 mnt]# ./cgroupsv2-jdk/bin/java > > > -XshowSettings:system -version > > > Operating System Metrics: > > > Provider: cgroupv2 > > > Effective CPU Count: 2 > > > CPU Period: 100000us > > > CPU Quota: -1 > > > CPU Shares: -1 > > > List of Processors, 2 total: > > > 0 1 > > > List of Effective Processors, 2 total: > > > 0 1 > > > List of Memory Nodes: N/A > > > List of Available Memory Nodes, 1 total: > > > 0 > > > Memory Limit: 300.00M > > > Memory Soft Limit: Unlimited > > > Memory & Swap Limit: 600.00M > > > > > > openjdk version "15-internal" 2020-09-15 > > > OpenJDK Runtime Environment (build 15-internal+0- > > > adhoc.sgehwolf.openjdk-head-2) > > > OpenJDK 64-Bit Server VM (build 15-internal+0- > > > adhoc.sgehwolf.openjdk-head-2, mixed mode, sharing) > > > > > > > In my setup, your command above works correctly. The only thing > > that doesn’t work is if I run the -XshowSettings > > directly on the host OR I run a container without specifying > > —cpuset-cpus. The default setup seems wrong. > > > > > > HOST OUTPUT: > > > > ./java -XshowSettings:system -version > > > > Operating System Metrics: > > > > Provider: cgroupv2 > > > > Effective CPU Count: 32 > > > > CPU Period: -1 > > > > CPU Quota: -1 > > > > CPU Shares: -1 > > > > List of Processors: N/A > > > > List of Effective Processors: N/A > > > > List of Memory Nodes: N/A > > > > List of Available Memory Nodes: N/A > > > > Memory Limit: Unlimited > > > > Memory Soft Limit: Unlimited > > > > Memory & Swap Limit: Unlimited > > > > > > > > CONTAINER OUTPUT: > > > > # podman run -it -v `pwd`:/mnt ubuntu bash > > > > root@3c6654a3b834:/mnt/jdk/bin# ./java -XshowSettings:system > > > > -version > > > > Operating System Metrics: > > > > Provider: cgroupv2 > > > > Effective CPU Count: 32 > > > > CPU Period: 100000us > > > > CPU Quota: -1 > > > > CPU Shares: -1 > > > > List of Processors: N/A > > > > List of Effective Processors, 32 total: > > > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 > > > > 24 25 26 27 28 29 30 31 > > > > List of Memory Nodes: N/A > > > > List of Available Memory Nodes, 1 total: > > > > 0 > > > > Memory Limit: Unlimited > > > > Memory Soft Limit: Unlimited > > > > Memory & Swap Limit: Unlimited > > > > > > Yes, host and container output differ depending on the configured > > > test > > > system. I remember that I had cpusets working on the host system > > > too at > > > some point, but I forgot what the magic was to get this properly > > > delegated via systemd. > > > > > > > Docker tests fail if /bin/docker is not available in podman > > > > setup. We > > > > probably should enhance the docker check to also look for > > > > podman. > > > > > > Could you be more specific about this? How do you run docker > > > tests? I > > > use: > > > > > > -e PATH -verbose:summary -Djdk.test.container.command=podman > > > -Djdk.test.docker.image.name=fedora > > > -Djdk.test.docker.image.version=30 > > > > > > with -Djdk.test.container.command=podman it shouldn't need > > > docker. Did > > > you specify that property? > > > > I thought that most podman setups try to provide docker > > compatibility. I did not try using the properties. > > In order to make our automation work cleaner, I was hoping that we > > could always just execute docker. > > > > > > Two container tests failed: > > > > > > > > FAILED: containers/cgroup/PlainRead.java failed Memory Limit > > > > is: -2 instead > > > > of unlimited or -1. This is because memory.max is not foumd. > > > > > > This doesn't fail for me, because I've got memory.max present on > > > host: > > > > > > # cat /sys/fs/cgroup$(cat /proc/self/cgroup | cut -d':' > > > -f3)/cgroup.controllers > > > cpu io memory pids > > > [root@f31 sgehwolf]# cat /sys/fs/cgroup$(cat /proc/self/cgroup | > > > cut -d':' -f3)/memory.max > > > max > > > > > > Note, howevre, this is a hotspot test. So support for it for > > > cgroup v2 > > > came with JDK-8230305. It seems we should return -1 if memory.max > > > is > > > not found (over -2, not supported). Could you file a bug for > > > this? It's > > > unrelated to this change. It should be a simple fix. > > > > Here are my controllers: > > cpuset cpu io memory pids rdma > > > > This seems like another case where this file doesn’t exist in the > > path we form on the host. > > > > [0.035s][debug][os,container] /sys/fs/cgroup/user.slice/user- > > 23603.slice/session-14.scope/cpu.max failed, No such file or > > directory > > > > It does exist here: /sys/fs/cgroup/user.slice so maybe it’s a > > delegation issue. > > > > > > FAILED: jdk/internal/platform/cgroup/TestCgroupMetrics.java > > > > This fails because nr_periods line doesn't always exist. I > > > > think you’ve got > > > > to enable a quota for this to appear (not sure). > > > > > > Passes for me, but it needs the cpu controller enabled on the > > > test > > > system. > > > > > > # cat /sys/fs/cgroup$(cat /proc/self/cgroup | cut -d':' > > > -f3)/cgroup.controllers > > > cpu io memory pids > > > # cat /sys/fs/cgroup$(cat /proc/self/cgroup | cut -d':' > > > -f3)/cpu.stat > > > usage_usec 1157537 > > > user_usec 606832 > > > system_usec 550705 > > > nr_periods 0 > > > nr_throttled 0 > > > throttled_usec 0 > > > > > > > Here’s the contents: > > > > % more cpu.stat > > > > usage_usec 23974562755 > > > > user_usec 22257183568 > > > > system_usec 1717379186 > > > > > > It suggests you've got the cpu controller disabled: > > > https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html*cpu__;Iw!!GqivPVa7Brio!Ot7tv-QJgGntRyaimzIRrRh7rqvnzMu9AoOaBWINWNnnpR9LUVNNvGUe9GWZKkNZMA$ > > > > > > > > > """ > > > and the following three when the controller is enabled: > > > > > > > > > nr_periods > > > nr_throttled > > > throttled_usec > > > “"" > > > > I do have the cpu controller enabled. I’ll look into your fixes > > for delegation issues and try again. > > > > Bob. > > > > > > CGROUPV2 results on Oracle Linux 8.1 > > > > --------- > > > > > > > > testing container APIs > > > > > > > > FAILED: containers/cgroup/PlainRead.java > > > > Passed: containers/docker/DockerBasicTest.java > > > > Passed: containers/docker/TestCPUAwareness.java > > > > Passed: containers/docker/TestCPUSets.java > > > > Passed: containers/docker/TestJcmdWithSideCar.java > > > > Passed: containers/docker/TestJFREvents.java > > > > Passed: containers/docker/TestJFRNetworkEvents.java > > > > Passed: containers/docker/TestMemoryAwareness.java > > > > Passed: containers/docker/TestMisc.java > > > > Test results: passed: 8; failed: 1 > > > > Results written to /export/users/bobv/jdk15/build/jtreg/JTwork > > > > Error: Some tests failed or other problems occurred. > > > > > > These are hotspot tests. Covered by JDK-8230305 (hotspot > > > changes). The > > > plain read test passes on a properly configured host system with > > > controller delegation. > > > > > > > testing jdk.internal.platform APIs > > > > > > > > FAILED: jdk/internal/platform/cgroup/TestCgroupMetrics.java > > > > Passed: > > > > jdk/internal/platform/cgroup/TestCgroupSubsystemController.java > > > > Passed: jdk/internal/platform/docker/TestDockerCpuMetrics.java > > > > Passed: > > > > jdk/internal/platform/docker/TestDockerMemoryMetrics.java > > > > Passed: jdk/internal/platform/docker/TestSystemMetrics.java > > > > Test results: passed: 4; failed: 1 > > > > Results written to /export/users/bobv/jdk15/build/jtreg/JTwork > > > > Error: Some tests failed or other problems occurred. > > > > > > I believe cgroup/TestCgroupMetrics.java fails due to bad host > > > config. > > > It passes here: > > > > > > [root@f31 jdk-jdk]# rm -rf JTwork/ JTreport && > > > /media/disk/jtreg/bin/jtreg -timeout:4 -jdk:../cgroupsv2-jdk/ -e > > > PATH -verbose:summary -Djdk.test.container.command=podman > > > -Djdk.test.docker.image.name=fedora > > > -Djdk.test.docker.image.version=30 test/jdk/jdk/internal/platform > > > Directory "JTwork" not found: creating > > > Directory "JTreport" not found: creating > > > Passed: jdk/internal/platform/cgroup/TestCgroupMetrics.java > > > Passed: > > > jdk/internal/platform/cgroup/TestCgroupSubsystemController.java > > > Passed: jdk/internal/platform/docker/TestDockerCpuMetrics.java > > > Passed: jdk/internal/platform/docker/TestDockerMemoryMetrics.java > > > Passed: jdk/internal/platform/docker/TestSystemMetrics.java > > > Test results: passed: 5 > > > Report written to /home/sgehwolf/jdk- > > > jdk/JTreport/html/report.html > > > Results written to /home/sgehwolf/jdk-jdk/JTwork > > > > > > JTR files available here: > > > http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8231111/10/jtreg-results/ > > > > > > > testing -XshowSettings:system launcher option > > > > > > > > Passed: tools/launcher/Settings.java > > > > Test results: passed: 1 > > > > > > Thanks for running the tests! > > > > > > FWIW, jdk-submit came back clean. If we could get the initial > > > support > > > of this pushed soon it would be great. I'd be happy to fix any > > > follow- > > > up issues. > > > > > > Thanks, > > > Severin > > > > > > > Bob. > > > > > > > > > On Feb 11, 2020, at 1:04 PM, Severin Gehwolf < > > > > > sgehw...@redhat.com> wrote: > > > > > > > > > > Hi Mandy, Bob, > > > > > > > > > > Thanks again for the reviews and patience on this. Sorry it > > > > > took me so > > > > > long to get back to this :-/ > > > > > > > > > > Updated webrev: > > > > > Full: > > > > > http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8231111/10/webrev/ > > > > > incremental: > > > > > http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8231111/10/incremental/webrev/ > > > > > > > > > > I've tested this with docker tests on cgroup v1 and via > > > > > podman on a > > > > > cgroup v2 system. They pass. I'll be running this through > > > > > jdk-submit as > > > > > well. > > > > > > > > > > More below. > > > > > > > > > > On Tue, 2020-01-21 at 16:09 -0800, Mandy Chung wrote: > > > > > > Hi Severin, > > > > > > > > > > > > Thanks for the update. > > > > > > > > > > > > On 1/21/20 11:30 AM, Severin Gehwolf wrote: > > > > > > > Full: > > > > > > > http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8231111/09/webrev/ > > > > > > > incremental: > > > > > > > http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8231111/09/incremental/webrev/ > > > > > > > > > > > > > > > > > > > I have answered my own question. Most of the metrics used > > > > > > to return 0 if unavailable due to IOException reading a > > > > > > file or malformed content and now are changed to return -2 > > > > > > due to error fetching the metric. > > > > > > > > > > > > The following are about limits which used to return -1 if > > > > > > unlimited or no limit set. > > > > > > public long getCpuQuota(); > > > > > > public long getCpuShares(); > > > > > > public long getMemoryLimit(); > > > > > > public long getMemoryAndSwapLimit(); > > > > > > public long getMemorySoftLimit(); > > > > > > > > > > > > With this patch, only getMemoryLimit and > > > > > > getMemoryAndSwapLimit specify to return -1 if unlimited or > > > > > > no limit set. However the implementation does return > > > > > > -1. All of the above specify to return -2 if unavailable > > > > > > due to error fetching the metric. > > > > > > > > > > > > I found the implementation quite hard to follow. I spent > > > > > > some time reviewing the code to see if the implementation > > > > > > matches the spec but I can't easily tell yet. For > > > > > > example, > > > > > > CgroupSubsystemController::getLongValueMatchingLine returns > > > > > > -1 when IOException occurs. > > > > > > CgroupSubsystemController::getLongEntry returns 0L if > > > > > > IOException occurs. > > > > > > > > > > > > CgroupV1SubsystemController::convertStringToLong returns > > > > > > Long.MAX_VALUE > > > > > > when value overflows > > > > > > > > > > This one is intentional. It's mapped back to unlimited via > > > > > longValOrUnlimited(). The reason for this is that cgroup v1 > > > > > doesn't > > > > > have a concept of "unlimited". Unlimited values will be a > > > > > very large > > > > > numbers in cgroup v1 files. > > > > > > > > > > > CgroupV2SubsystemController::convertStringToLong returns -1 > > > > > > when IOException occurs > > > > > > > > > > > > CgroupV1Subsystem::getCpuShares return -1 if cpu.shares == > > > > > > 0 or 1024 > > > > > > CgroupV2Subsystem::getCpuShares returns -1 if cpu.weight == > > > > > > 100 or 0 > > > > > > > > > > These two are special cases too. See the implementation note > > > > > of > > > > > Metrics.getCpuShares(). In the cgroup v2 case the default > > > > > value is 100 > > > > > (over 1024 in cgroup v1). That's why unlimited is being > > > > > returned for > > > > > those values. > > > > > > > > > > > CgroupV2Subsystem::getFromCpuMax returns -1 if error in > > > > > > reading cpu.max or malformed content > > > > > > CgroupV2Subsystem::sumTokensIOStat returns -2 if > > > > > > IOException error > > > > > > This is called by getBlkIOServiceCount and getBlkIOServiced > > > > > > > > > > > > I think this can be improved and add the documentation to > > > > > > describe > > > > > > what the methods do. Since Metrics APIs consistently > > > > > > return -2 if > > > > > > unavailable due to error in fetching the metric, why some > > > > > > utility > > > > > > methods in *Subsystem and *SubsystemController return -1 > > > > > > upon error > > > > > > and 0 when unlimited? > > > > > > > > > > > > I suspect if the getXXXValue and other methods are clearly > > > > > > documented > > > > > > with the error cases (possibly renaming the method name if > > > > > > appropriate) > > > > > > CgroupV1Subsystem and CgroupV2SubSystem will become very > > > > > > explicit > > > > > > to understand. > > > > > > > > > > This should be fixed now. > > > > > > > > > > I've gone through the API doc of Metrics.java and have > > > > > updated it. In > > > > > general, I've updated it to return -1 if metric is > > > > > unavailable (due to > > > > > error in reading some files or content being empty), and -2 > > > > > if not > > > > > supported. No method returns -2 currently, but it might > > > > > change and it's > > > > > good to have some way of saying "not implementable" for this > > > > > subsystem > > > > > in the spec. That's my take on it anyway. > > > > > > > > > > There is also a new unit test for shared controller logic: > > > > > TestCgroupSubsystemController.java > > > > > > > > > > It execises various cases of error/success. > > > > > > > > > > That is to ensure proper symmetry across the various cases > > > > > (including > > > > > IOException). I've also documented static methods in > > > > > CgroupSubsystemController. Overall, all methods now return > > > > > the same > > > > > values for cgroup v1 and cgroup v2 (given the impl nuances) > > > > > for the > > > > > various cases. > > > > > > > > > > > CgroupSubsystem.java > > > > > > > > > > > > 44 public static final double > > > > > > DOUBLE_RETVAL_NOT_SUPPORTED = LONG_RETVAL_NOT_SUPPORTED; > > > > > > 49 public static final Boolean > > > > > > BOOL_RETVAL_NOT_SUPPORTED = null; > > > > > > > > > > > > They are no longer needed, right? > > > > > > > > > > Removed. > > > > > > > > > > > CgroupSubsystemFactory.java > > > > > > > > > > > > 89 System.err.println("Warning: Mixed cgroupv1 > > > > > > and cgroupv2 not supported. Metrics disabled."); > > > > > > > > > > > > > > > > > > I expect this be a System.Logger log > > > > > > > > > > Updated. > > > > > > > > > > > 114 if > > > > > > (!Integer.valueOf(0).toString().equals(tokens[0])) { > > > > > > > > > > > > This can be simplified to if (!"0".equals(tokens[0])) > > > > > > > > > > Done, thanks! > > > > > > > > > > > LauncherHelper.java > > > > > > > > > > > > 407 // Extended cgroupv1 specific metrics > > > > > > 408 if (c instanceof MetricsCgroupV1) { > > > > > > 409 MetricsCgroupV1 cgroupV1 = > > > > > > (MetricsCgroupV1)c; > > > > > > 410 limit = cgroupV1.getKernelMemoryLimit(); > > > > > > 411 ostream.println(formatLimitString(limit, > > > > > > INDENT + "Kernel Memory Limit: ")); > > > > > > 412 limit = cgroupV1.getTcpMemoryLimit(); > > > > > > 413 ostream.println(formatLimitString(limit, > > > > > > INDENT + "TCP Memory Limit: ")); > > > > > > 414 Boolean value = > > > > > > cgroupV1.isMemoryOOMKillEnabled(); > > > > > > 415 ostream.println(formatBoolean(value, INDENT > > > > > > + "Out Of Memory Killer Enabled: ")); > > > > > > 416 value = > > > > > > cgroupV1.isCpuSetMemoryPressureEnabled(); > > > > > > 417 ostream.println(formatBoolean(value, INDENT > > > > > > + "CPUSet Memory Pressure Enabled: ")); > > > > > > 418 } > > > > > > > > > > > > MetricsCgroupV1 is linux-only. It will fail the > > > > > > compilation when > > > > > > building on non-linux. One option is to move this code to > > > > > > src/java.base/linux/share/sun/launcher/CgroupMetrics.java > > > > > > > > > > > > Are they continued to be interesting metrics to be output > > > > > > from > > > > > > -XshowSetting? I wonder if they can simply be dropped from > > > > > > the output. > > > > > > Bob will have an opinion. > > > > > > > > > > I've removed those extra cgroup v1 specific metrics printed > > > > > via > > > > > -XshowSettings:system. Not sure what to do with > > > > > MetricsCgroupV1. It's > > > > > only used in tests in webrev 10. On the other hand the idea > > > > > would be > > > > > for consumers to downcast it to MetricsCgroupV1 if they > > > > > needed those > > > > > extra metrics. > > > > > > > > > > Thanks, > > > > > Severin > > > > >
