Hi All, Good timing – I think I just ran into this issue this morning running a Groovy authentication policy but most of our services are CAS. I’m going to keep an eye on it and will let you know what I find.
CAS 6.3.0 on Java 11 Thanks, Tom From: [email protected] <[email protected]> On Behalf Of William Jojo Sent: Monday, March 8, 2021 1:46 PM To: Morning Star <[email protected]> Cc: CAS Community <[email protected]> Subject: [EXT] [cas-user] Re: CAS 6.1.6 inotify instances skyrocketing with Groovy files in SAML service. CAUTION: This email originated from outside of SIG. Exercise caution when opening attachments or clicking links, especially from unknown senders. [EXT-STAMP-ADDED] Morning Star, The problem persists on 6.2.8 - which is the latest one I have available to test at the moment. Seems to only be increasing the inotify list with each use of the service that uses the external Groovy file (meaning not inline). Maybe an improvement, but it has been some time since I dug into this issue. Bill On Thu, Mar 4, 2021 at 7:46 AM <[email protected]<mailto:[email protected]>> wrote: Morning Star, Which CAS and OS versions are you using? The only solution for us was to in-line them. We lost logging, but since no one else spoke up about the issue and I had no time to further research it myself, I moved on since in-line worked. Let me try it on my test system and see if 6.2 or 6.3 still have the problem. It is possible that it is already fixed. Bill Sent from a device. On Mar 4, 2021, at 12:15 AM, Morning Star <[email protected]<mailto:[email protected]>> wrote: Hi William, We are also facing the same issue like you. Could you please help us? On Tuesday, June 2, 2020 at 8:37:03 PM UTC+5:30 William Jojo wrote: Well, I was able to stop the hemorrhaging. Have been watching it for about an hour and the inotify list remains constant. By using an inline Groovy script, CAS no longer needed to setup a watcher service for the Groovy scripts. However, I lost the ability to log debug info and compound statements seem to not be allowed in the inline model. Fortunately my code was not so horribly complex that I was able to work it out. For example: AD groups are horribly disfigured (read in DN form) from the LDAP query, so we rewrite them like so: memberOf: [ java.util.ArrayList [ groovy { def groups = attributes['memberOf']; def result = []; for ( cn in groups ) result.add( ( cn =~ /CN=([^,]+),/)[0][1] ) ; return result; } ] ] If anyone has an idea on where to begin looking for the cause of this issue, I am quite happy to help determine why this flies out of control. Also, if anyone has info on the syntax limitations of inline Groovy scripts and how to still do logging in the inline script that would be very helpful for debugging. Thank you! Bill On Monday, June 1, 2020 at 12:44:23 PM UTC-4, William Jojo wrote: Been running 6.1.6 for about 2 weeks. No issues - until I added SAML support. This morning I noticed CAS no longer working. Checked log and found: >From log: 2020-06-01 09:05:32,086 INFO [org.apereo.cas.util.io<http://org.apereo.cas.util.io>.PathWatcherService] - <Watching directory at [/etc/cas/saml]> 2020-06-01 09:05:32,086 ERROR [org.apereo.cas.services.ReturnMappedAttributeReleasePolicy] - <User limit of inotify instances reached or too many open files> java.io.IOException: User limit of inotify instances reached or too many open files at sun.nio.fs.LinuxWatchService.<init>(LinuxWatchService.java:64) ~[?:?] at sun.nio.fs.LinuxFileSystem.newWatchService(LinuxFileSystem.java:47) ~[?:?] at org.apereo.cas.util.io<http://org.apereo.cas.util.io>.PathWatcherService.<init>(PathWatcherService.java:62) ~[cas-server-core-util-api-6.1.6.jar:6.1.6] at org.apereo.cas.util.io<http://org.apereo.cas.util.io>.PathWatcherService.<init>(PathWatcherService.java:40) ~[cas-server-core-util-api-6.1.6.jar:6.1.6] at org.apereo.cas.util.io<http://org.apereo.cas.util.io>.FileWatcherService.<init>(FileWatcherService.java:26) ~[cas-server-core-util-api-6.1.6.jar:6.1.6] at org.apereo.cas.util.scripting.WatchableGroovyScriptResource.<init>(WatchableGroovyScriptResource.java:31) ~[cas-server-core-util-api-6.1.6.jar:6.1.6] Thought this was odd since never had this problem with any other area of CAS watch areas. Did some digging and seems this is NOT an issue UNTIL I added the groovy files to a SAML service. The portion of the JSON is as follows: memberOf: [ java.util.ArrayList [ file:/etc/cas/saml/memberOf.groovy ] ] eduPersonPrimaryAffiliation: [ java.util.ArrayList [ file:/etc/cas/saml/eduPersonPrimaryAffiliation.groovy ] ] Now look at this output: root@casdev-master:~# while (( 1 == 1 )); do date; lsof | grep inotify | grep 31744 | wc -l; sleep 120; done Mon Jun 1 11:28:05 EDT 2020 178 Mon Jun 1 11:30:05 EDT 2020 178 Mon Jun 1 11:32:06 EDT 2020 178 Mon Jun 1 11:34:06 EDT 2020 178 Mon Jun 1 11:36:07 EDT 2020 178 Mon Jun 1 11:38:08 EDT 2020 178 Mon Jun 1 11:40:08 EDT 2020 1872 Mon Jun 1 11:42:09 EDT 2020 2500 Mon Jun 1 11:44:10 EDT 2020 3192 Mon Jun 1 11:46:11 EDT 2020 3948 Mon Jun 1 11:48:12 EDT 2020 4768 Mon Jun 1 11:50:13 EDT 2020 5652 Mon Jun 1 11:52:14 EDT 2020 6600 There are 178 inotify watches consistently UNTIL I edit the service file and allow the Groovy files to be used. Then it just goes out of control. There were this many entries for each: root@casdev-master:~# lsof | grep inotify | grep 31744 | grep edu | wc -l 1200 root@casdev-master:~# lsof | grep inotify | grep 31744 | grep member | wc -l 1104 It seems too be increasing by hundreds of entries per TID in a very brief period of time and it also seems to be affecting other inotify counts as a result. Any thoughts on why this would suddenly go out of control when adding Groovy files to the service? Thank you! Bill -- - Website: https://apereo.github.io/cas - Gitter Chatroom: https://gitter.im/apereo/cas - List Guidelines: https://goo.gl/1VRrw7 - Contributions: https://goo.gl/mh7qDG --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAOGev1jGXZNNoTTod%2B04s5pELEyX20%3D78oSQyNGsLK5MoOxu1Q%40mail.gmail.com<https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAOGev1jGXZNNoTTod%2B04s5pELEyX20%3D78oSQyNGsLK5MoOxu1Q%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- - Website: https://apereo.github.io/cas - Gitter Chatroom: https://gitter.im/apereo/cas - List Guidelines: https://goo.gl/1VRrw7 - Contributions: https://goo.gl/mh7qDG --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/MN2PR02MB665505FF11F1376CFA7190F9CB939%40MN2PR02MB6655.namprd02.prod.outlook.com.
