Re: War Maven artifacts deployment
Le 17 mars 2019 à 16:22, Romain Manni-Bucau a écrit : > Le dim. 17 mars 2019 à 14:51, Gaël Lalire a > écrit : > >> >> Le 17 mars 2019 à 13:21, Romain Manni-Bucau a >> écrit : >> >>> Le dim. 17 mars 2019 à 12:56, Gaël Lalire a >>> écrit : >>> >>>> Hello Romain, >>>> >>>> I already explained why I do not want to give file or jar:file URL, even >>>> if I have it. >>>> Maven resolver is giving me File, I create a VestigeJar from it >>>> >> https://gaellalire.fr/gitlab/vestige/vestige.spi.resolver/blob/master/src/main/java/fr/gaellalire/vestige/spi/resolver/VestigeJar.java >>>> I also create a mvn: URL accessible through getCodeBase however it is >>>> only for policy permission you should not use openStream on it >> (although I >>>> had to make it possible to get BouncyCastle working). >>>> >>> >>> I get that but it also can conflict with other resolver using mvn:, open >>> new security issues and break some libs - speaking having dropped a lib >>> doing exactly that at work for all these reasons. >> >> mvn: was implemented two weeks before, it is only for ProtectionDomain. >> And ProtectionDomain is used by almost no library. >> VestigeClassloader send vrt: URL. These ones start with / and does not >> break libraries relying on context URL. >> It will break those expecting jar:file URL though but Java9 with its jrt: >> URL will also break them, so I don't care. >> > > Paxurl uses it for a long tile. Also jrt is different cause excluded from > most scanner - it is only the jvm - whereas your jars are not excluded and > scanning will fail. Check out spring, weld, openwebbeans etc... > I mean implemented in Vestige. I use a sub format of Paxurl (no repository, no version range). Also I could use vmvn: instead, as I said it's only for code base URL. Scanning in openwebbeans will not work because it is expecting an URLClassLoader or using java.class.path property. Java9 is not providing URLClassLoader ... these scanners must evolve. I see 2 types of scanning : - hinted one -> for example with beans.xml. These ones can be written to be working only using clean Java (context URL will do the job). - blind one -> searching for annotations in all jars. These ones have to hack the system to find all jars (there is no listAllJar in classloader), they should provide an API so its container can give it the jar list, and by jar list I think of something like (InputStream, JarName, JarSize) rather than a JarFile or URL. > > >>> >>> >>>> I think that SecureClassLoader is not secure enough because it only >> checks >>>> classes not resources. >>>> In a world using XML to create, configure and link instances (Spring), >> it >>>> is terrible to let resources unchecked. >>>> >>>> That's also why VestigeClassLoader#getResource is returning vrt: URL and >>>> not jar:file: URL. >>>> >>> >>> Classloaders generally check in their own calls, not in the resource >>> itself. Can' t you validate the resource before actually reasing it? >> Sounds >>> saner and closer to security manager common configs. >> >> Yes but it is not secure even with read lock. >> The read lock is on the file not the path. You can remove and even replace >> the file while the read lock is still here. >> So my check succeeds, I give a jar:file URL then the file is replaced, the >> jar:file is open and get the new file (hacked one). >> > > Sounds like you want to implement a jvm filesystem and not a new url > handler explained this way otherwise you still have this issue with other > concurrent accesses. > I have not yet implemented it. Concurrent accesses can be solved in multiple ways : - open a new RandomAccessFile each time and check the signature each time (probably the slowest way) - create a cache system - share the RandomAccessFile and synchronize read calls between each InputStream vrt format is vrt:/[ClassLoaderNumber]/[JarOfClassLoaderNumber]!/[PathOfFile] for example vrt:/56/3!/META-INF/MANIFEST.MF I don't see it as a file system. > >>> >>> >>>> The easiest way to implements War Maven artifacts deployment is to use >>>> directly Maven resolver API and give file or jar:file URL to Tomcat. >>>> >>> >>> Or war: since tomcat supports it. >> >> Of course >> >>> >>> >>> >>>> I could have done that, but guess what, I'm willing to give TomEE EAR >>>
Re: War Maven artifacts deployment
Le 17 mars 2019 à 13:21, Romain Manni-Bucau a écrit : > Le dim. 17 mars 2019 à 12:56, Gaël Lalire a > écrit : > >> Hello Romain, >> >> I already explained why I do not want to give file or jar:file URL, even >> if I have it. >> Maven resolver is giving me File, I create a VestigeJar from it >> https://gaellalire.fr/gitlab/vestige/vestige.spi.resolver/blob/master/src/main/java/fr/gaellalire/vestige/spi/resolver/VestigeJar.java >> I also create a mvn: URL accessible through getCodeBase however it is >> only for policy permission you should not use openStream on it (although I >> had to make it possible to get BouncyCastle working). >> > > I get that but it also can conflict with other resolver using mvn:, open > new security issues and break some libs - speaking having dropped a lib > doing exactly that at work for all these reasons. mvn: was implemented two weeks before, it is only for ProtectionDomain. And ProtectionDomain is used by almost no library. VestigeClassloader send vrt: URL. These ones start with / and does not break libraries relying on context URL. It will break those expecting jar:file URL though but Java9 with its jrt: URL will also break them, so I don't care. > > >> I think that SecureClassLoader is not secure enough because it only checks >> classes not resources. >> In a world using XML to create, configure and link instances (Spring), it >> is terrible to let resources unchecked. >> >> That's also why VestigeClassLoader#getResource is returning vrt: URL and >> not jar:file: URL. >> > > Classloaders generally check in their own calls, not in the resource > itself. Can' t you validate the resource before actually reasing it? Sounds > saner and closer to security manager common configs. Yes but it is not secure even with read lock. The read lock is on the file not the path. You can remove and even replace the file while the read lock is still here. So my check succeeds, I give a jar:file URL then the file is replaced, the jar:file is open and get the new file (hacked one). > > >> The easiest way to implements War Maven artifacts deployment is to use >> directly Maven resolver API and give file or jar:file URL to Tomcat. >> > > Or war: since tomcat supports it. Of course > > > >> I could have done that, but guess what, I'm willing to give TomEE EAR >> Maven artifacts deployment after Tomcat. >> In the TomEE my company uses there is a horrible CLASSPATH because they >> wanted to avoid RMI call between EAR. >> That is where Vestige is useful, it allows to choose which jar(s) should >> be shared without any global impact. >> > > Hmm, you use a tomee? Did you check jars.txt? Can be worth pinging tomee > about it cause all is there to do it afaik. Thanks for the pointer, I will check it because I really don't like global CLASSPATH. For my business case it will almost certainly work, however I'm pretty sure it has not the power of Vestige. I don't know if you chose to create one classloader grouping all jars in jar.txt in which case you will have issues when a library is deployed in 2 different versions. Or if you chose to create one classloader per entry in jar.txt in which case the jar will be missing its dependencies unless you expect them to be in MANIFEST.MF. In Maven repositories Class-Path is generally not set in MANIFEST.MF. > > >> You have 3 scopes and 2 modes in Vestige. >> Mode CLASSPATH is creating one classloader with all jars inside, not easy >> to share. >> Mode FIXED_DEPENDENCIES is creating one classloader per jar, but implies >> some good practice (use of context classloader for example). >> >> After setting FIXED_DEPENDENCIES you have to use the PLATFORM scope to get >> it shared, this scope implies some other good practice (static field >> immutable for example). >> >> About memory issue, I don't get your point. >> I will not keep all jars content in memory I will use shared locks (fcntl) >> to keep the content of RandomAccessFile the way it was when I checked it. >> VestigeJar#open will create an InputStream reading from this locked >> RandomAccessFile. >> > > Oki but then you are in jar:file mode ;) In a way, but I give only the InputStream not the URL so the JarURLConnection of JarScannerCallback can't be fetched. > > >> Regards, >> Gaël Lalire >> >> Le 17 mars 2019 à 10:17, Romain Manni-Bucau a >> écrit : >> >> Hi Gaël, >> >> In Tomee we plugged before to enrich the classloader and then tomcat -and >> other libs - works normally using jar urls. >> >> Cant you use a listener to do that and conve
Re: War Maven artifacts deployment
Hello Romain, I already explained why I do not want to give file or jar:file URL, even if I have it. Maven resolver is giving me File, I create a VestigeJar from it https://gaellalire.fr/gitlab/vestige/vestige.spi.resolver/blob/master/src/main/java/fr/gaellalire/vestige/spi/resolver/VestigeJar.java I also create a mvn: URL accessible through getCodeBase however it is only for policy permission you should not use openStream on it (although I had to make it possible to get BouncyCastle working). I think that SecureClassLoader is not secure enough because it only checks classes not resources. In a world using XML to create, configure and link instances (Spring), it is terrible to let resources unchecked. That's also why VestigeClassLoader#getResource is returning vrt: URL and not jar:file: URL. The easiest way to implements War Maven artifacts deployment is to use directly Maven resolver API and give file or jar:file URL to Tomcat. I could have done that, but guess what, I'm willing to give TomEE EAR Maven artifacts deployment after Tomcat. In the TomEE my company uses there is a horrible CLASSPATH because they wanted to avoid RMI call between EAR. That is where Vestige is useful, it allows to choose which jar(s) should be shared without any global impact. You have 3 scopes and 2 modes in Vestige. Mode CLASSPATH is creating one classloader with all jars inside, not easy to share. Mode FIXED_DEPENDENCIES is creating one classloader per jar, but implies some good practice (use of context classloader for example). After setting FIXED_DEPENDENCIES you have to use the PLATFORM scope to get it shared, this scope implies some other good practice (static field immutable for example). About memory issue, I don't get your point. I will not keep all jars content in memory I will use shared locks (fcntl) to keep the content of RandomAccessFile the way it was when I checked it. VestigeJar#open will create an InputStream reading from this locked RandomAccessFile. Regards, Gaël Lalire Le 17 mars 2019 à 10:17, Romain Manni-Bucau a écrit : > Hi Gaël, > > In Tomee we plugged before to enrich the classloader and then tomcat -and > other libs - works normally using jar urls. > > Cant you use a listener to do that and convert m2 urls to plain jar files - > at the end it is local files i guess otherwise you generally consume too > much memory to be prod friendly? > > > Le dim. 17 mars 2019 à 09:46, Gaël Lalire a > écrit : > >> Hello Tomcat developers, >> >> I made a software to enable update of Java applications named Vestige. >> To achieve that, Vestige use Maven, downloading Maven artifacts and >> creating classloaders linked with jar inside m2 repository. >> >> I made it to update my IBM notes connector (POP access provider). >> >> The fact it is downloading Maven artifacts makes the assembly >> (jar-with-dependencies of maven-assembly-plugin) of the connector not >> mandatory. >> >> In a business project I saw that war artifacts were filling the >> repository, so they had to regularly remove older version from it. >> I thought it would be great if we could remove the WEB-INF/lib from the >> published war and still be able to deploy it with Tomcat. >> >> I did that, the WebResource API helps me a lot. >> However I had to disable JarScanner API (tld & fragments) because it's >> using JarURLConnection and my API is not providing jar:file: nor file: URL. >> My API won't provide them because I want to be able to check a pgp >> signature before any use of an artifact in m2 repository. >> If I check the signature and send a jar:file: or file: URL it won't be >> secure because there is no way to prevent the modification of the file >> after the check. >> To be secure I will probably lock the file for reading, then check the >> signature, and give locked InputStream. >> >> I would like you to change the JarScanner API/Impl so it won't rely on >> JarURLConnection anymore (maybe WebResource ?). >> Also I have to replace some Tomcat classes ( >> https://gaellalire.fr/gitlab/vestige_app/tomcat_vestige/commit/67dea6054c9da30047ebba3e9a376fa44b544f13) >> that is not future proof. >> Could you provide some extension(s) so I could do the same thing without >> replacing any Tomcat class ? >> >> Hoping that you get interested enough to help me improve the Maven >> artifact deployment support, I send you my best regards. >> >> PS: >> You can test the vwar, an xml which describes the war to deploy >> (essentially repository URL, groupId, artifactId, version), deployment by : >> - download (https://gaellalire.fr/vestige/) & install & run Vestige >> - go to http://localhost:8480/ >> - clic
War Maven artifacts deployment
Hello Tomcat developers, I made a software to enable update of Java applications named Vestige. To achieve that, Vestige use Maven, downloading Maven artifacts and creating classloaders linked with jar inside m2 repository. I made it to update my IBM notes connector (POP access provider). The fact it is downloading Maven artifacts makes the assembly (jar-with-dependencies of maven-assembly-plugin) of the connector not mandatory. In a business project I saw that war artifacts were filling the repository, so they had to regularly remove older version from it. I thought it would be great if we could remove the WEB-INF/lib from the published war and still be able to deploy it with Tomcat. I did that, the WebResource API helps me a lot. However I had to disable JarScanner API (tld & fragments) because it's using JarURLConnection and my API is not providing jar:file: nor file: URL. My API won't provide them because I want to be able to check a pgp signature before any use of an artifact in m2 repository. If I check the signature and send a jar:file: or file: URL it won't be secure because there is no way to prevent the modification of the file after the check. To be secure I will probably lock the file for reading, then check the signature, and give locked InputStream. I would like you to change the JarScanner API/Impl so it won't rely on JarURLConnection anymore (maybe WebResource ?). Also I have to replace some Tomcat classes (https://gaellalire.fr/gitlab/vestige_app/tomcat_vestige/commit/67dea6054c9da30047ebba3e9a376fa44b544f13) that is not future proof. Could you provide some extension(s) so I could do the same thing without replacing any Tomcat class ? Hoping that you get interested enough to help me improve the Maven artifact deployment support, I send you my best regards. PS: You can test the vwar, an xml which describes the war to deploy (essentially repository URL, groupId, artifactId, version), deployment by : - download (https://gaellalire.fr/vestige/) & install & run Vestige - go to http://localhost:8480/ - click on install - write "tomcat" in repository application name - write "8.0.32" in repository application version - write "tc" in local application name - click install button - click play button - go to http://localhost:8080/mywar/hello (servlet test) and http://localhost:8080/mywar/hi.jsp?max=5 (jsp test) The vwar will be at $VESTIGE_BASE/app/tc/webapps/mywar.vwar Where $VESTIGE_BASE is : - $HOME/Vestige on Mac OS X - $HOME/vestige on Linux - %userprofile%\Vestige on Windows - the place you unzip the file if you chose to install the standalone version (a ZIP file) You can also see it at https://gaellalire.fr/gitlab/vestige_app/tomcat_vestige/blob/master/installer/src/main/resources/mywar.vwar tomcat_vestige sources at https://gaellalire.fr/gitlab/vestige_app/tomcat_vestige tomcat_vestige descriptor at https://gaellalire.fr/vestige/repository/tomcat/tomcat-8.0.32.xml mywar sources at https://gaellalire.fr/gitlab/vestige_app/mywar (its pom https://gaellalire.fr/gitlab/vestige_app/mywar/blob/master/pom.xml excludes lib folder) signature.asc Description: Message signed with OpenPGP using GPGMail