Re: Java serizalization compatibility issues
At 02:43 AM 9/23/2005, Jeremy Boynes wrote: Sachin's problem is not related to configuration persistence but to the serialization of classes between plugin and server when using JMX remoting over RMI. The upshot of it all is unless we are going to ditch all use of serialization and replace it with XML then we need to exercise the necessary discipline and version the classes involved. IMO most people underestimate the degree of compatibility that can be achieved with serialization - but then I have written RMI stacks for a living and am naturally biased :) Given that RMI-based JMX is in Java SE 5 now it seems a little strange to entirely ditch this approach. Simply fixing the svuid for all serialized classes will make things considerably better since without this svuid's change for the most minor (and inconsequential) reasons. This could be done mehcanilcally with appropriate use of sed and serialver ... $0.02 andy
Re: Java serizalization compatibility issues
Matt Hogstrom wrote: Not being totally familiar with all the nuances in G WRT to serialization my comments should be taken with a grain of salt. From my perspetive there are two major problems with serialized data. One, its very fragile Yes. and two you can't change it if you need to. One could argue users shouldn't be changing it but in extreme circumstances it is unavoidable. I'd vote for the move to XML (ouch, did I say that?) Matt My inclination as well. I've been scratching my head trying to understand 'why serialization?' rather than nice, flat, intuitive text files. There may be a very good answer I just don't what it is. Jeremy Boynes wrote: Sachin's problem is not related to configuration persistence but to the serialization of classes between plugin and server when using JMX remoting over RMI. The upshot of it all is unless we are going to ditch all use of serialization and replace it with XML then we need to exercise the necessary discipline and version the classes involved. -- Jeremy
Re: Java serizalization compatibility issues
Sachin Patel wrote: This is because eclipse can not reference classes or jars at runtime that are not packaged within a plug-in and marked as visible in either the plugin.xml or manifest. A big problem resides as now the same jars I'm packaging must be the same exact jars that reside in the target server I'm deploying. This causes a dependency on a particular server image. This thread wasn't about configuration files but about communication between JVMs. Sachin's plugin fails when talking to the server because there are different versions of the classes in the Eclipse client and in the running server. RMI is the only transport guaranteed to be available by the JMX remoting specification. To use it the classes passed on the wire must be Serializable. We do not control the versions used on either end and hence need to ensure that skewed versions can interoperate. This means following the rules for serialization compatibility (such as adding serialUIDs). -- Jeremy Bill Stoddard wrote: Matt Hogstrom wrote: Not being totally familiar with all the nuances in G WRT to serialization my comments should be taken with a grain of salt. From my perspetive there are two major problems with serialized data. One, its very fragile Yes. and two you can't change it if you need to. One could argue users shouldn't be changing it but in extreme circumstances it is unavoidable. I'd vote for the move to XML (ouch, did I say that?) Matt My inclination as well. I've been scratching my head trying to understand 'why serialization?' rather than nice, flat, intuitive text files. There may be a very good answer I just don't what it is. Jeremy Boynes wrote: Sachin's problem is not related to configuration persistence but to the serialization of classes between plugin and server when using JMX remoting over RMI. The upshot of it all is unless we are going to ditch all use of serialization and replace it with XML then we need to exercise the necessary discipline and version the classes involved. -- Jeremy
Re: Java serizalization compatibility issues
My bad Jeremy. You are correct. I latched onto serialization and immediately went to configuration. I realized the error of my ways this morning when you mentioned the plugin. Doh. Jeremy Boynes wrote: Sachin Patel wrote: This is because eclipse can not reference classes or jars at runtime that are not packaged within a plug-in and marked as visible in either the plugin.xml or manifest. A big problem resides as now the same jars I'm packaging must be the same exact jars that reside in the target server I'm deploying. This causes a dependency on a particular server image. This thread wasn't about configuration files but about communication between JVMs. Sachin's plugin fails when talking to the server because there are different versions of the classes in the Eclipse client and in the running server. RMI is the only transport guaranteed to be available by the JMX remoting specification. To use it the classes passed on the wire must be Serializable. We do not control the versions used on either end and hence need to ensure that skewed versions can interoperate. This means following the rules for serialization compatibility (such as adding serialUIDs). -- Jeremy Bill Stoddard wrote: Matt Hogstrom wrote: Not being totally familiar with all the nuances in G WRT to serialization my comments should be taken with a grain of salt. From my perspetive there are two major problems with serialized data. One, its very fragile Yes. and two you can't change it if you need to. One could argue users shouldn't be changing it but in extreme circumstances it is unavoidable. I'd vote for the move to XML (ouch, did I say that?) Matt My inclination as well. I've been scratching my head trying to understand 'why serialization?' rather than nice, flat, intuitive text files. There may be a very good answer I just don't what it is. Jeremy Boynes wrote: Sachin's problem is not related to configuration persistence but to the serialization of classes between plugin and server when using JMX remoting over RMI. The upshot of it all is unless we are going to ditch all use of serialization and replace it with XML then we need to exercise the necessary discipline and version the classes involved. -- Jeremy
Re: Java serizalization compatibility issues
Ditto, sorry. (Still not a fan of serialized configs :) -David On Sep 23, 2005, at 9:47 AM, Matt Hogstrom wrote: My bad Jeremy. You are correct. I latched onto serialization and immediately went to configuration. I realized the error of my ways this morning when you mentioned the plugin. Doh. Jeremy Boynes wrote: Sachin Patel wrote: This is because eclipse can not reference classes or jars at runtime that are not packaged within a plug-in and marked as visible in either the plugin.xml or manifest. A big problem resides as now the same jars I'm packaging must be the same exact jars that reside in the target server I'm deploying. This causes a dependency on a particular server image. This thread wasn't about configuration files but about communication between JVMs. Sachin's plugin fails when talking to the server because there are different versions of the classes in the Eclipse client and in the running server. RMI is the only transport guaranteed to be available by the JMX remoting specification. To use it the classes passed on the wire must be Serializable. We do not control the versions used on either end and hence need to ensure that skewed versions can interoperate. This means following the rules for serialization compatibility (such as adding serialUIDs). -- Jeremy Bill Stoddard wrote: Matt Hogstrom wrote: Not being totally familiar with all the nuances in G WRT to serialization my comments should be taken with a grain of salt. From my perspetive there are two major problems with serialized data. One, its very fragile Yes. and two you can't change it if you need to. One could argue users shouldn't be changing it but in extreme circumstances it is unavoidable. I'd vote for the move to XML (ouch, did I say that?) Matt My inclination as well. I've been scratching my head trying to understand 'why serialization?' rather than nice, flat, intuitive text files. There may be a very good answer I just don't what it is. Jeremy Boynes wrote: Sachin's problem is not related to configuration persistence but to the serialization of classes between plugin and server when using JMX remoting over RMI. The upshot of it all is unless we are going to ditch all use of serialization and replace it with XML then we need to exercise the necessary discipline and version the classes involved. -- Jeremy
Re: Java serizalization compatibility issues
On Sep 22, 2005, at 6:07 AM, Sachin Patel wrote: So if you are not aware, I'm pulling in and packaging several jars from the lib and lib/endorsed directory into one of the eclipse plug-in, so the classes can be used and referenced by the rest of the eclipse plugins. This is because eclipse can not reference classes or jars at runtime that are not packaged within a plug-in and marked as visible in either the plugin.xml or manifest. A big problem resides as now the same jars I'm packaging must be the same exact jars that reside in the target server I'm deploying. This causes a dependency on a particular server image. If a user modifies classes I reference and rebuilds their server, the plug-in is broken as during deployment I'll receive error messages like the following... Caused by: java.io.InvalidClassException: org.apache.geronimo.kernel.config.ConfigurationModuleType; local class incompatible: stream classdesc serialVersionUID = 6296527685792707191, local class serialVersionUID = -4121586344416418391 So looking at that particular class, it looks like the serialVersionUID is generated by Java compiler. This is bad as now jars/classes risk compatibility between every build. We need a solution for this. The only other option I'm aware of is for these serializable classes to hard code and explicity assign a value. Of course we must then assue that we manually maintain backward compatibility to support the N-2 model for these classes. This problem will eventually have to be solved anyway when there is multiple server support and across different versions. I think this is a must fix for 1.0 as without doing so we risk product migration, mixed version interoperability, tooling, possibly user applications, etc... I agree with you. We need a solution for this and it does affect compatibility even to the extent of breaking it on every build. There was a one week battle on this in May that resulted in people saying I dont' want to talk about it anymore. I think the problem is an issue that can't be solved through serialization in an effective and reliable way. Largely because of how easy it is to break serialized data and that broken serialized data is nearly unrecoverable. Here is my perspective: With the proper amount of forethought and code, you can get a very limited amount of compatibility with serialization between class file versions, but it's unrealistic and unreliable at best. Unless you can convince everyone to do that, this entire thread is moot regardless of what you or I think, and our existing system will never serve as a reliable way to store configuration data across upgrades or patches. Here are a few snippets of that argument, completely biased to my perspective: On May 13, 2005, at 3:54 PM, David Blevins wrote: On Fri, May 13, 2005 at 03:42:58PM -0700, Jeremy Boynes wrote: Solving the compatibility problem will require the developers to pay attention to versioning. We will need to require that at some point though. Paying attention is an understatement. In order to serialize data to and from class definitions, those classes cannot change in any significant way, ever. No refactoring, repackaging, reorganizing, no removing methods, no removing class variables. This doesn't just apply to public APIs, but even the internal classes of each component. Considering that at deploy time we are building and serializing Jetty web containers, Tomcat web containers, ActiveMQ queues/ topics, OpenEJB ejb containers, Axis webservices, TranQL connectors and more, is it reasonable to ask or expect these projects basically not to change their code? Clearly, projects will clean and improve their code, we can't and shouldn't try and stop them. We want them to optimize and improve things. As this change is going to occur, people *will* have to redeploy their apps as we support no other way to store configurations. Serializing and deserializing server state for fast startup is a cool feature, but not something we are in any position to guarantee between releases of Axis, ActiveMQ, OpenEJB, Jetty, Tomcat, et. al. On May 19, 2005, at 9:35 AM, David Blevins wrote: On Wed, May 18, 2005 at 09:04:07PM -0700, Jeremy Boynes wrote: David Blevins wrote: I know exactly how much room for error there is as well as the amount forethought and code required to guarantee even a small amout of future compatibility. Using SUIDs alone does not cut it. I'm not arguing it's impossible, I'm arguing that it's unrealistic and unreliable at best. I think you're looking at it backward - without a crystal ball you don't know what future versions of the class will look like. It is the responsbility of the evolved class to handle upgrading older data. You won't know what order the fields were written to the stream, so unless you had the
Re: Java serizalization compatibility issues
Sachin's problem is not related to configuration persistence but to the serialization of classes between plugin and server when using JMX remoting over RMI. The upshot of it all is unless we are going to ditch all use of serialization and replace it with XML then we need to exercise the necessary discipline and version the classes involved. -- Jeremy
RE: Java serizalization compatibility issues
Hi Sachin, I'm interested in contributing to the Eclipse tools project of Geronimo. I have sent a mail before asking where to find more information, since the link given to download the unstable build of the plug-in was broken when I tried. Can you please give me some pointers on where to begin and to get hold of the source for the plug-in? Tyrell -Original Message- From: Sachin Patel [mailto:[EMAIL PROTECTED] Sent: Thursday, September 22, 2005 7:08 PM To: geronimo-dev Subject: Java serizalization compatibility issues So if you are not aware, I'm pulling in and packaging several jars from the lib and lib/endorsed directory into one of the eclipse plug-in, so the classes can be used and referenced by the rest of the eclipse plugins. This is because eclipse can not reference classes or jars at runtime that are not packaged within a plug-in and marked as visible in either the plugin.xml or manifest. A big problem resides as now the same jars I'm packaging must be the same exact jars that reside in the target server I'm deploying. This causes a dependency on a particular server image. If a user modifies classes I reference and rebuilds their server, the plug-in is broken as during deployment I'll receive error messages like the following... Caused by: java.io.InvalidClassException: org.apache.geronimo.kernel.config.ConfigurationModuleType; local class incompatible: stream classdesc serialVersionUID = 6296527685792707191, local class serialVersionUID = -4121586344416418391 So looking at that particular class, it looks like the serialVersionUID is generated by Java compiler. This is bad as now jars/classes risk compatibility between every build. We need a solution for this. The only other option I'm aware of is for these serializable classes to hard code and explicity assign a value. Of course we must then assue that we manually maintain backward compatibility to support the N-2 model for these classes. This problem will eventually have to be solved anyway when there is multiple server support and across different versions. I think this is a must fix for 1.0 as without doing so we risk product migration, mixed version interoperability, tooling, possibly user applications, etc... Sachin.
Re: Java serizalization compatibility issues
Not being totally familiar with all the nuances in G WRT to serialization my comments should be taken with a grain of salt. From my perspetive there are two major problems with serialized data. One, its very fragile and two you can't change it if you need to. One could argue users shouldn't be changing it but in extreme circumstances it is unavoidable. I'd vote for the move to XML (ouch, did I say that?) Matt Jeremy Boynes wrote: Sachin's problem is not related to configuration persistence but to the serialization of classes between plugin and server when using JMX remoting over RMI. The upshot of it all is unless we are going to ditch all use of serialization and replace it with XML then we need to exercise the necessary discipline and version the classes involved. -- Jeremy
Re: Java serizalization compatibility issues
Hi. Tyrell, I've already fixed the link. In the future, please start a new thread when discussing a new topic, or respond with your original thread. Thanks. Tyrell Perera wrote: Hi Sachin, I'm interested in contributing to the Eclipse tools project of Geronimo. I have sent a mail before asking where to find more information, since the link given to download the unstable build of the plug-in was broken when I tried. Can you please give me some pointers on where to begin and to get hold of the source for the plug-in? Tyrell -Original Message- From: Sachin Patel [mailto:[EMAIL PROTECTED] Sent: Thursday, September 22, 2005 7:08 PM To: geronimo-dev Subject: Java serizalization compatibility issues So if you are not aware, I'm pulling in and packaging several jars from the lib and lib/endorsed directory into one of the eclipse plug-in, so the classes can be used and referenced by the rest of the eclipse plugins. This is because eclipse can not reference classes or jars at runtime that are not packaged within a plug-in and marked as visible in either the plugin.xml or manifest. A big problem resides as now the same jars I'm packaging must be the same exact jars that reside in the target server I'm deploying. This causes a dependency on a particular server image. If a user modifies classes I reference and rebuilds their server, the plug-in is broken as during deployment I'll receive error messages like the following... Caused by: java.io.InvalidClassException: org.apache.geronimo.kernel.config.ConfigurationModuleType; local class incompatible: stream classdesc serialVersionUID = 6296527685792707191, local class serialVersionUID = -4121586344416418391 So looking at that particular class, it looks like the serialVersionUID is generated by Java compiler. This is bad as now jars/classes risk compatibility between every build. We need a solution for this. The only other option I'm aware of is for these serializable classes to hard code and explicity assign a value. Of course we must then assue that we manually maintain backward compatibility to support the N-2 model for these classes. This problem will eventually have to be solved anyway when there is multiple server support and across different versions. I think this is a must fix for 1.0 as without doing so we risk product migration, mixed version interoperability, tooling, possibly user applications, etc... Sachin.
Re: Java serizalization compatibility issues
Matt Hogstrom wrote: Not being totally familiar with all the nuances in G WRT to serialization my comments should be taken with a grain of salt. From my perspetive there are two major problems with serialized data. One, its very fragile and two you can't change it if you need to. One could argue users shouldn't be changing it but in extreme circumstances it is unavoidable. I'd vote for the move to XML (ouch, did I say that?) What protocol should the plugin use to communicate with the server? I think the only XML based protocol supported by MX4J is SOAP over HTTP. -- Jeremy
Re: Java serizalization compatibility issues
I think it would be nice to be able to publish using either SOAP or RMI, and provide that option to the user in the server configuration panel in the tools. I think RMI has better performance once we remote deployment is supported, but SOAP is more reliable through firewalls, so thats why it would be good to have it to fall back on. Jeremy Boynes wrote: Matt Hogstrom wrote: Not being totally familiar with all the nuances in G WRT to serialization my comments should be taken with a grain of salt. From my perspetive there are two major problems with serialized data. One, its very fragile and two you can't change it if you need to. One could argue users shouldn't be changing it but in extreme circumstances it is unavoidable. I'd vote for the move to XML (ouch, did I say that?) What protocol should the plugin use to communicate with the server? I think the only XML based protocol supported by MX4J is SOAP over HTTP. -- Jeremy
