[PROPOSAL] - Java OffHeap Memory Pool
Project Name: Jillegal 1. Abstract: GC is one of the time taken operations in Java. GC run anytime, marks, swaps and compacts objects at memory. If there are so many live objects, managing them by GC leads to overhead. If objects can be allocated outside of GC, there will be no overhead for the application. The session will go through the new method of creating and using object off-heap with no additional serialization/deserialization or any other overheads. 2. Proposal: For off-heap memory, I propose a solution that objects are allocated and initialized at off-heap instead of heap. Not only object attributes are allocated at off-heap, but also object header and metadata are allocated at off-heap. So while a reference to this object at off-heap is being interpreted, JVM knows which class this object references to. You can get your off-heap object from an object pool instead of with new operator. This object pool allocates a fixed size memory region from off-heap and create empty objects with given class on there. These empty objects can be created as lazy or eager. Another advantage off this technique is that objects in pool are layout in memory as sequential. While accessing an object, its neighbour objects are also fetched to CPU cache, so CPU cache hit rate for sequential object accesses are increased. On the other hand, freeing unused objects is responsibility of developer by calling free method of object pool which means there is no dead object detection mechanism like GC. Therefore, getting all objects from off-heap for whole application is dangerous and not recommended. Because, this will cause memory leaks. In addition, this technique can be combined with Java Instrumentation API. With @FromOffHeap annotation, developer can sign classes these must be allocated from off-heap instead of heap. With Java Instrumentation API, all new operators for signed classes with @FromOffHeap annotation can ben transformed to code that allocates objects of signed class from off-heap via object pool. So developer doesn't change all new keywords for getting from object pool in code. Instread of this, just sign class with @FromOffHeap annotation and all new keywords transformed for getting from object pool at class load time. This technique was used at a real time CDR processing system for Turk Telekom. There were billions of objects were used. Managing these objects by GC caused to performance overhead. For some most used classes, we allocated these objects from off-heap instead of new keyword. After some processings on them (takes 4-5 hours), we release these allocated memory regions to operating system by freeing them. Allocating objects from off-heap pool helps us to gain significant execution time performance. 3. Rationale: In general, off-heap pool implementations are implemented by serialization/deserialization to allocated off-heap memory region via ByteBuffer class. But this technique leads to extra execution overhead. Because while reading from an object, the target object must be created by deserializing all primitive fields eagerly or only required fields on demand and while writing to an object, the attribute has been set by application, must be deserialized to allocated off-heap memory region. In addition, objects itself is created at heap, so GC knows and tracks it. With my solution, all of these overheads are overcomed. 4. Initial Goals: * Allocating objects from off-heap and using them as normal on-heap Java object. * Allocating arrays for object types from off-heap and using them as normal on-heap Java object arrays. * Allocating arrays for primitive types from off-heap and using them as normal on-heap Java primitive type arrays. * Allocating strings from off-heap and using them as normal on-heap strings. * Implementing auto expandable off-heap pool that expands when its delegated off-heap pool implementation is full. * All features must be supported for 32 bit and 64 bit JVM. * All features must be supported for Sun HotSpot JVM, Oracle JRockit, IBM J9. 5. Currently Implemented Features: * Allocating objects from off-heap and using them as normal on-heap Java object * Allocating arrays for object types from off-heap and using them as normal on-heap Java object array * Allocating arrays for primitive types from off-heap and using them as normal on-heap Java primitive type arrays * Implementing auto expandable off-heap pool that expands when its delegated off-heap pool implementation is full. * All features are supported for 32 bit and 64 bit JVM. * All features are supported for Sun HotSpot JVM, Oracle JRockit (IBM J9 support will be added). 6. Roadmap * Automatic detection and binding for complex off-heap objects will be implemented. * All allocated objects with new operator will be automatically allocated
Re: [PROPOSAL] jclouds Proposal for Incubator
I contributed some code back in the day, so happy to help too. On Tue, Apr 16, 2013 at 9:24 PM, Jean-Baptiste Onofré j...@nanthrax.netwrote: I'm very interested as well. I'm also volunteer to be mentor on the project. Regards JB On 04/16/2013 09:19 PM, David Nalley wrote: On Tue, Apr 16, 2013 at 2:34 PM, Rebecca Wood silky...@gmail.com wrote: Dear ASF members, We would like to propose the jclouds project to the Incubator. The jclouds Proposal is available at: http://wiki.apache.org/**incubator/jcloudsProposalhttp://wiki.apache.org/incubator/jcloudsProposal We welcome your feedback and suggestions. Excited to see jclouds proposal. If you are looking for additional mentors, I'd be happy to step up. --David --**--**- To unsubscribe, e-mail: general-unsubscribe@incubator.**apache.orggeneral-unsubscr...@incubator.apache.org For additional commands, e-mail: general-help@incubator.apache.**orggeneral-h...@incubator.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com --**--**- To unsubscribe, e-mail: general-unsubscribe@incubator.**apache.orggeneral-unsubscr...@incubator.apache.org For additional commands, e-mail: general-help@incubator.apache.**orggeneral-h...@incubator.apache.org
Re: [PROPOSAL] jclouds Proposal for Incubator
Thanks, Carlos, both for the code bidd as well offering to be our 10th mentor :) Now, I promised mattstep that if we got to 10 mentors, we'd start creating mentor titles. I was thinking Eternal mentor, Supreme mentor, but quickly ran out of ideas. Can you take this on? :D On Tue, Apr 16, 2013 at 12:26 PM, Carlos Sanchez car...@apache.org wrote: I contributed some code back in the day, so happy to help too. On Tue, Apr 16, 2013 at 9:24 PM, Jean-Baptiste Onofré j...@nanthrax.net wrote: I'm very interested as well. I'm also volunteer to be mentor on the project. Regards JB On 04/16/2013 09:19 PM, David Nalley wrote: On Tue, Apr 16, 2013 at 2:34 PM, Rebecca Wood silky...@gmail.com wrote: Dear ASF members, We would like to propose the jclouds project to the Incubator. The jclouds Proposal is available at: http://wiki.apache.org/**incubator/jcloudsProposal http://wiki.apache.org/incubator/jcloudsProposal We welcome your feedback and suggestions. Excited to see jclouds proposal. If you are looking for additional mentors, I'd be happy to step up. --David --**--**- To unsubscribe, e-mail: general-unsubscribe@incubator.**apache.org general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-help@incubator.apache.**org general-h...@incubator.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com --**--**- To unsubscribe, e-mail: general-unsubscribe@incubator.**apache.org general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-help@incubator.apache.**org general-h...@incubator.apache.org
Re: [VOTE] Release JSPWiki version 2.9.1-incubating
On Tue, Apr 16, 2013 at 2:24 PM, Juan Pablo Santos Rodríguez juanpablo.san...@gmail.com wrote: - I'm especially worried about the NOTICE file, how short is short enough? Could you provide a good example from any other project so we can take a look at it? To take a grasp of how we should do it. We'd like to have it as good as the LICENSE file (also, the incorrect year in NOTICE was also pointed by Siegfried, but not as a blocker, so it's fixed in trunk). The ASF now has documentation which describes a formulaic approach to creating LICENSE and NOTICE. If you follow the recipe, you should be OK. http://www.apache.org/dev/licensing-howto.html Marvin Humphrey - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Marmotta 3.0.0-incubating (RC8)
+1 (non-binding) - checked signatures/digests - installer: installed and started - src: installed Thanks to the dev team for all the great work done! On 16 April 2013 12:36, Sergio Fernández sergio.fernan...@salzburgresearch.at wrote: Hi, as Sebastian pointed, from the Marmotta PMC we'd find quite interesting all your comments regarding this first incubating release. In the last weeks we have done an important effort to clarify many details, and finally I think we got something pretty close to what it should be; but we'd need to certify this is actually true. Thanks for all your support. Kind regards, On 15/04/13 15:40, Sebastian Schaffert wrote: Dear all, Apache Marmotta is an implementation of the Linked Data Platform and accompanying services. In the last months, we have migrated our old source code to the Apache infrastructure and tried to adapt the Apache policies in all parts of the project. In particular, we have fixed all remaining licensing issues and spent much time on developing the appropriate NOTICE and LICENSE files. We have executed 8 voting cycles on release candidates throughout the last weeks and we think we are now (finally) ready to really do the release. The vote on release candidate RC8 has been open for more than 72 hours on the developer mailinglist [1]. After closing the vote, we have the following voting results: - 2 binding votes (from the mentors Andy Seaborne and Nandana Mihindukulasooriya) - 6 non-binding votes (the remaining developers) I'd therefore like to ask now the general incubator to check our release candidate. Since this is our first release, please check thoroughly, I'd like to be sure we did not miss anything. The release notes (fixed issues) are available at the Jira Issue Tracker [2]. [1] http://markmail.org/message/**uncutapltb3mopgy?q=list:org%** 2Eapache%2Emarmotta%2Edevhttp://markmail.org/message/uncutapltb3mopgy?q=list:org%2Eapache%2Emarmotta%2Edev [2] https://issues.apache.org/**jira/secure/ReleaseNote.jspa?** projectId=12314321version=**12323952https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314321version=12323952 Please find attached below the concrete details on the release and on the vote. Greetings, Sebastian ==**= A candidate for the Marmotta 3.0.0-incubating release is available at: https://dist.apache.org/repos/**dist/dev/incubator/marmotta/3.** 0.0-incubating/https://dist.apache.org/repos/dist/dev/incubator/marmotta/3.0.0-incubating/ The release candidate is based on the sources tagged with 3.0.0-incubating in: https://git-wip-us.apache.org/**repos/asf/incubator-marmotta.**githttps://git-wip-us.apache.org/repos/asf/incubator-marmotta.git The release candidate consists of the following source distribution archives: - apache-marmotta-3.0.0-**incubating-src.[zip|tar.gz] SHA1 of ZIP: ac0cc9aa641a37fea96adaf128e44b**0d6a06cc8f SHA1 of TGZ: e176b0b7816e5c334a3d8dccb960cd**a0b64d4b4c In addition, the following supplementary binary distributions are provided: - apache-marmotta-3.0.0-**incubating-installer.[zip|tar.**gz] SHA1 of ZIP: 4e5bf3e5c59e04bea3776d76c37990**fa1efd6995 SHA1 of TGZ: a8736dd1700c7993762dea0446a23c**2d314e03ec - apache-marmotta-3.0.0-**incubating-ldpath.[zip|tar.gz] SHA1 of ZIP: 3dab792c935bf0380c98fb2767ea75**653df3b9b1 SHA1 of TGZ: 237040387801aa6538fd07e3c75265**b06e3d4ce3 - apache-marmotta-3.0.0-**incubating-webapp.[zip|tar.gz] SHA1 of ZIP: 7784ee71b665782f1011a8d1333524**c825b90c6d SHA1 of TGZ: 217e17709e6e84ffe755841ac69f85**0538e867f7 A staged Maven repository is available for review at: https://repository.apache.org/**content/repositories/** orgapachemarmotta-093/https://repository.apache.org/content/repositories/orgapachemarmotta-093/ Please vote on releasing this package as Apache Marmotta 3.0.0-incubating. The vote is open for the next 72 hours and passes if a majority of at least three +1 Marmotta PMC votes are cast. [ ] +1 Release this package as Apache Marmotta 3.0.0-incubating [ ] 0 I don't feel strongly about it, but I'm okay with the release [ ] -1 Do not release this package because... -- Sergio Fernández Salzburg Research +43 662 2288 318 Jakob-Haringer Strasse 5/II A-5020 Salzburg (Austria) http://www.salzburgresearch.at --**--**- To unsubscribe, e-mail: general-unsubscribe@incubator.**apache.orggeneral-unsubscr...@incubator.apache.org For additional commands, e-mail: general-help@incubator.apache.**orggeneral-h...@incubator.apache.org
Re: [VOTE] Release Apache Marmotta 3.0.0-incubating (RC8)
On 15 April 2013 14:40, Sebastian Schaffert sschaff...@apache.org wrote: Dear all, Apache Marmotta is an implementation of the Linked Data Platform and accompanying services. In the last months, we have migrated our old source code to the Apache infrastructure and tried to adapt the Apache policies in all parts of the project. In particular, we have fixed all remaining licensing issues and spent much time on developing the appropriate NOTICE and LICENSE files. We have executed 8 voting cycles on release candidates throughout the last weeks and we think we are now (finally) ready to really do the release. The vote on release candidate RC8 has been open for more than 72 hours on the developer mailinglist [1]. After closing the vote, we have the following voting results: - 2 binding votes (from the mentors Andy Seaborne and Nandana Mihindukulasooriya) - 6 non-binding votes (the remaining developers) I'd therefore like to ask now the general incubator to check our release candidate. Since this is our first release, please check thoroughly, I'd like to be sure we did not miss anything. The release notes (fixed issues) are available at the Jira Issue Tracker [2]. [1] http://markmail.org/message/uncutapltb3mopgy?q=list:org%2Eapache%2Emarmotta%2Edev [2] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314321version=12323952 Please find attached below the concrete details on the release and on the vote. Greetings, Sebastian === A candidate for the Marmotta 3.0.0-incubating release is available at: https://dist.apache.org/repos/dist/dev/incubator/marmotta/3.0.0-incubating/ The release candidate is based on the sources tagged with 3.0.0-incubating in: https://git-wip-us.apache.org/repos/asf/incubator-marmotta.git The NOTICE file in the source archive and Git tree looks rather big - are all of those really present in the source? Also the formatting of the header is non-standard. The release candidate consists of the following source distribution archives: - apache-marmotta-3.0.0-incubating-src.[zip|tar.gz] SHA1 of ZIP: ac0cc9aa641a37fea96adaf128e44b0d6a06cc8f SHA1 of TGZ: e176b0b7816e5c334a3d8dccb960cda0b64d4b4c In addition, the following supplementary binary distributions are provided: - apache-marmotta-3.0.0-incubating-installer.[zip|tar.gz] SHA1 of ZIP: 4e5bf3e5c59e04bea3776d76c37990fa1efd6995 SHA1 of TGZ: a8736dd1700c7993762dea0446a23c2d314e03ec - apache-marmotta-3.0.0-incubating-ldpath.[zip|tar.gz] SHA1 of ZIP: 3dab792c935bf0380c98fb2767ea75653df3b9b1 SHA1 of TGZ: 237040387801aa6538fd07e3c75265b06e3d4ce3 - apache-marmotta-3.0.0-incubating-webapp.[zip|tar.gz] SHA1 of ZIP: 7784ee71b665782f1011a8d1333524c825b90c6d SHA1 of TGZ: 217e17709e6e84ffe755841ac69f850538e867f7 A staged Maven repository is available for review at: https://repository.apache.org/content/repositories/orgapachemarmotta-093/ Please vote on releasing this package as Apache Marmotta 3.0.0-incubating. The vote is open for the next 72 hours and passes if a majority of at least three +1 Marmotta PMC votes are cast. [ ] +1 Release this package as Apache Marmotta 3.0.0-incubating [ ] 0 I don't feel strongly about it, but I'm okay with the release [ ] -1 Do not release this package because...
Re: [PROPOSAL] - Java OffHeap Memory Pool
Is code for this available for review anywhere? -- Brane On 16.04.2013 19:46, serkan özal wrote: Project Name: Jillegal 1. Abstract: GC is one of the time taken operations in Java. GC run anytime, marks, swaps and compacts objects at memory. If there are so many live objects, managing them by GC leads to overhead. If objects can be allocated outside of GC, there will be no overhead for the application. The session will go through the new method of creating and using object off-heap with no additional serialization/deserialization or any other overheads. 2. Proposal: For off-heap memory, I propose a solution that objects are allocated and initialized at off-heap instead of heap. Not only object attributes are allocated at off-heap, but also object header and metadata are allocated at off-heap. So while a reference to this object at off-heap is being interpreted, JVM knows which class this object references to. You can get your off-heap object from an object pool instead of with new operator. This object pool allocates a fixed size memory region from off-heap and create empty objects with given class on there. These empty objects can be created as lazy or eager. Another advantage off this technique is that objects in pool are layout in memory as sequential. While accessing an object, its neighbour objects are also fetched to CPU cache, so CPU cache hit rate for sequential object accesses are increased. On the other hand, freeing unused objects is responsibility of developer by calling free method of object pool which means there is no dead object detection mechanism like GC. Therefore, getting all objects from off-heap for whole application is dangerous and not recommended. Because, this will cause memory leaks. In addition, this technique can be combined with Java Instrumentation API. With @FromOffHeap annotation, developer can sign classes these must be allocated from off-heap instead of heap. With Java Instrumentation API, all new operators for signed classes with @FromOffHeap annotation can ben transformed to code that allocates objects of signed class from off-heap via object pool. So developer doesn't change all new keywords for getting from object pool in code. Instread of this, just sign class with @FromOffHeap annotation and all new keywords transformed for getting from object pool at class load time. This technique was used at a real time CDR processing system for Turk Telekom. There were billions of objects were used. Managing these objects by GC caused to performance overhead. For some most used classes, we allocated these objects from off-heap instead of new keyword. After some processings on them (takes 4-5 hours), we release these allocated memory regions to operating system by freeing them. Allocating objects from off-heap pool helps us to gain significant execution time performance. 3. Rationale: In general, off-heap pool implementations are implemented by serialization/deserialization to allocated off-heap memory region via ByteBuffer class. But this technique leads to extra execution overhead. Because while reading from an object, the target object must be created by deserializing all primitive fields eagerly or only required fields on demand and while writing to an object, the attribute has been set by application, must be deserialized to allocated off-heap memory region. In addition, objects itself is created at heap, so GC knows and tracks it. With my solution, all of these overheads are overcomed. 4. Initial Goals: * Allocating objects from off-heap and using them as normal on-heap Java object. * Allocating arrays for object types from off-heap and using them as normal on-heap Java object arrays. * Allocating arrays for primitive types from off-heap and using them as normal on-heap Java primitive type arrays. * Allocating strings from off-heap and using them as normal on-heap strings. * Implementing auto expandable off-heap pool that expands when its delegated off-heap pool implementation is full. * All features must be supported for 32 bit and 64 bit JVM. * All features must be supported for Sun HotSpot JVM, Oracle JRockit, IBM J9. 5. Currently Implemented Features: * Allocating objects from off-heap and using them as normal on-heap Java object * Allocating arrays for object types from off-heap and using them as normal on-heap Java object array * Allocating arrays for primitive types from off-heap and using them as normal on-heap Java primitive type arrays * Implementing auto expandable off-heap pool that expands when its delegated off-heap pool implementation is full. * All features are supported for 32 bit and 64 bit JVM. * All features are supported for Sun HotSpot JVM, Oracle JRockit (IBM J9 support will be added). 6. Roadmap
Re: [VOTE] Release Apache Marmotta 3.0.0-incubating (RC8)
Hi, first of all, thanks for taking the time to review the release; we really appreciate it. On 17/04/13 17:52, sebb wrote: The NOTICE file in the source archive and Git tree looks rather big - are all of those really present in the source? The main reason is that Marmotta uses many Javascript libraries. Therefore, until we'll move to depend on webjars artifacts, those libraries are present in the source tree and distributed, both in the source and binary distributions. This has been internally discussed too; please see http://markmail.org/message/ydnk257k4p6tapkg for further details. Basically, although there is no need to add them because the license, for the moment we find useful to have a single audit point for such cases. Future releases will gradually reduce that list. Also the formatting of the header is non-standard. Ummm, you are right: our formatting is slightly different than the standard one. Actually I don't know where this custom formatting comes from, but I hope it wouldn't be a blocking issue. I already registered MARMOTTA-206 on jira to handle it properly; thanks for pointing us such detail. Kind regards, -- Sergio Fernández Salzburg Research +43 662 2288 318 Jakob-Haringer Strasse 5/II A-5020 Salzburg (Austria) http://www.salzburgresearch.at - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Marmotta 3.0.0-incubating (RC8)
Hi sebb, there are different NOTICE files for the different distributions, because the binary distributions are rather framework applications that bundle a lot of third party libraries. - The NOTICE file for the source distribution contains the notices for the 3rd party source code and data files contained in the source distribution and excludes the dependencies. We had to include some 3rd party source code temporarily, because the libraries are not available via Maven or we needed to patch the code. The NOTICE files for the different binary distributions also contain the notices for the binary dependencies we are bundling, because users of course need to be aware of any licensing issues that might arise when using the binary distributions. Each binary distribution will therefore have a different NOTICE file. We had long discussions about the content of the NOTICE files. We followed as much as possible the guidelines on the ASF websites (which are sometimes a bit contradictory) and also the examples of several other well-known Apache projects that are of similar nature (e.g. Apache SOLR also has similar NOTICE files, likewise ODE and Geronimo). See e.g. the original discussion [1] and the refinements we did after the recommendations of our mentors [2]. [1] http://markmail.org/message/rrvpchj7jetn3yj7?q=list:org%2Eapache%2Emarmotta%2Edev+notice [2] http://markmail.org/message/tnxoyavqwykgrzhf?q=list:org%2Eapache%2Emarmotta%2Edev+notice In case you have suggestions for improvement, I am happy to accept any feedback you might have. Even after several weeks of digging through licenses and notices, I am still a software engineer and not a lawyer. In case of doubt, we therefore decided to stay on the safe side and rather add a notice than forget about it. I don't want any users with legal problems ... ;-) Greetings, Sebastian 2013/4/17 sebb seb...@gmail.com On 15 April 2013 14:40, Sebastian Schaffert sschaff...@apache.org wrote: Dear all, Apache Marmotta is an implementation of the Linked Data Platform and accompanying services. In the last months, we have migrated our old source code to the Apache infrastructure and tried to adapt the Apache policies in all parts of the project. In particular, we have fixed all remaining licensing issues and spent much time on developing the appropriate NOTICE and LICENSE files. We have executed 8 voting cycles on release candidates throughout the last weeks and we think we are now (finally) ready to really do the release. The vote on release candidate RC8 has been open for more than 72 hours on the developer mailinglist [1]. After closing the vote, we have the following voting results: - 2 binding votes (from the mentors Andy Seaborne and Nandana Mihindukulasooriya) - 6 non-binding votes (the remaining developers) I'd therefore like to ask now the general incubator to check our release candidate. Since this is our first release, please check thoroughly, I'd like to be sure we did not miss anything. The release notes (fixed issues) are available at the Jira Issue Tracker [2]. [1] http://markmail.org/message/uncutapltb3mopgy?q=list:org%2Eapache%2Emarmotta%2Edev [2] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314321version=12323952 Please find attached below the concrete details on the release and on the vote. Greetings, Sebastian === A candidate for the Marmotta 3.0.0-incubating release is available at: https://dist.apache.org/repos/dist/dev/incubator/marmotta/3.0.0-incubating/ The release candidate is based on the sources tagged with 3.0.0-incubating in: https://git-wip-us.apache.org/repos/asf/incubator-marmotta.git The NOTICE file in the source archive and Git tree looks rather big - are all of those really present in the source? Also the formatting of the header is non-standard. The release candidate consists of the following source distribution archives: - apache-marmotta-3.0.0-incubating-src.[zip|tar.gz] SHA1 of ZIP: ac0cc9aa641a37fea96adaf128e44b0d6a06cc8f SHA1 of TGZ: e176b0b7816e5c334a3d8dccb960cda0b64d4b4c In addition, the following supplementary binary distributions are provided: - apache-marmotta-3.0.0-incubating-installer.[zip|tar.gz] SHA1 of ZIP: 4e5bf3e5c59e04bea3776d76c37990fa1efd6995 SHA1 of TGZ: a8736dd1700c7993762dea0446a23c2d314e03ec - apache-marmotta-3.0.0-incubating-ldpath.[zip|tar.gz] SHA1 of ZIP: 3dab792c935bf0380c98fb2767ea75653df3b9b1 SHA1 of TGZ: 237040387801aa6538fd07e3c75265b06e3d4ce3 - apache-marmotta-3.0.0-incubating-webapp.[zip|tar.gz] SHA1 of ZIP: 7784ee71b665782f1011a8d1333524c825b90c6d SHA1 of TGZ: 217e17709e6e84ffe755841ac69f850538e867f7 A staged Maven repository is available for review at: https://repository.apache.org/content/repositories/orgapachemarmotta-093/ Please vote on
Re: [VOTE] Release JSPWiki version 2.9.1-incubating
Hi Marvin, Sebb, thanks for the insights :-) br, juan pablo On Wed, Apr 17, 2013 at 5:13 PM, Marvin Humphrey mar...@rectangular.comwrote: On Tue, Apr 16, 2013 at 2:24 PM, Juan Pablo Santos Rodríguez juanpablo.san...@gmail.com wrote: - I'm especially worried about the NOTICE file, how short is short enough? Could you provide a good example from any other project so we can take a look at it? To take a grasp of how we should do it. We'd like to have it as good as the LICENSE file (also, the incorrect year in NOTICE was also pointed by Siegfried, but not as a blocker, so it's fixed in trunk). The ASF now has documentation which describes a formulaic approach to creating LICENSE and NOTICE. If you follow the recipe, you should be OK. http://www.apache.org/dev/licensing-howto.html Marvin Humphrey - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Marmotta 3.0.0-incubating (RC8)
On 17 April 2013 17:35, Sebastian Schaffert sschaff...@apache.org wrote: Hi sebb, there are different NOTICE files for the different distributions, because the binary distributions are rather framework applications that bundle a lot of third party libraries. That's as it should be. - The NOTICE file for the source distribution contains the notices for the 3rd party source code and data files contained in the source distribution and excludes the dependencies. We had to include some 3rd party source code temporarily, because the libraries are not available via Maven or we needed to patch the code. The NOTICE files for the different binary distributions also contain the notices for the binary dependencies we are bundling, because users of course need to be aware of any licensing issues that might arise when using the binary distributions. Each binary distribution will therefore have a different NOTICE file. We had long discussions about the content of the NOTICE files. We followed as much as possible the guidelines on the ASF websites (which are sometimes a bit contradictory) Please can you advise where the contradictions are so they can be sorted out? and also the examples of several other well-known Apache projects that are of similar nature (e.g. Apache SOLR also has similar NOTICE files, likewise ODE and Geronimo). See e.g. the original discussion [1] and the refinements we did after the recommendations of our mentors [2]. From Marvin's recent post in another VOTE thread: The ASF now has documentation which describes a formulaic approach to creating LICENSE and NOTICE. If you follow the recipe, you should be OK. http://www.apache.org/dev/licensing-howto.html [1] http://markmail.org/message/rrvpchj7jetn3yj7?q=list:org%2Eapache%2Emarmotta%2Edev+notice [2] http://markmail.org/message/tnxoyavqwykgrzhf?q=list:org%2Eapache%2Emarmotta%2Edev+notice In case you have suggestions for improvement, I am happy to accept any feedback you might have. Even after several weeks of digging through licenses and notices, I am still a software engineer and not a lawyer. In case of doubt, we therefore decided to stay on the safe side and rather add a notice than forget about it. I don't want any users with legal problems ... ;-) Greetings, Sebastian 2013/4/17 sebb seb...@gmail.com On 15 April 2013 14:40, Sebastian Schaffert sschaff...@apache.org wrote: Dear all, Apache Marmotta is an implementation of the Linked Data Platform and accompanying services. In the last months, we have migrated our old source code to the Apache infrastructure and tried to adapt the Apache policies in all parts of the project. In particular, we have fixed all remaining licensing issues and spent much time on developing the appropriate NOTICE and LICENSE files. We have executed 8 voting cycles on release candidates throughout the last weeks and we think we are now (finally) ready to really do the release. The vote on release candidate RC8 has been open for more than 72 hours on the developer mailinglist [1]. After closing the vote, we have the following voting results: - 2 binding votes (from the mentors Andy Seaborne and Nandana Mihindukulasooriya) - 6 non-binding votes (the remaining developers) I'd therefore like to ask now the general incubator to check our release candidate. Since this is our first release, please check thoroughly, I'd like to be sure we did not miss anything. The release notes (fixed issues) are available at the Jira Issue Tracker [2]. [1] http://markmail.org/message/uncutapltb3mopgy?q=list:org%2Eapache%2Emarmotta%2Edev [2] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314321version=12323952 Please find attached below the concrete details on the release and on the vote. Greetings, Sebastian === A candidate for the Marmotta 3.0.0-incubating release is available at: https://dist.apache.org/repos/dist/dev/incubator/marmotta/3.0.0-incubating/ The release candidate is based on the sources tagged with 3.0.0-incubating in: https://git-wip-us.apache.org/repos/asf/incubator-marmotta.git The NOTICE file in the source archive and Git tree looks rather big - are all of those really present in the source? Also the formatting of the header is non-standard. The release candidate consists of the following source distribution archives: - apache-marmotta-3.0.0-incubating-src.[zip|tar.gz] SHA1 of ZIP: ac0cc9aa641a37fea96adaf128e44b0d6a06cc8f SHA1 of TGZ: e176b0b7816e5c334a3d8dccb960cda0b64d4b4c In addition, the following supplementary binary distributions are provided: - apache-marmotta-3.0.0-incubating-installer.[zip|tar.gz] SHA1 of ZIP: 4e5bf3e5c59e04bea3776d76c37990fa1efd6995 SHA1 of
Re: [VOTE] Release Apache Marmotta 3.0.0-incubating (RC8)
Hi Sebb, 2013/4/17 sebb seb...@gmail.com We had long discussions about the content of the NOTICE files. We followed as much as possible the guidelines on the ASF websites (which are sometimes a bit contradictory) Please can you advise where the contradictions are so they can be sorted out? Yes, sure. There are currently the following documents (at least the ones I am aware of) that are somehow related to LICENSE and NOTICE: 1. http://www.apache.org/dev/licensing-howto.html 2. http://www.apache.org/legal/3party.html 3. http://www.apache.org/legal/src-headers.html 4. http://incubator.apache.org/guides/releasemanagement.html#best-practice-license While similar in the most important issues, all four documents vary in some of the details. For example: - The section NOTICE file in [3] says that The NOTICE file may also include copyright notices moved from source files submitted to the ASF. The 3rd party Javascript files we include are submitted to the ASF, but they are often under MIT license and therefore category A in [2]. If I read the document correctly, category A only requires to include notices if a NOTICE file is contained in the product. According to [1], MIT and New BSD even can have a reduced entry in LICENSE. [4] says, however, that All the licenses on all the files to be included within a package should be included in the LICENSE document. - for dependencies of category B, [2] specifies that Although the source must not be included in Apache products, the NOTICE file, which is required to be included in each ASF distribution, must point to the source form of the included binary (more on that in the forthcoming Receiving and Releasing Contributions document)., a fact that is not mentioned in any of the other documents. - The labelling requirement source access in [2] requires that the NOTICE file contains pointers to the location of the source code for any 3rd party library that is bundled in a binary distribution (not only category B). [1] on the other hand does not mention this requirement and says that everything that is not legally necessary does not belong into NOTICE. - [1] says that NOTICE must only include notices not satisfied by licensing information embedded within the dependency subtree. [2] says Users of Apache products must be provided with all licensing terms applicable to any part of the product and must be given prominent notice as fourth guiding principle (emphasis on *prominent*, which for me means not somewhere in the dependency subtree) Maybe contradictory was too strong a word. But all the four documents are incomplete in the guidance they provide. What I found very helpful, though, are the 4 guiding principles mentioned in [2]. Also very helpful was looking at how other projects are doing it, in particular we looked at SOLR and Geronimo, as they are also web frameworks bundling many different libraries. For example, SOLR has a very extensive NOTICE file: http://svn.apache.org/repos/asf/lucene/dev/trunk/solr/NOTICE.txt For future podlings, what would be very helpful is to improve the document [1] as a single point of guidance that collects all the information from the other documents (if it is correct). In particular, best practices for different types of projects would also help a lot, especially to people who are not legal experts (like me). Because apparently, many of the top level projects (like SOLR) are not really best practices. What would also help is to give more understanding what a notice actually means. For example, if I understand correctly, the Apache License, when used by 3rd party libraries, always requires notice. How can I see whether other licenses also require this notice? As an example, consider the MIT license. It explicitly says The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.. Does this mean that a copyright notice must go into the NOTICE file (under guiding principle number 4 I would read it like yes)? If so, then there is a contradiction with [1]. and also the examples of several other well-known Apache projects that are of similar nature (e.g. Apache SOLR also has similar NOTICE files, likewise ODE and Geronimo). See e.g. the original discussion [1] and the refinements we did after the recommendations of our mentors [2]. From Marvin's recent post in another VOTE thread: The ASF now has documentation which describes a formulaic approach to creating LICENSE and NOTICE. If you follow the recipe, you should be OK. http://www.apache.org/dev/licensing-howto.html We followed this document very closely (I can almost recite it in my sleep). If the recipe was complete, yes :-) But recipes typically do not contain sentences like NOTICE is reserved for a certain subset of legally required notifications. :-P Thanks for checking, BTW. From your point of view, is the NOTICE and LICENSE we have for Marmotta too extensive? Where
LICENSE/NOTICE revisited (was Release Apache Marmotta 3.0.0-incubating (RC8))
On Wed, Apr 17, 2013 at 11:00 AM, Sebastian Schaffert sschaff...@apache.org wrote: We had long discussions about the content of the NOTICE files. We followed as much as possible the guidelines on the ASF websites (which are sometimes a bit contradictory) Please can you advise where the contradictions are so they can be sorted out? Yes, sure. Thank you very much for working hard to get LICENSE and NOTICE right, and for providing us with this valuable feedback. There are currently the following documents (at least the ones I am aware of) that are somehow related to LICENSE and NOTICE: 1. http://www.apache.org/dev/licensing-howto.html 2. http://www.apache.org/legal/3party.html 3. http://www.apache.org/legal/src-headers.html 4. http://incubator.apache.org/guides/releasemanagement.html#best-practice-license For what it's worth, the last document in your list is significantly less reliable than the others. It's a goal of ours to remove all those best practice recommendations and (frequently erroneous) duplicated policy assertions from that page, whittling it down until it contains only Incubator-specific requirements. While similar in the most important issues, all four documents vary in some of the details. For example: - The section NOTICE file in [3] says that The NOTICE file may also include copyright notices moved from source files submitted to the ASF. The 3rd party Javascript files we include are submitted to the ASF, but they are often under MIT license and therefore category A in [2]. If I read the document correctly, category A only requires to include notices if a NOTICE file is contained in the product. According to [1], MIT and New BSD even can have a reduced entry in LICENSE. [4] says, however, that All the licenses on all the files to be included within a package should be included in the LICENSE document. Here's Roy Fielding, ASF Board member and author of the Apache License 2.0: http://s.apache.org/Hqj Pointers are sufficient. Roy elaborates here: http://s.apache.org/Iw6 By pointers, I mean a relative path reference to the license file within some subdirectory of the package being redistributed. It used to be quite common for such license files to be alongside the respective jar in a lib or srclib directory, or inside the third party directory containing the source. The top-level LICENSE can simply point to those files (with a simple description like under the MIT license) rather than include all of the text verbatim, particularly when the same license is repeated for many optional libraries. In any case, the intention is to allow use of whatever form seems most appropriate for the given distribution form, while still satisfying the legal requirements for each licensed work. Some projects will want to include all the text in one file. Others may not. The releasemanagement.html page is wrong. I will delete the offending section, as it offers no information that isn't available elsewhere. - for dependencies of category B, [2] specifies that Although the source must not be included in Apache products, the NOTICE file, which is required to be included in each ASF distribution, must point to the source form of the included binary (more on that in the forthcoming Receiving and Releasing Contributions document)., a fact that is not mentioned in any of the other documents. This passage has somehow escaped my notice until now. Based on my understanding about the origins of the NOTICE file, it does not ring true. It seems to me that what works for category A should also work for category B: reference/quote the license in LICENSE and address mandatory attribution requirements in NOTICE. The goal is to satisfy the licensing requirements of the dependency, not to give credit -- so IMO linking only makes sense if that's a requirement of the dependency's license. Does anybody know any TLPs that are actually following the advice to link to source for category B dependencies in binary NOTICE files? I suspect we will want to bring this up on legal-discuss and see if we can't get that recommendation removed. - The labelling requirement source access in [2] requires that the NOTICE file contains pointers to the location of the source code for any 3rd party library that is bundled in a binary distribution (not only category B). [1] on the other hand does not mention this requirement and says that everything that is not legally necessary does not belong into NOTICE. Here's Roy again: http://s.apache.org/ZEA Hey, I'm all for people having opinions on development and credits and documentation. NOTICE and LICENSE are none of those. They are not open to anyone's opinions other than the copyright owners that require such notices, and they must not be added where they are not required. Each additional notice places a burden on the ASF and
Re: [PROPOSAL] - Java OffHeap Memory Pool
Hi, Could that be part of directmemory as sub project ? 2013/4/17 serkan özal serkanoza...@hotmail.com: Project Name: Jillegal 1. Abstract: GC is one of the time taken operations in Java. GC run anytime, marks, swaps and compacts objects at memory. If there are so many live objects, managing them by GC leads to overhead. If objects can be allocated outside of GC, there will be no overhead for the application. The session will go through the new method of creating and using object off-heap with no additional serialization/deserialization or any other overheads. 2. Proposal: For off-heap memory, I propose a solution that objects are allocated and initialized at off-heap instead of heap. Not only object attributes are allocated at off-heap, but also object header and metadata are allocated at off-heap. So while a reference to this object at off-heap is being interpreted, JVM knows which class this object references to. You can get your off-heap object from an object pool instead of with new operator. This object pool allocates a fixed size memory region from off-heap and create empty objects with given class on there. These empty objects can be created as lazy or eager. Another advantage off this technique is that objects in pool are layout in memory as sequential. While accessing an object, its neighbour objects are also fetched to CPU cache, so CPU cache hit rate for sequential object accesses are increased. On the other hand, freeing unused objects is responsibility of developer by calling free method of object pool which means there is no dead object detection mechanism like GC. Therefore, getting all objects from off-heap for whole application is dangerous and not recommended. Because, this will cause memory leaks. In addition, this technique can be combined with Java Instrumentation API. With @FromOffHeap annotation, developer can sign classes these must be allocated from off-heap instead of heap. With Java Instrumentation API, all new operators for signed classes with @FromOffHeap annotation can ben transformed to code that allocates objects of signed class from off-heap via object pool. So developer doesn't change all new keywords for getting from object pool in code. Instread of this, just sign class with @FromOffHeap annotation and all new keywords transformed for getting from object pool at class load time. This technique was used at a real time CDR processing system for Turk Telekom. There were billions of objects were used. Managing these objects by GC caused to performance overhead. For some most used classes, we allocated these objects from off-heap instead of new keyword. After some processings on them (takes 4-5 hours), we release these allocated memory regions to operating system by freeing them. Allocating objects from off-heap pool helps us to gain significant execution time performance. 3. Rationale: In general, off-heap pool implementations are implemented by serialization/deserialization to allocated off-heap memory region via ByteBuffer class. But this technique leads to extra execution overhead. Because while reading from an object, the target object must be created by deserializing all primitive fields eagerly or only required fields on demand and while writing to an object, the attribute has been set by application, must be deserialized to allocated off-heap memory region. In addition, objects itself is created at heap, so GC knows and tracks it. With my solution, all of these overheads are overcomed. 4. Initial Goals: * Allocating objects from off-heap and using them as normal on-heap Java object. * Allocating arrays for object types from off-heap and using them as normal on-heap Java object arrays. * Allocating arrays for primitive types from off-heap and using them as normal on-heap Java primitive type arrays. * Allocating strings from off-heap and using them as normal on-heap strings. * Implementing auto expandable off-heap pool that expands when its delegated off-heap pool implementation is full. * All features must be supported for 32 bit and 64 bit JVM. * All features must be supported for Sun HotSpot JVM, Oracle JRockit, IBM J9. 5. Currently Implemented Features: * Allocating objects from off-heap and using them as normal on-heap Java object * Allocating arrays for object types from off-heap and using them as normal on-heap Java object array * Allocating arrays for primitive types from off-heap and using them as normal on-heap Java primitive type arrays * Implementing auto expandable off-heap pool that expands when its delegated off-heap pool implementation is full. * All features are supported for 32 bit and 64 bit JVM. * All features are supported for Sun HotSpot JVM, Oracle JRockit (IBM J9 support will be added). 6. Roadmap
