Re: Using of final for variables and parameters
On 2015-08-28 08:36, Thomas Mueller wrote: Hi, I wonder what does the team think about using final for variables and parameters. In Oak, so far we didn't use it a lot. This question has come up with OAK-3148. The patch uses final for variables, but not for parameters. Lately, I have seens some code (I forgot where) that uses final for parameters in an _interface_. Please note this is not about using final for fields, where I think everybody agrees it should be used. It's just about variables and parameters. I think we have 3 options: (a) use final for variables and parameters everywhere possible (b) let the individual developer decide (c) don't use it except if needed Some links: http://stackoverflow.com/questions/154314/when-should-one-use-final-for-method-parameters-and-local-variables http://stackoverflow.com/questions/137868/using-final-modifier-whenever-applicable-in-java Personally, my favorite is (c), followed by (b). As for (a), I think (same as Alex Miller at StackOverflow) it clutters the code and decreases readability. Too bad final is not the default in Java, but Java will not change, and we are stuck with Java. I think using final will not improve the code, because people don't accidentally change variables and parameters, so it will not help the writer of a method, it will not help the compiler or performance (the JVM can easily see if a variable or parameter is effectively assigned only once). To improve the code, I'm all for using Checkstyle, unit tests, code coverage, mutation testing, enforcing to write Javadocs for interfaces, and so on. But using final wherever possible, I think it would be a step backwards. Regards, Thomas +1. (c)!
Using of final for variables and parameters
Hi, I wonder what does the team think about using final for variables and parameters. In Oak, so far we didn't use it a lot. This question has come up with OAK-3148. The patch uses final for variables, but not for parameters. Lately, I have seens some code (I forgot where) that uses final for parameters in an _interface_. Please note this is not about using final for fields, where I think everybody agrees it should be used. It's just about variables and parameters. I think we have 3 options: (a) use final for variables and parameters everywhere possible (b) let the individual developer decide (c) don't use it except if needed Some links: http://stackoverflow.com/questions/154314/when-should-one-use-final-for-method-parameters-and-local-variables http://stackoverflow.com/questions/137868/using-final-modifier-whenever-applicable-in-java Personally, my favorite is (c), followed by (b). As for (a), I think (same as Alex Miller at StackOverflow) it clutters the code and decreases readability. Too bad final is not the default in Java, but Java will not change, and we are stuck with Java. I think using final will not improve the code, because people don't accidentally change variables and parameters, so it will not help the writer of a method, it will not help the compiler or performance (the JVM can easily see if a variable or parameter is effectively assigned only once). To improve the code, I'm all for using Checkstyle, unit tests, code coverage, mutation testing, enforcing to write Javadocs for interfaces, and so on. But using final wherever possible, I think it would be a step backwards. Regards, Thomas
Re: Using of final for variables and parameters
Hi,
- I know that the variable ³state created at the beginning of the method
is the same one I can access at the end.
For short methods, it's easy to see this, by reading the code. In my view
easier than using final, which makes the code less readable.
For large methods,... well you should avoid large methods :-) But sure,
for large methods, using final is OK for me, if you have variables on
the outer level. But for example here:
for (final String id : newBlobIds) {
assertTrue(splitBlobStore.isMigrated(id));
}
it's very easy to see that id is not changed. Or here:
public String getPath() {
final StringBuilder path = new StringBuilder(/);
return Joiner.on('/').appendTo(path, nameQueue).toString();
}
Or here:
while (true) {
final int expectedByte = expected.read();
final int actualByte = actual.read();
assertEquals(expectedByte, actualByte);
if (expectedByte == -1) {
break;
}
}
(I think it¹s some kind of code smell to reassign parameter anyway).
There is a Checkstyle feature to ensure parameters are not reassigned. I'm
fine using it, even if it means we need to change some code.
That¹s why I vote for the (b).
Even thought I prefer (a), I can live with (b), but I would very much
prefer using final only very sparsely.
Regards,
Thomas
Re: Using of final for variables and parameters
Hi Tomek,
On Fri, 2015-08-28 at 08:30 +, Tomek Rekawek wrote:
Hello,
I’m trying to avoid re-assigning local variables whenever it’s
possible. For me it makes the code easier to reason about - I know
that the variable “state created at the beginning of the method is
the same one I can access at the end. If I need the child state at
some point I can create the new variable. The same applies for the
parameters (I think it’s some kind of code smell to reassign
parameter anyway).
I partly agree with your statement about ease of reasoning, but I would
not assign that much weight to the gains, compared to the effort.
Note that final variables / parameters guarantee that the variable's
_identity_ does not change, not the variable's _state_. The concepts
overlap for primitives and immutable objects but not for mutable
objects. In fact, I'd argue that a larger class of errors stems from
changing the state received parameters, like collections, rather than
changing the identity of a local parameter.
As for changing parameters, I think it boils down to code style
preference. I for one prefer to change local parameters for instance
when 'cleaning' passed in arguments. A couple of straw-man examples
follow:
public static boolean hasEmpty(Collection? coll) {
if ( coll == null )
coll = Collections.emptyList();
return coll.contains();
}
public static boolean isParent(String maybeParent, String
maybeChild) {
if ( maybeParent == null )
maybeParent = /;
else if ( maybeParent.endsWith(/) !maybeParent.isEmpty() )
maybeParent = maybeParent.substring(0, maybeParent.length()
- 1);
// TODO actual implementation
}
I am sure that there are elegant ways of doing while not changing the
parameter's identity, but for me the above idiom is the right balance
between conciseness and understandability.
Cheers,
Robert
Re: Using of final for variables and parameters
Hello, I’m trying to avoid re-assigning local variables whenever it’s possible. For me it makes the code easier to reason about - I know that the variable “state created at the beginning of the method is the same one I can access at the end. If I need the child state at some point I can create the new variable. The same applies for the parameters (I think it’s some kind of code smell to reassign parameter anyway). For this reason I put the dreadful “final” keyword before each variable declaration. It allows me to put some of the responsibility on the compiler. I don’t add the “final” to the parameter names, cause I like the method signatures to fit in one line. I also understand that this is subjective matter and I’m sure there is a lot of better programmers than me that don’t need finals. That’s why I vote for the (b). Best regards, Tomek On 28/08/15 08:36, Thomas Mueller muel...@adobe.com wrote: Hi, I wonder what does the team think about using final for variables and parameters. In Oak, so far we didn't use it a lot. This question has come up with OAK-3148. The patch uses final for variables, but not for parameters. Lately, I have seens some code (I forgot where) that uses final for parameters in an _interface_. Please note this is not about using final for fields, where I think everybody agrees it should be used. It's just about variables and parameters. I think we have 3 options: (a) use final for variables and parameters everywhere possible (b) let the individual developer decide (c) don't use it except if needed Some links: http://stackoverflow.com/questions/154314/when-should-one-use-final-for-method-parameters-and-local-variables http://stackoverflow.com/questions/137868/using-final-modifier-whenever-applicable-in-java Personally, my favorite is (c), followed by (b). As for (a), I think (same as Alex Miller at StackOverflow) it clutters the code and decreases readability. Too bad final is not the default in Java, but Java will not change, and we are stuck with Java. I think using final will not improve the code, because people don't accidentally change variables and parameters, so it will not help the writer of a method, it will not help the compiler or performance (the JVM can easily see if a variable or parameter is effectively assigned only once). To improve the code, I'm all for using Checkstyle, unit tests, code coverage, mutation testing, enforcing to write Javadocs for interfaces, and so on. But using final wherever possible, I think it would be a step backwards. Regards, Thomas
Re: [Oak origin/trunk] Apache Jackrabbit Oak matrix - Build # 366 - Still Failing
java.lang.OutOfMemoryError: PermGen space [1]. I'm seeing these quite often lately. This looks like Maven itself runs out of PermGen space. Does anyone know how to configure this for Jenkins? Michael [1] Exception in thread main java.lang.OutOfMemoryError: PermGen space at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClassFromSelf(ClassRealm.java:389) at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:42) at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:259) at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:242) at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:227) at org.eclipse.sisu.inject.MildElements.iterator(MildElements.java:74) at org.eclipse.sisu.inject.RankedBindings.remove(RankedBindings.java:70) at org.eclipse.sisu.inject.InjectorPublisher.unsubscribe(InjectorPublisher.java:81) at org.eclipse.sisu.inject.RankedBindings.remove(RankedBindings.java:123) at org.eclipse.sisu.inject.DefaultBeanLocator.remove(DefaultBeanLocator.java:105) at org.eclipse.sisu.inject.DefaultBeanLocator.clear(DefaultBeanLocator.java:118) at org.codehaus.plexus.DefaultPlexusContainer.dispose(DefaultPlexusContainer.java:588) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:240) at org.apache.maven.cli.MavenCli.main(MavenCli.java:157) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) On 27.8.15 3:27 , Apache Jenkins Server wrote: The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build #366) Status: Still Failing Check console output at https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/366/ to view the results. Changes: [stefanegli] OAK-3292 : using explicit, logged, configurable seed for testLargeStartStopFiesta so that error cases can easily be reproduced [mreutegg] OAK-3079: LastRevRecoveryAgent can update _lastRev of children but not the root Test results: All tests passed
Re: Using of final for variables and parameters
On 28.8.15 8:36 , Thomas Mueller wrote: Personally, my favorite is (c), followed by (b). As for (a), (a): -1 (b): +99 (c): +100 I think (same as Alex Miller at StackOverflow) it clutters the code and decreases readability. Too bad final is not the default in Java +10^10 Michael
Re: Detecting unclosed NodeStore instances
Hi, I'm a bit reluctant about finalizers as it is all too easy to introduce subtle problems causing instance to be resurrected. IMO a better approach would be to use phantom references, avoiding such problems. Michael On 27.8.15 2:48 , Robert Munteanu wrote: Hi, The oak test suite often fails for me due to a large number of running threads. These are caused by NodeStore instances which are opened but not disposed. My suggestion would be to override the finalize method and log a warning when a NodeStore instance is GCed but not disposed, which is most likely a programming error. Since the usual scenario ( outside of testing ) is to have 1 or ( maybe ) just a few NodeStore instances per repository, the impact on memory should be minimal. For the record, with a patched DocumentNodeStore I get 56 unclosed repository instances in oak-core. If this is agreeable to everyone I'll create a Jira issue and submit a patch. Cheers, Robert
Re: SegmentStore: MultiStore Compaction
Hi Thomas The idea is most welcome. Just had an OOME with 4G heap on a local dev instance yesterday due to compaction (running Oak 1.2.4). Which API do you want to use for copying? The NodeStore API? Note that for a complete copy you need to also consider copying checkpoints. The idea is very similar to what's happening in oak-upgrade. From my experiments in that area, I think this approach could result in a reasonably fast compaction. For a simplistic approach (and I may just be summarizing your idea, not quite sure): - the NodeStore could be wrapped with a CompactingNodeStore - CompactingNodeStore delegates to store 1 and store 2; a bloom filter recording copied paths optimizes correct delegation; false positives check both store 2 then store 1 - once copy is complete, store 1 can be shut down - repeat when necessary Challenges may be in the caching layers, where references to store 1 may be hard to get rid of? Regards Julian On Fri, Aug 28, 2015 at 10:05 AM, Thomas Mueller muel...@adobe.com wrote: Hi, I thought about SegmentStore compaction and made a few slides: http://www.slideshare.net/ThomasMueller12/multi-store-compaction Feedback is welcome! The idea is at quite an early stage, so if you don't understand or agree with some items, I'm to blame. Regards, Thomas
Re: Using of final for variables and parameters
cba Am 28. August 2015 02:36:27 GMT-04:00, schrieb Thomas Mueller muel...@adobe.com: Hi, I wonder what does the team think about using final for variables and parameters. In Oak, so far we didn't use it a lot. This question has come up with OAK-3148. The patch uses final for variables, but not for parameters. Lately, I have seens some code (I forgot where) that uses final for parameters in an _interface_. Please note this is not about using final for fields, where I think everybody agrees it should be used. It's just about variables and parameters. I think we have 3 options: (a) use final for variables and parameters everywhere possible (b) let the individual developer decide (c) don't use it except if needed Some links: http://stackoverflow.com/questions/154314/when-should-one-use-final-for-method-parameters-and-local-variables http://stackoverflow.com/questions/137868/using-final-modifier-whenever-applicable-in-java Personally, my favorite is (c), followed by (b). As for (a), I think (same as Alex Miller at StackOverflow) it clutters the code and decreases readability. Too bad final is not the default in Java, but Java will not change, and we are stuck with Java. I think using final will not improve the code, because people don't accidentally change variables and parameters, so it will not help the writer of a method, it will not help the compiler or performance (the JVM can easily see if a variable or parameter is effectively assigned only once). To improve the code, I'm all for using Checkstyle, unit tests, code coverage, mutation testing, enforcing to write Javadocs for interfaces, and so on. But using final wherever possible, I think it would be a step backwards. Regards, Thomas
SegmentStore: MultiStore Compaction
Hi, I thought about SegmentStore compaction and made a few slides: http://www.slideshare.net/ThomasMueller12/multi-store-compaction Feedback is welcome! The idea is at quite an early stage, so if you don't understand or agree with some items, I'm to blame. Regards, Thomas
Re: SegmentStore: MultiStore Compaction
Hi Julian, I did some testing and came to the conclusion for AEM 6.1 (with Oak 1.2.2) to run properly ,with automated compaction enabled, it requires a minimum of around 4.5 GB in heap. On Fri, Aug 28, 2015 at 1:14 PM, Julian Sedding jsedd...@gmail.com wrote: Hi Thomas The idea is most welcome. Just had an OOME with 4G heap on a local dev instance yesterday due to compaction (running Oak 1.2.4). Which API do you want to use for copying? The NodeStore API? Note that for a complete copy you need to also consider copying checkpoints. The idea is very similar to what's happening in oak-upgrade. From my experiments in that area, I think this approach could result in a reasonably fast compaction. For a simplistic approach (and I may just be summarizing your idea, not quite sure): - the NodeStore could be wrapped with a CompactingNodeStore - CompactingNodeStore delegates to store 1 and store 2; a bloom filter recording copied paths optimizes correct delegation; false positives check both store 2 then store 1 - once copy is complete, store 1 can be shut down - repeat when necessary Challenges may be in the caching layers, where references to store 1 may be hard to get rid of? Regards Julian On Fri, Aug 28, 2015 at 10:05 AM, Thomas Mueller muel...@adobe.com wrote: Hi, I thought about SegmentStore compaction and made a few slides: http://www.slideshare.net/ThomasMueller12/multi-store-compaction Feedback is welcome! The idea is at quite an early stage, so if you don't understand or agree with some items, I'm to blame. Regards, Thomas -- With kind regards Mit freundlichen Grüßen *Stephan Becker* | Senior System Engineer Netcentric Deutschland GmbH M D: +49 (0) 175 2238120 Skype: stephanhs.b stephan.bec...@netcentric.biz | www.netcentric.biz Other disclosures according to §35a GmbhG, §161, 125a HGB: www.netcentric.biz/imprint.html
Re: SegmentStore: MultiStore Compaction
Hi, AFAIK your tests re. restart have been done from within an OSGi container (AEM) restarting the repository bundle. Actually I restarted the JVM. But I probably didn't use the very latest version of Oak, checkpoints may also have played a role in my case (there were 2 checkpoints; probably from Lucene indexing). I still have access to that repository (to both the pre-compacted and the compacted version), it should be quite simple to test various ideas. But of course each test takes a long time (an hour or so). Regards, Thomas
Re: Repo Inconsistencies due to OAK-3169
Upon further consideration, here’s an alternative proposal. To recap the problem: Say version 1.0.x has an issue that leads to repo inconsistencies or even (repairable) data loss The issue is fixed in 1.0.y, but users running an inbetween version might have experienced that data loss (maybe without noticing). If we only put the repair code in oak-run then users first need to notice the problem, before even running the repair. Otoh I still think that we should stay clear of complicating the core code. So, alternative proposal: put such repair code in a new module (bundle), say, oak-selfheal. That way we could at least keep core clean from that. In this case oak-jcr could automatically detect the problem and invoke the repair code. However, I am not sure if we have many of these situations where a) the problem can be detected and b) repaired automatically to warrant such a new module. Thoughts? Michael On 24/08/15 10:55, Marcel Reutegger mreut...@adobe.com wrote: Hi, On 24/08/15 10:02, Michael Marth wrote: IMO we should collect such repair actions in oak-run. This should be a one-off action that should not complicate code in oak-core I think. agreed. in addition, the tool could be split into two phases. in the first phase the tool just checks the repository and finds all the dangling references. this phase could also be run on a copy of the data. in a second phase the tool fixes nodes based on the output from the first phase. this reduces impact on the production instance. Regards Marcel
Multiple DocumentNodeStores sharing a single DocumentStore
Hi, I'm looking at LastRevRecoveryAgentTest [1] which uses a shared document store for two DocumentNodeStore instances. Disposing both node store instances makes the tests fail, as the document store will be closed by the first call to dispose() and the second call will fail. The reason is the following snippet in DocumentNodeStore.dispose() ( full context at [2] ) : internalRunBackgroundUpdateOperations(); // other stuff store.dispose(); If we have two node stores being shutdown, one of them will call store.dispose() before the other is able to run its backgroundUpdateOperations(). For the RDB and Mongo stores, the connection will be closed. Also of interest - the LastRecoveryTest[3] , which has a similar setup, only uses a shared MemoryDocumentStore and does not have this problem. I don't think this behaviour exists only in tests, so I'd be interested to see if there's a solution to this. The only idea that I have is the DocumentStore instances knowing how many 'users' it has and only accepting to be disposed when the last 'user' is gone, but that requires that all 'users' obey this contract. Thoughts? Thanks, Robert [1]: https://github.com/apache/jackrabbit-oak/blob/1b6e52242be61c162be4 b1f1280d6545cc489d37/oak -core/src/test/java/org/apache/jackrabbit/oak/plugins/document/LastRevR ecoveryAgentTest.java#L90-L105 [2]: https://github.com/apache/jackrabbit-oak/blob/1b6e52242be61c162be4 b1f1280d6545cc489d37/oak -core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Document NodeStore.java#L551-L601 [3]: https://github.com/apache/jackrabbit-oak/blob/1b6e52242be61c162be4 b1f1280d6545cc489d37/oak -core/src/test/java/org/apache/jackrabbit/oak/plugins/document/LastRevR ecoveryTest.java#L60
Re: [Oak origin/trunk] Apache Jackrabbit Oak matrix - Build # 366 - Still Failing
On Fri, 2015-08-28 at 12:26 +0200, Michael Dürig wrote: java.lang.OutOfMemoryError: PermGen space [1]. I'm seeing these quite often lately. This looks like Maven itself runs out of PermGen space. Does anyone know how to configure this for Jenkins? My first gues would be the 'JVM Options' input in the 'Build' section. If that does not work, I would try setting an environment variable for the build. Can you try enabling 'Prepare an environment for the run' and then filling in 'Properties content' with something like MAVEN_OPTS=-XX:MaxPermSize=256m -Xmx... HTH, Robert
[Oak origin/trunk] Apache Jackrabbit Oak matrix - Build # 368 - Failure
The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build #368) Status: Failure Check console output at https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/368/ to view the results. Changes: Test results: All tests passed
[Oak origin/trunk] Apache Jackrabbit Oak matrix - Build # 369 - Still Failing
The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build #369) Status: Still Failing Check console output at https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/369/ to view the results. Changes: Test results: 1 tests failed. REGRESSION: org.apache.jackrabbit.oak.run.osgi.OakOSGiRepositoryFactoryTest.testRepositoryTar Error Message: null Stack Trace: java.lang.reflect.UndeclaredThrowableException at com.sun.proxy.$Proxy16.login(Unknown Source) at org.apache.jackrabbit.oak.run.osgi.OakOSGiRepositoryFactoryTest.basicCrudTest(OakOSGiRepositoryFactoryTest.java:131) at org.apache.jackrabbit.oak.run.osgi.OakOSGiRepositoryFactoryTest.testRepositoryTar(OakOSGiRepositoryFactoryTest.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46) at org.junit.rules.RunRules.evaluate(RunRules.java:18) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.jackrabbit.oak.run.osgi.OakOSGiRepositoryFactory$RepositoryProxy.invoke(OakOSGiRepositoryFactory.java:458) ... 35 more Caused by: javax.jcr.LoginException: Login Failure: all modules ignored at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(RepositoryImpl.java:287) at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(RepositoryImpl.java:243) ... 40 more Caused by: javax.security.auth.login.LoginException: Login Failure: all modules ignored at javax.security.auth.login.LoginContext.invoke(LoginContext.java:921) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:706) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:703) at javax.security.auth.login.LoginContext.login(LoginContext.java:575) at
Re: SegmentStore: MultiStore Compaction
AFAIU this is pretty much what we are now doing under the hood. That is, your proposal would make the compaction step more explicit and visible above the node store API. An advantage of your approach is preventing mix segments all together (i.e. compacted segments still referring to uncompacted ones). This is something we were having problems with in the past, which I however believe we have solved by now. However, you approach most likely suffers from the same problems we currently have re. contention, performance, in memory references, ... Michael On 28.8.15 10:05 , Thomas Mueller wrote: Hi, I thought about SegmentStore compaction and made a few slides: http://www.slideshare.net/ThomasMueller12/multi-store-compaction Feedback is welcome! The idea is at quite an early stage, so if you don't understand or agree with some items, I'm to blame. Regards, Thomas
Re: SegmentStore: MultiStore Compaction
Hi, I'm not an expert on compaction, but the differences the current approach I see are: * No compaction maps. No memory problem. No persistent compaction map. As far as I understand, currently you can have _multiple_ compaction map at the same time. I think that persisting the compaction map is problematic for code complexity and performance reasons, specially with large repositories. As for performance, it depends a lot on how the compaction maps are stored (randomized access patterns will hurt performance a lot). * Simpler architecture: Multi-store compaction is implemented on top of the current SegmentStore, which makes it more modular and easier to (unit-) test. The current compaction code, on the other hand, is more interwoven with the regular SegmentStore code. you approach most likely suffers from the same problems we currently have re. contention, performance, in memory references, ... Contention and performance, what are the problems right now? Memory references: with a restart, old memory references are gone, so the old segment store can be removed fully, without risk. Right now, at least with the version of Oak I have tested, 1.2.3 I think, running online compaction multiple times, each time with a restart, did not shrink the repository (size is 3 times the size of a fully compacted repo, with very little writes). Without restart, access to very old objects can result in a easy to understand exception message. Regards, Thomas On 28/08/15 13:57, Michael Dürig mdue...@apache.org wrote: AFAIU this is pretty much what we are now doing under the hood. That is, your proposal would make the compaction step more explicit and visible above the node store API. An advantage of your approach is preventing mix segments all together (i.e. compacted segments still referring to uncompacted ones). This is something we were having problems with in the past, which I however believe we have solved by now. However, you approach most likely suffers from the same problems we currently have re. contention, performance, in memory references, ... Michael On 28.8.15 10:05 , Thomas Mueller wrote: Hi, I thought about SegmentStore compaction and made a few slides: http://www.slideshare.net/ThomasMueller12/multi-store-compaction Feedback is welcome! The idea is at quite an early stage, so if you don't understand or agree with some items, I'm to blame. Regards, Thomas
Re: SegmentStore: MultiStore Compaction
On 28.8.15 2:41 , Thomas Mueller wrote: Hi, I'm not an expert on compaction, but the differences the current approach I see are: * No compaction maps. No memory problem. No persistent compaction map. As far as I understand, currently you can have _multiple_ compaction map at the same time. I think that persisting the compaction map is problematic for code complexity and performance reasons, specially with large repositories. As for performance, it depends a lot on how the compaction maps are stored (randomized access patterns will hurt performance a lot). Good point. This circles back to what Chetan had in mind originally. I.e. doing the mapping via the path. Only that with you approach we would do it on top of the NodeStore as opposed to within the NodeStore. Memory references: with a restart, old memory references are gone, so the old segment store can be removed fully, without risk. Right now, at least with the version of Oak I have tested, 1.2.3 I think, running online compaction multiple times, each time with a restart, did not shrink the repository (size is 3 times the size of a fully compacted repo, with very little writes). Without restart, access to very old objects can result in a easy to understand exception message. AFAIK your tests re. restart have been done from within an OSGi container (AEM) restarting the repository bundle. I could easily imagine this not being sufficient to get rid of all the in memory references. There is likely tons of stuff still lurking around even after a bundle restart. You approach would most likely suffer from the same problem. Overall it seems like a promising approach though. As you pointed out it would be more modular and would help us getting rid of the compaction map. So I'm all +1 for trying it out. Michael Regards, Thomas On 28/08/15 13:57, Michael Dürig mdue...@apache.org wrote: AFAIU this is pretty much what we are now doing under the hood. That is, your proposal would make the compaction step more explicit and visible above the node store API. An advantage of your approach is preventing mix segments all together (i.e. compacted segments still referring to uncompacted ones). This is something we were having problems with in the past, which I however believe we have solved by now. However, you approach most likely suffers from the same problems we currently have re. contention, performance, in memory references, ... Michael On 28.8.15 10:05 , Thomas Mueller wrote: Hi, I thought about SegmentStore compaction and made a few slides: http://www.slideshare.net/ThomasMueller12/multi-store-compaction Feedback is welcome! The idea is at quite an early stage, so if you don't understand or agree with some items, I'm to blame. Regards, Thomas
Re: Is the hashing of long paths still needed?
Hi, On Fri, 2015-08-21 at 14:07 +, Thomas Mueller wrote: Hi, The DocumentStore doesn't really know the path, it only knows the key, and if the key is hashed you can't calculate the path. There are some options: (a) Each document that has a hashed path as the key also has a path property (with the real path). You could use that (cache it, read it if needed, possibly from all backends). That's mostly the approach that we have taken for now. I don't think a (slight) performance penalty for hashed documents is a huge issue in the prototype stage, so if I can't extract the path out of the key I simply ask all the stores ( details at [1] ). This seems to work out fine so far. Thanks, Robert [1]: https://github.com/rombert/jackrabbit-oak/blob/f05398694fa3bb7b2fb 7feb96e16efc2d97a7e84/oak -core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Multiple xingDocumentStore.java#L32-L62
[Oak origin/trunk] Apache Jackrabbit Oak matrix - Build # 370 - Still Failing
The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build #370) Status: Still Failing Check console output at https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/370/ to view the results. Changes: [alexparvulescu] OAK-3294 Read-only live FileStore implementation [alexparvulescu] OAK-3301 AbstractRepositoryUpgrade leaks Repository instances Test results: 8 tests failed. REGRESSION: org.apache.jackrabbit.oak.security.authentication.ldap.LdapProviderTest.testAuthenticate Error Message: ERR_171 Failed to bind an LDAP service (1,024) to the service registry. Stack Trace: org.apache.directory.api.ldap.model.exception.LdapConfigurationException: ERR_171 Failed to bind an LDAP service (1,024) to the service registry. at org.apache.directory.server.ldap.LdapServer.startNetwork(LdapServer.java:695) at org.apache.directory.server.ldap.LdapServer.start(LdapServer.java:547) at org.apache.jackrabbit.oak.security.authentication.ldap.AbstractServer.setUp(AbstractServer.java:232) at org.apache.jackrabbit.oak.security.authentication.ldap.InternalLdapServer.setUp(InternalLdapServer.java:31) at org.apache.jackrabbit.oak.security.authentication.ldap.LdapProviderTest.before(LdapProviderTest.java:82) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:414) at sun.nio.ch.Net.bind(Net.java:406) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) at org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) at org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) at
Multiplexing and the journal collection
Hi, My current multiplexing prototype only handles multiplexing for the NODES collection. However, upon a review it seems to me that the JOURNAL collection is quite closely tied to the NODES collection. By reading the code it looks to me like the only use it has is the cache invalidation when changes happen between cluster nodes. Is that correct? If it is I will be able to delay looking into the Journal consistency a bit, but if it can lead to hard-to-debug errors later I'd rather look into it now. Thanks, Robert
jackrabbit-oak build #6269: Fixed
Build Update for apache/jackrabbit-oak - Build: #6269 Status: Fixed Duration: 1952 seconds Commit: 44edcca310d3b2956a4161da776efdb14326c631 (trunk) Author: Alexandru Parvulescu Message: OAK-3301 AbstractRepositoryUpgrade leaks Repository instances - patch provided by Robert Munteanu git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/oak/trunk@1698326 13f79535-47bb-0310-9956-ffa450edef68 View the changeset: https://github.com/apache/jackrabbit-oak/compare/53574af21bae...44edcca310d3 View the full build log and details: https://travis-ci.org/apache/jackrabbit-oak/builds/77683156 -- sent by Jukka's Travis notification gateway
