Adding new cluster node when old journals were already removed by janitor thread
Hi, All In page https://wiki.apache.org/jackrabbit/Clustering, there is several caveats if the janitor thread is enabled The current solution has three known caveats: If the janitor is enabled then you loose the possibility to easily add cluster nodes. (It is still possible but takes detailed knowledge of Jackrabbit.) From the statement, it look we could not directly adding one new node into the cluster environment if some old journals has already been deleted by janitor thread.But based on my current understanding, during the startup,this new node already does following 2 things sequentially: Reading the whole repository, including all nodes and the version history, and built index for them Read all revisions to sync You see, for new added node, since step#1 already reads all nodes from database, even some revisions are already removed, step#2 seems has no problem. If step#2 does have problem, or there is any other potential issue. Please let me know or give me one example. Regards, -Liang
About Jackrabbit cluster
Hi, all In our cluster environment, the Journal is very huge (about 60g), so we'd like to enalbing janitor configuration into repository.xml. As the wiki page said, there are 3 caveats after enabling the janitor. I can understand the first and third caveat, but not quite catch the second one as below. What does it mean? Does it mean we need do some extra thing (coding, configuration??) to make sure enabling janitor can work well? You must make sure that all cluster nodes have written their local revision to the database before the clean-up task runs for the first time because otherwise cluster nodes might miss updates (because they have been purged) and their local caches and search-indexes get out of sync. What I expected is that we only need to add janitorEnabled=true, janitorSleep and janitorFirstRunHourOfDay under Cluster element, and then the journal table could be purged as expected and no extra work that needs us to do Please help me! Thanks. Regards, -Liang
[jira] [Commented] (JCR-3981) Null pointer exception on journal synchronization
[
https://issues.apache.org/jira/browse/JCR-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472430#comment-15472430
]
liang cheng commented on JCR-3981:
--
one of our customer also hit this issue. It's really urgent for us
> Null pointer exception on journal synchronization
> -
>
> Key: JCR-3981
> URL: https://issues.apache.org/jira/browse/JCR-3981
> Project: Jackrabbit Content Repository
> Issue Type: Bug
> Components: jackrabbit-core
>Affects Versions: 2.11.3
>Reporter: Illia Khokholkov
>
> Using Jackrabbit 2.x, I am experiencing an issue that is similar if not
> identical to the JCR-773, which was "remedied" a long time. The stack trace
> of an exception looks like this:
> {noformat}
> java.lang.NullPointerException: null
> at java.io.FilterInputStream.close(FilterInputStream.java:181)
> at
> org.apache.jackrabbit.core.journal.ReadRecord.close(ReadRecord.java:212)
> at
> org.apache.jackrabbit.core.journal.DatabaseRecordIterator.close(DatabaseRecordIterator.java:155)
> at
> org.apache.jackrabbit.core.journal.DatabaseRecordIterator.close(DatabaseRecordIterator.java:121)
> at
> org.apache.jackrabbit.core.journal.AbstractJournal.doSync(AbstractJournal.java:263)
> at
> org.apache.jackrabbit.core.journal.DatabaseJournal.doSync(DatabaseJournal.java:458)
> at
> org.apache.jackrabbit.core.journal.AbstractJournal.internalSync(AbstractJournal.java:222)
> at
> org.apache.jackrabbit.core.journal.AbstractJournal.sync(AbstractJournal.java:190)
> at
> org.apache.jackrabbit.core.cluster.ClusterNode.internalSync(ClusterNode.java:340)
> at
> org.apache.jackrabbit.core.cluster.ClusterNode.syncOnStartup(ClusterNode.java:365)
> at
> org.apache.jackrabbit.core.cluster.ClusterNode.start(ClusterNode.java:277)
> at
> org.apache.jackrabbit.core.RepositoryImpl.(RepositoryImpl.java:344)
> {noformat}
> Unfortunately, I cannot determine the root cause of the problem yet, because
> it appears to be sporadic. However, when it happens, none of the cluster
> members can ever get back to normal.
> *Questions*
> As pointed out before, the JCR-773 is marked as fixed so I assume there
> should be no errors like the ones I see. Anyway, it would be great if I could
> get an answer to the following question:
> * Is this a known problem and if so, how can it be mitigated/eliminated?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: about removing Old Revisions from journal table.
Could someone kindly give me some help? Thanks. Regards, -Liang 2013/5/29 liang cheng lcheng...@gmail.com Hi, all In our production environment, the Jackrabbit Journal table would become large (more than 100, 000 records) after running 2 weeks. As a result, we plan to utilize the janitor thread to remove old revisions mentioned in http://wiki.apache.org/jackrabbit/Clustering#Removing Old Revisions. After enabling it, there would be several caveats as mentioned in the wiki page too. 1. If the janitor is enabled then you loose the possibility to easily add cluster nodes. (It is still possible but takes detailed knowledge of Jackrabbit.) 2. You must make sure that all cluster nodes have written their local revision to the database before the clean-up task runs for the first time because otherwisecluster nodes might miss updates (because they have been purged) and their local caches and search-indexes get out of sync. 3. If a cluster node is removed permanently from the cluster, then its entry in the LOCAL_REVISIONS table should be removed manually. Otherwise, the clean-up thread will not be effective. I can understand point #3.But not quite sure about #1 and #2. #1 is our biggest concern. In our production environment, we have cases to need add new cluster node(s), e.g. If system capacity could not handle current workload, or if some running node needs to be stopped for some while for maintenance and then new node needs to be added. In #1, you only say that you loose the possibility to easily add cluster nodes, but doesn't give more explaination about the reason. As I know, when new node is added into the JR cluster, there is no lucene index, then Jackrabbit would build the index for the whole current repository nodes (build from root node). After this step, Jackrabbit then process the revisions generated by other nodes. *I wonder what's the possible issue when processing old revisions with latest repository content in cache and indexes? * For #2, *does it mean any manual work needed to keep the consistency?* Although the wiki page give one approch to add new cluster node manually (i.e. clone indexes and local revision number from existing node), we still hope there is some safe programming way to avoid the manual work, because our production is deployed in Amazon EC2 environment and adding new node needs easily as much as possible. Could you please give some comments to my concerns? Thanks. Regards, -Liang
about removing Old Revisions from journal table.
Hi, all In our production environment, the Jackrabbit Journal table would become large (more than 100, 000 records) after running 2 weeks. As a result, we plan to utilize the janitor thread to remove old revisions mentioned in http://wiki.apache.org/jackrabbit/Clustering#Removing Old Revisions. After enabling it, there would be several caveats as mentioned in the wiki page too. 1. If the janitor is enabled then you loose the possibility to easily add cluster nodes. (It is still possible but takes detailed knowledge of Jackrabbit.) 2. You must make sure that all cluster nodes have written their local revision to the database before the clean-up task runs for the first time because otherwisecluster nodes might miss updates (because they have been purged) and their local caches and search-indexes get out of sync. 3. If a cluster node is removed permanently from the cluster, then its entry in the LOCAL_REVISIONS table should be removed manually. Otherwise, the clean-up thread will not be effective. I can understand point #3.But not quite sure about #1 and #2. #1 is our biggest concern. In our production environment, we have cases to need add new cluster node(s), e.g. If system capacity could not handle current workload, or if some running node needs to be stopped for some while for maintenance and then new node needs to be added. In #1, you only say that you loose the possibility to easily add cluster nodes, but doesn't give more explaination about the reason. As I know, when new node is added into the JR cluster, there is no lucene index, then Jackrabbit would build the index for the whole current repository nodes (build from root node). After this step, Jackrabbit then process the revisions generated by other nodes. *I wonder what's the possible issue when processing old revisions with latest repository content in cache and indexes? * For #2, *does it mean any manual work needed to keep the consistency?* Although the wiki page give one approch to add new cluster node manually (i.e. clone indexes and local revision number from existing node), we still hope there is some safe programming way to avoid the manual work, because our production is deployed in Amazon EC2 environment and adding new node needs easily as much as possible. Could you please give some comments to my concerns? Thanks. Regards, -Liang
