Christine and other AS community members who have successfully migrated
AS 1.4.2 to AS 1.5.x,
We are trying to draw attention and seek help from AS development team
and other community members who have successfully migrated AS 1.4.2 to
AS 1.5.x. If necessary, we are willing to provide our MySQL dump file
for you to test.
Test migration from AS 1.4.2 to AS1.5.x really drives us mad. Though we
don't quite see any (out of mem) error in the log, we still blindly pump
up heap size (Xmx )to 6GB, stack size set o 3MB and rerun rebuild index
or the complete migration steps, the problem is still there - Solr
seemingly very silently stops writing or committing index files when the
number of docs in Solr reaches to about 15,xxx as opposed to 500,xxx
indexed docs found in our 1.4.2 instance, though thereafter we still
can see the " end_commit_flush" message pop up in the log from time to
time .Our test box environments are
1. Redhat 6.8 ,JAVA build 1.7.0_67 . MySQL 5.1.7 and 8GB memory
2.. Redhat 7.2 Virtual Machine,NYSQL 5.5.45, java 1.80_65,and 4GB.
We also can confirm database side migration is largely done with new
tables and fields added, existing records converted and top container
records added. The problem is sticking to rebuilding the whole index -
Solr stops writing and committing index prematurely after the number
of docs grows to a small number(12xxx-15xxx) though commit flush can
still be seen thereafter .
Now we have exhausted ideas and means in figuring out why this "no
record found " problem that seemingly only happens to us.
Other experiment we did is export EAD file of one resource record from
our production and load it into a 1.4.2 test instance and then upgrade
this test instance to 1.5.x. this time the resource record can be
indexed and appear(some issue with the Name ) in the test 1.5.x instance.
I think there is some fundamental but very sneaky issue with 1.5.x
indexing (rebuild index). The challenge is that no explicit error
message that could be easily found in the log that can give us some hint
and therefore this is a nightmare given the long waiting time before you
can confirm a successful indexing or not. May be a standalone tool
allowing user to manually rebuild index could be very helpful and
straightforward, and less arcane.
As mentioned earlier, records have been converted and migrated but not
indexed, what else we tried is use direct route path (can be inferred
from our production) to pull records out and update them by toggling
through Suppress/Unsuppress or Publisesh/Unpublished , then we can get
those updated records indexed. in the following diagram, we can see 3
resources, 2 agent records and some other record types highlighted in
yellow appear in the types field, as opposed to the types (not
highlighted) originally indexed and generated by the migration process.
Also in the same diagram, the number of indexed docs is too small (we
can see about half million docs in our 1.4.2 production instance).Also
we found a side issue with agent record, no matter what you set for
Publish, Solr only sees it unpublished.But At least, this experiment
can confirm there is not obvious problem with dynamic indexing.
If AS developers or other AS users want to reproduce the issue or give a
try for a solution, we can provide our v1.4.2 database dump or the
v1.5.x database dump.
Thank you very much.
Mang Sun
Rice U.
On 9/9/2016 11:55 AM, Christine Di Bella wrote:
Mang,
How long ago did you start the reindex and how big is your database?
The v1.5.1 upgrade improved indexing speed, especially for large
databases, but the indexing does still take some time. People on this
list can probably give you a sense of how long it took them for
different sizes of databases. It’s probably been most common to run it
overnight or over a weekend, when use of the system is low.
(Some people were large databases were experiencing problems with
indexing taking an excessive amount of time or not completing at all,
which was one of the reasons for the quick release of v1.5.1.)
Christine
Christine Di Bella
Community Outreach Manager
[email protected] <mailto:[email protected]>
800.999.8558 x2905
678-235-2905
cdibella13 (Skype)
cid:[email protected]
*From:*[email protected]
[mailto:[email protected]] *On
Behalf Of *Mang Sun
*Sent:* Friday, September 9, 2016 12:31 PM
*To:* Archivesspace Users Group
<[email protected]>
*Subject:* Re: [Archivesspace_Users_Group] No record found after test
migration from v1.4.2 to 1.5.0
I even didn't copy the data folder of the v1.4.2 into v.1.5.0. but
even after I follow the recommended way by copying the data folder and
removing those two subfolders mentioned by Joshua the problem is still
there.Peeping into Solr schema, I found field "types" only have the
following four types but not resource and digital_object,subject and
agent . The missing types may explain why we find no records under
every record type tab. Puzzled~ Mang
·12608
otop_container
<http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22top_container%22>
·1,169
otree_view
<http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22tree_view%22>
·75
oarchival_object
<http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22archival_object%22>
·1
orepository
<http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22repository%22>
On 9/9/2016 8:52 AM, Sally Vermaaten wrote:
Hi Mang Sun,
I agree with Joshua - this sounds to me like an indexing problem
to me. In the course of upgrading to 1.5.0 did you delete the
data/solr_index/index directory and all files in the
data/indexer_state directory? This is one of the recommended steps
in the instructions for upgrading to v.1.5.0
<https://github.com/archivesspace/archivesspace/blob/master/UPGRADING_1.5.0.md>.
We accidentally missed this step once in a running a test upgrade
and definitely experienced some indexing problems though I don't
remember the details. As it was a test upgrade (and it sounds like
that's what you're doing as well) we just started afresh and ran
the upgrade process over again. When we included that step, things
were fine.
On a related note - there's a 1.5.1 release of ArchivesSpace
<https://github.com/archivesspace/archivesspace/releases/tag/v1.5.1> that
includes several fixes for bugs that were found in the weeks after
1.5.0 was released. If are looking to update to a 1.5.x version,
you may want to consider upgrading to 1.5.1 one rather than 1.5.0.
Good luck,
Sally
On Fri, Sep 9, 2016 at 9:50 AM, Joshua D. Shaw
<[email protected] <mailto:[email protected]>>
wrote:
I believe the suggestion for migration to 1.5+ was to delete
the existing index and indexer states which will trigger a
complete reindex.
Best,
Joshua
On Sep 9, 2016, at 8:20 AM, "Mang Sun" <[email protected]
<mailto:[email protected]>> wrote:
> i
>
> After we migrate our v.1.4.2 test instance to v1.5.0, we
can't find any records but just no record in Staff interface
and public interface.
>
> But if we open a migrated record (saycollection record)
using the direct route path to that record and resave it, this
record but only this one starts to show up under Collections
when browsing collections. This problem does not come with any
new record created within v.1.5.0. Any suggestion?
>
> Thank you.
>
> Mang Sun
>
> Rice U.
>
>
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> [email protected]
<mailto:[email protected]>
>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
_______________________________________________
Archivesspace_Users_Group mailing list
[email protected]
<mailto:[email protected]>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
--
Sally Vermaaten
Project Manager, Archival Systems
New York University Libraries
1-212-992-6259
_______________________________________________
Archivesspace_Users_Group mailing list
[email protected]
<mailto:[email protected]>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
_______________________________________________
Archivesspace_Users_Group mailing list
[email protected]
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
_______________________________________________
Archivesspace_Users_Group mailing list
[email protected]
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
