Curiouser and curiouser... After one of my changes introduced weird formatting errors to the webpage I rolled back an earlier version. On inspection I saw it was reporting even more records than before (differences of 123, 70, 0 from what I was expecting). This was great news. But an hour later when I checked again the differences had increased to the figures I noted before! What happened? None of the dspace cron scripts were due to run for >12hrs so I doubt they were involved.
I've hunted for records which are missing on the new system to see if there's an obvious reason why they were dropped. We did find a difference in the number of fields in one of the tables but I don't know if that's even related to this issue. What exactly is the difference between the old database format and the new? We haven't introduced any modifications that I'm aware of so there must be another reason migration is partially failing for us. . On Tuesday, March 13, 2018 at 5:09:51 PM UTC-8, wlruth...@alaska.edu wrote: > > Yes, I checked browse by author, title, and subject. All were off a bit: > by 288, 182, and 11 records respectively. But I hadn't > applied custom themes and *perhaps* some of the configuration changes. > I'm sure the customized themes can wait but I'll > double-check that all of the configuration settings are being set > correctly. I also hadn't noticed any errors in the logs but I > did skim them pretty quickly to verify I was seeing (thousands of) > re-ingest notices. I'll go through the logs with a finer comb. > > Thanks for the possible gotchas. > > > On Tuesday, March 13, 2018 at 10:24:06 AM UTC-8, Tim Donohue wrote: >> >> Hi, >> >> Have you checked your DSpace logs to see if there were any errors >> reported there, either from the 'dspace database migrate', or from the >> full site reindex (over the weekend)? That'd be the first place I'd >> recommend checking if there are missing items. >> >> More on checking logs for errors at: >> https://wiki.duraspace.org/display/DSPACE/Troubleshoot+an+error >> >> Also, I'd recommend comparing the number of items listed in "Browse by >> Title" between the two versions (DSpace 3.6 and 5.5). That will give you >> better information on whether all the Items have been updated/upgraded >> successfully. It sounds like you are comparing the numbers in "Browse by >> Author"...and while those should pretty much match up, it is possible that >> DSpace 5 has reindexed your author information in such a way that it >> removed some duplicates that may have existed in DSpace 3. >> >> You also may wish to verify that any configurations from your old DSpace >> (version 3.6) have been brought forward into DSpace 5.5. So, for example, >> if you customized Browse by Author settings (in dspace.cfg) to also use >> extra fields (not just "dc.contributor.author"), then you should verify >> that DSpace 5 is using those same fields (if it is not, then its author >> count would be lower). >> >> That's just a couple of ideas that come to mind. Hopefully one of them >> will help you start to figure out what may be going on. As always, let us >> know if you have further questions or figure out more of what may be >> happening. >> >> Good luck, >> >> Tim >> >> On Tue, Mar 13, 2018 at 12:37 PM <wlruth...@alaska.edu> wrote: >> >>> I gave it the weekend to be sure it was done then manually ran 'dspace >>> database migrate' again. If returned much quicker but nothing >>> had changed. Of the 5105 author records I expected it found 4817. I >>> guess that's not bad. If I knew a way to tell what it's not finding >>> it might not be too much work to just reentering them by hand. But so >>> far I haven't turned up what is different between the two copies. >>> >>> >>> On Thursday, March 8, 2018 at 9:19:00 AM UTC-9, Tim Donohue wrote: >>> >>>> Hello, >>>> >>>> While I'm not certain this is the case, it *sounds* like the behavior >>>> you are seeing is related to a full reindex of the site. >>>> >>>> After any database upgrade (via `dspace database migrate`), DSpace will >>>> trigger a full reindex of your site the next time you start up Tomcat (or >>>> whatever servlet container you are using). This full reindex may take some >>>> time (minutes or even hours, depending on the amount of content), but it >>>> runs in the backend. A reindex is required for major upgrades, as often >>>> the >>>> database structure changes, and your search/browse indexes then need to be >>>> updated based on those changes. >>>> >>>> You may wish to check your DSpace log files to see if you are seeing >>>> messages related to individual Items being "indexed". If so, this is the >>>> reindex processing each file, and once it is complete all your records >>>> should reappear in the search/browse. While it is reindexing you still >>>> can >>>> use the site, but not all records may appear (they each will appear as >>>> they >>>> are reindexed). >>>> >>>> And just to clarify, you no longer need to upgrade from one major >>>> version to the next. So, you should not need to upgrade from DSpace 3 to >>>> 5 >>>> to 6. You can now *directly* upgrade from any prior release to DSpace 6 >>>> (even from DSpace 1.0 -> 6.0 would work). The only requirement is that >>>> you >>>> need to make sure your database backend is upgraded first, to a compatible >>>> release. So, for example, if you are running a very old version of >>>> Postgres or Oracle, you'd want to upgrade that database backend to a >>>> recent >>>> version (as required by DSpace 6), and then you should be able to upgrade >>>> directly to DSpace 6 per the upgrade instructions at: >>>> https://wiki.duraspace.org/display/DSDOC6x/Upgrading+DSpace >>>> >>>> Hopefully that helps! If you have further questions, feel free to ask >>>> them on this list. >>>> >>>> Tim >>>> >>>> On Thu, Mar 8, 2018 at 12:04 PM <wlruth...@alaska.edu> wrote: >>>> >>> Interesting... It looks like the number of records has updated itself >>>>> since yesterday. >>>>> Perhaps the automatic database migration is working, just slowly. I'll >>>>> watch it. >>>>> >>>>> >>>>> On Wednesday, March 7, 2018 at 3:17:03 PM UTC-9, wlruth...@alaska.edu >>>>> wrote: >>>>>> >>>>>> >>>>>> While upgrading our repository we initially built DSpace-6.0 but it >>>>>> was unable to fully ingest >>>>>> the legacy postgres database used by the older DSpace-3.1 system. We >>>>>> were told we'd need >>>>>> to first install DSpace-3.x, upgrade that to DSpace-5.x, then upgrade >>>>>> to DSpace-6.0. >>>>>> >>>>>> So we built a DSpace-3.6 system and it was able to read the >>>>>> DSpace-3.1 database. We then >>>>>> built a DSpace-5.5 system with a copy of the entire database. All >>>>>> seemed well at first but on >>>>>> closer inspection it's doing the same thing as our original >>>>>> DSpace-6.0 system - ignoring most >>>>>> of the database. The manual says DSpace will automatically fix the >>>>>> database or that the >>>>>> 'dspace database migrate' command will accomplish this. Neither seems >>>>>> to have had any >>>>>> effect. I'm scanning through logfiles to see if it silently told me >>>>>> that something went wrong. >>>>>> >>>>>> I know other DSpace users have successfully upgraded from DSpace-3.x >>>>>> to DSpace-6.0 so >>>>>> that's encouraging. Am I missing a step? Do I have install DSpace-4.x >>>>>> before DSpace-5.x? >>>>>> >>>>>> Thank you for any help you can provide. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "DSpace Technical Support" group. >>>>> >>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to dspace-tech...@googlegroups.com. >>>>> To post to this group, send email to dspac...@googlegroups.com. >>>> >>>> >>>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>>> Tim Donohue >>>> Technical Lead for DSpace & DSpaceDirect >>>> DuraSpace.org | DSpace.org | DSpaceDirect.org >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "DSpace Technical Support" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to dspace-tech...@googlegroups.com. >>> To post to this group, send email to dspac...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/dspace-tech. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> Tim Donohue >> Technical Lead for DSpace & DSpaceDirect >> DuraSpace.org | DSpace.org | DSpaceDirect.org >> > -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To post to this group, send email to dspace-tech@googlegroups.com. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
