[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Victor Grousset/tuxayo changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=35659 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Lauren Denny changed: What|Removed |Added CC||lauren_de...@sil.org -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #320 from David Cook --- By the way, the Fedora Commons community is planning to have some conversations about OAI-PMH for Fedora 6.x as well, although they are having their meeting at 10am Eastern Time (in Canada/USA) on March 30th, so that timing might not work out great. I also haven't found them to be responsive online... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #319 from David Cook --- (In reply to Koha Team University Lyon 3 from comment #318) > We would like to use an existing OAI-PMH harvester and 2 of them seem to be > good candidates : > - Catmandu harverster > - HTTP::OAI::HARVESTER module > > Both are in perl. At the moment, we thought that the second could be a > better choice because it's more up to date and we don't necessarily need to > use Catmandu. Currently, we're still stuck using HTTP::OAI 3.27 from 2011 in Koha for OAI-PMH server functionality. Bug 17704 is looking at trying to get a later version of HTTP::OAI working with Koha, but it's been open for about 6 years now. (HTTP::OAI was also a dead project for a few years but it was resurrected in 2017 by one of the Catmandu authors. On that note, I think it is a good idea to avoid Catmandu.) One problem with HTTP::OAI that I encountered back in 2016 was that it needed to parse the entire XML response into a DOM Document tree rather than processing the XML response while it parsed it. This usually isn't a problem because most repositories use resumptionToken elements and limit responses to approximately 100 records. But LIBRIS in Sweden would stream the entire response back without resumptionToken elements, so 1 XML response could contain the entire catalogue's worth of records. That said, in theory the HTTP::OAI module uses event-driven SAX XML parsing, so it shouldn't be building a DOM Document tree from the response. Maybe the dev environment I was using in 2016 didn't have the correct SAX parser dependencies, so it was using a DOM-based parser in lieu of the SAX parser unintentionally. Plus, I suppose we could say that the Koha OAI-PMH harvester doesn't support OAI-PMH repositories that don't use resumptionToken elements for flow control. Or, since HTTP::OAI is no longer dead, that issue can always be pursued with the current maintainer. So overall... HTTP::OAI is probably the way to go. Just wanted to add a warning about my past experience with it. > What we would like is to use all the import tools already existing in Koha > (XSLT, Record matching rules, MARC modification templates, Stage marc for > import, Manage MARC overlay rules). > > We would like to add a OAI-PMH setting (like Z39-50 / SRU) in the staff > interface with URL, SET, XML Format, authentication login, biblio/authority > records, deleted records handling, email for logs, XSLT file, encoding, > items handling, profile import. Sounds like a plan. I suspect it will involve a lot of testing. It might be worthwhile to break some of that functionality out into separate tickets, so that the whole patch set doesn't need to be re-tested for minor fixes outside the core harvester functionality. > Every harvesting would be scheduled only via the cronjobs. That should make it easy to implement and test. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #318 from Koha Team University Lyon 3 --- Hi, here are some news of KohaLa's thoughts. We would like to use an existing OAI-PMH harvester and 2 of them seem to be good candidates : - Catmandu harverster - HTTP::OAI::HARVESTER module Both are in perl. At the moment, we thought that the second could be a better choice because it's more up to date and we don't necessarily need to use Catmandu. What we would like is to use all the import tools already existing in Koha (XSLT, Record matching rules, MARC modification templates, Stage marc for import, Manage MARC overlay rules). We would like to add a OAI-PMH setting (like Z39-50 / SRU) in the staff interface with URL, SET, XML Format, authentication login, biblio/authority records, deleted records handling, email for logs, XSLT file, encoding, items handling, profile import. Every harvesting would be scheduled only via the cronjobs. We are currently trying to propose specifications during the next 2 days (KohaLa hackathon). Don't hesitate to give us feedback on these first ideas. Sonia -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #317 from David Cook --- Occasionally I think about how I could do an OAI-PMH harvester plugin, but the tough part is that the harvester would need extra database tables, and I think having a plugin changing the database schema is not a great idea (or even possible depending on your database permissions). I suppose a person could take the OAI-PMH repository URL and the OAI-PMH record identifier, then hash them together, and then store that on a file system as a file. The content of that file could then contain additional data (like the Koha biblionumber). Of course, writing a lot of files to the same directory can be problematic, but Fedora Commons has already done some math on that and has recommended pair-tree patterns for creating an optimized sub-folder structure. -- I think that would just leave the need for a "koha-plugin" command that can invoke a particular plugin from the command line. A sysadmin could then schedule that whenever. Alternatively, one could use the "cronjob_nightly" Koha plugin hook. -- It wouldn't be all things to all people, but it would be a minimum viable OAI-PMH harvester... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #316 from David Cook --- Sonia, do you have any updates on KohaLA is going with OAI-PMH harvesting? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #315 from David Cook --- I just played a bit with the MarcEdit OAI-PMH tool. It can download OAI-PMH data, transform it, and save it to a local file. If you use the Windows Task Scheduler, you can also schedule this task. The problem I see is that you end up with MARC files that Koha won't be able to necessarily differentiate. DSpace's OAI-PMH harvester and my OAI-PMH harvesters all track the local system identifier against the OAI-PMH identifier, so it keeps a linkage between the upstream and downstream systems, so that updates and deletions are always perfectly matched up. Technically, you could probably write a crosswalk for MarcEdit that saves the OAI-PMH identifier and OAI-PMH repository URL into a custom field, so you could use it with Koha's record matching rules, but you'd have to make sure that custom field is indexed... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #314 from David Cook --- Fortunately, since we have RabbitMQ now, some of it should be a lot easier! (I was just looking at bug 27421 which asynchronously stages, imports, and reverts MARC imports. It could be helpful for this work too.) *The hard part for the OAI-PMH harvester in Koha is the task scheduling.* Koha doesn't have any way to let users define their own task schedules. (Back in 2018, Frido also mentioned Koha support companies might not want to let librarians set task schedules for OAI-PMH anyway for performance/API rate limiting reasons.) That said, if librarians controlling the scheduling doesn't matter for you, you could just create a cronjob that sends OAI-PMH tasks to RabbitMQ (or a plugin that uses the nightly plugin cronjob). Then all that's left is to create a Koha::BackgroundJob::OAIPMHHarvest class. -- The background job class could probably encapsulate the entire task. (For bug 10662, the requirement was to download records every 3 seconds, so I had to split the harvest/download and import tasks into two separate asynchronous tasks to achieve fast enough download speeds.) For bug 10662, I also had a requirement to handle very long XML streams over HTTP rather than the usual short XML responses, which technically is allowed according to the OAI-PMH specification, and that meant a custom downloader. It was high performance but it meant I had to add even more code. In theory, you might be able to have Koha::BackgroundJob::OAIPMH::Download, Koha::BackgroundJob::OAIPMH::Stage and Koha::BackgroundJob::OAIPMH::Import classes. The scheduler (e.g. cronjob) could enqueue a Koha::BackgroundJob::OAIPMH::Download task which downloads the records, that could then enqueue a Koha::BackgroundJob::OAIPMH::Stage task to stage (ie run the matcher/duplicate finder and ideally do some OAI-PMH specific checks), and that could enqueue the final Koha::BackgroundJob::OAIPMH::Import task to run the actual import. (The advantage of breaking it into 3 different tasks is that Koha by default only has 1 background job worker, so very long tasks could prevent other tasks from running in a timely way.) (However, if we had more than 1 background job worker, I'd be a little concerned about race conditions where Worker B tries to import Record 1-A after Worker A has imported Record 1-B where Record 1-A is older than Record 1-B. There needs to be a sanity check to make sure that records only overwrite older records.) Depending on how bug 27421 works, Koha::BackgroundJob::OAIPMH::Stage and Koha::BackgroundJob::OAIPMH::Import could potentially be subclasses of Koha::BackgroundJob::StageMARCForImport and Koha::BackgroundJob::StageMARCForImport. Although I don't really like Koha's built-in MARC import classes for OAI-PMH, because once records are staged they're imported without any sanity checks. Also record matching rules are user-controllable and solely MARC based so they're unreliable and not great for matching incoming OAI-PMH records to past harvested records. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #313 from David Cook --- (In reply to Koha Team University Lyon 3 from comment #311) > Hi, > KohaLA is intersted in continuing this feature and we are probably going to > fund some devs on that. > > Sonia That's great to hear! I think the problem with bug 10662 is that I tried to create too many things in one bug report. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #312 from Michal Denar --- Sonia, that's great news. I can get involved in testing and commenting on this functionality. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #311 from Koha Team University Lyon 3 --- Hi, KohaLA is intersted in continuing this feature and we are probably going to fund some devs on that. Sonia -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #310 from David Cook --- (In reply to Eugene Espinoza from comment #309) > Hi David! Any updates on this? Thanks! No, no updates on this one. Occasionally, I think about doing a simpler version, but I don't really have the time for it. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #309 from Eugene Espinoza --- Hi David! Any updates on this? Thanks! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #308 from David Cook --- (In reply to Michal Denar from comment #307) > Hi David, > any updates? > Not really. I haven't had the energy for working during my free time, so haven't looked at this in a couple months. It's always in the back of my mind though. I have been thinking a bit about waiting until after Bug 22417 is pushed though, as that could be useful for improving performance and robustness. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #307 from Michal Denar --- Hi David, any updates? Thank you. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=25905 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #306 from David Cook --- While I hate to keep waffling, I've been thinking that having the OAI-PMH ingest code in mainstream Koha is probably a good idea. I've opened #25905 to that end. I'll change my koha-plugin-oaipmh-import work to fit into mainstream Koha, and submit a patch on #25905. The initial work won't be very complicated, so it shouldn't be very difficult to test. I'll still do the OAI-PMH harvester as a plugin though, since what I have in mind is complex and not necessarily suitable for everyone. My hope is that when #25905 is done, people might set up their own harvesters as best fits their needs. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #305 from David Cook --- (In reply to Michal Denar from comment #304) > Hi, > I could participate on documetation. But I'm not developer, I need some "how > to" from David or Tomás, how integrate translation into code. No worries. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #304 from Michal Denar --- Hi, I could participate on documetation. But I'm not developer, I need some "how to" from David or Tomás, how integrate translation into code. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #303 from David Cook --- (In reply to Magnus Enger from comment #301) > (In reply to David Cook from comment #299) > > Looks like Tomas has already added translations to a plugin > > (https://gitlab.com/thekesolutions/plugins/koha-plugin-pay-via-paypal/-/tree/ > > master/Koha/Plugin/Com/Theke/PayViaPayPal), so I'll model my work on that. > > Bonus points to the person who documents how it's done, for example on the > wiki! :-) I'm only one man... But noted heh. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #302 from David Cook --- (In reply to Michal Denar from comment #300) > Hi David, > yes, I saw this solution. JSON file is easy to edit, I like it. Oh, it's not a JSON file. It'll be Template::Toolkit template files. Still very easy. Examples: https://gitlab.com/thekesolutions/plugins/koha-plugin-pay-via-paypal/-/blob/master/Koha/Plugin/Com/Theke/PayViaPayPal/i18n/cs-CZ.inc https://gitlab.com/thekesolutions/plugins/koha-plugin-pay-via-paypal/-/blob/master/Koha/Plugin/Com/Theke/PayViaPayPal/i18n/zh-Hans-CN.inc -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #301 from Magnus Enger --- (In reply to David Cook from comment #299) > Looks like Tomas has already added translations to a plugin > (https://gitlab.com/thekesolutions/plugins/koha-plugin-pay-via-paypal/-/tree/ > master/Koha/Plugin/Com/Theke/PayViaPayPal), so I'll model my work on that. Bonus points to the person who documents how it's done, for example on the wiki! :-) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #300 from Michal Denar --- Hi David, yes, I saw this solution. JSON file is easy to edit, I like it. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #299 from David Cook --- (In reply to Michal Denar from comment #296) > Hello David, > I'm ready for testing:-) Just one basic point: Koha is global, plugins have > to be translatable. > Looks like Tomas has already added translations to a plugin (https://gitlab.com/thekesolutions/plugins/koha-plugin-pay-via-paypal/-/tree/master/Koha/Plugin/Com/Theke/PayViaPayPal), so I'll model my work on that. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #298 from David Cook --- (In reply to David Cook from comment #297) > Thanks. I'm in a coding mood tonight, so I'm going to see how far I get > tonight. I've made significant process, but not quite done "koha-plugin-oaipmh-import" yet. I think maybe one or two more evenings, and I should have something testable. That's just for the OAI-PMH import functionality though. It doesn't include any OAI-PMH harvest/download functionality. That'll be handled by a separate plugin, although there's lots of possibilities for the download functionality. A person could write a simple script using HTTP::OAI or use a command line tool from online. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #297 from David Cook --- (In reply to Michal Denar from comment #296) > I'm ready for testing:-) Just one basic point: Koha is global, plugins have > to be translatable. > This is my first time working on Koha plugins, so I don't know what capacity there is for Koha plugin translations, but I'll keep that in mind. It might be that koha-plugin-oaipmh-import serves as a proof-of-concept which gets merged into the mainstream codebase, which would be easier to translate too. > I like and support your solution with standalone harvester and plugins for > administration and import process. Thanks. I'm in a coding mood tonight, so I'm going to see how far I get tonight. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #296 from Michal Denar --- Hello David, I'm ready for testing:-) Just one basic point: Koha is global, plugins have to be translatable. I like and support your solution with standalone harvester and plugins for administration and import process. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #295 from David Cook --- Now that I know the basics of Koha plugins and OpenAPI specs for API routes, I think the plugin progress will be quite quick. I am hoping that "koha-plugin-oaipmh-import" may actually be leveraged by many different OAI-PMH harvester tools. The standalone OAI-PMH harvester was required by Stockholm University Library, as they wanted a high performance interactive tool, but it might be overkill for many other libraries, which might just want to run a nightly cronjob. This plugin would allow any OAI-PMH harvester to feed records into Koha via the REST API. So I think this plugin is probably the most important component of the work really, as it will add the fundamental ability to ingest MARCXML records encapsulated within OAI-PMH records. With the "koha-plugin-oaipmh-import" plugin, people could use any number of existing OAI-PMH tools to create their own OAI-PMH harvesters to import records into Koha. But I'll still be working on creating an OAI-PMH harvester that meets the original stated needs of Stockholm University Library, as I still want them to be able to meet their goals. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #294 from David Cook --- (In reply to Michal Denar from comment #293) > I really like this idea. Czech Koha community want to help change this idea > to project. How can we cooperate? If we'll work together, we'll finish this > sooner. I hope. > > Michal That's great to hear, Michal! The thing that I will probably need most is testers. I don't have anything to test yet, but I am hoping to have something soon. Here's the list of components I'll be developing: 1. Koha UI - Koha plugin (Web UI to interact with OAI-PMH harvester HTTP management API) 2. Koha Import API - Koha plugin (Import API to receive harvested OAI-PMH records) 3. Standalone OAI-PMH harvester (Statically compiled Golang daemon with HTTP API for management) Last night, I started work on the "Koha Import API" plugin (https://github.com/minusdavid/koha-plugin-oaipmh-import). My next steps for that are clear, so I just need to sit down and do some work on that. Maybe tomorrow night. I've done experimental work on the "Standalone OAI-PMH harvester", but I haven't posted that to Github yet. I think that I've mostly settled on a scheduler methodology so hoping to make rapid progress on this one too. The plan with this will be to post the source code on Github, but also to provide a compiled binary that people can just download and run. (I might start with a Linux binary, but I've been thinking about doing a Windows binary too.) I may also provide a Docker image for the harvester as well to ease testing and deployment. The "Koha UI" plugin will probably be the last thing I do, since it depends on the HTTP API for the "Standalone OAI-PMH harvester". However, it should be fairly straight forward. Sorry for all the words! Just sharing my current plans! A lot of the work will be modeled off what I did for Bug 10662, but will probably be simpler. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #293 from Michal Denar --- I really like this idea. Czech Koha community want to help change this idea to project. How can we cooperate? If we'll work together, we'll finish this sooner. I hope. Michal -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #292 from David Cook --- I've been thinking more about this, and I like the following model: 1. Koha plugin (Web UI to interact with OAI-PMH harvester) 2. Koha plugin (Import API to receive harvested OAI-PMH records) 3. Standalone OAI-PMH harvester The OAI-PMH harvester's only job would be to schedule harvests and to execute those harvests (ie to download records) and then send those harvested records to Koha. A Koha plugin providing an OAI-PMH Import API would also allow any OAI-PMH harvester to work with Koha. (I might even pursue this as a separate Bugzilla bug for the core codebase as well, since it would be a nice integration to have that many different people could leverage. It would allow Koha to have OAI-PMH harvesting capabilities without being wedded to a particular OAI-PMH client implementation. This functionality would also be massively improved using a job queue like RabbitMQ...) A Koha plugin for interacting with the OAI-PMH harvester would just be for my OAI-PMH harvester implementation, but it would provide library administrators a lot more power than they usually would have with an OAI-PMH harvester. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #291 from David Cook --- On one hand, I feel like we're so close with the current patches. It just needs more unit tests. On the other hand, the unit tests for the task scheduling and concurrent processing are actually quite challenging. Also, these patches are a huge chunk of functionality, which increase Koha's overall size. I am tempted to take this code and split it into two parts: 1. Koha plugin for Web functionality 2. Standalone OAI-PMH harvester My thinking is the Koha plugin will allow you to connect to a separately packaged OAI-PMH Harvester API in order to add/start/stop/update/remove harvesting tasks. Easy! The standalone OAI-PMH harvester will then take care of the scheduling of tasks and high-performance downloading of records. The OAI-PMH harvester will then have actions to take on downloaded record batches. Not too hard! This is where things get interesting. Ideally, there would be a Koha API backed by a queue and Koha worker(s) to handle the processing of records. But that doesn't currently exist. I can use the Koha plugin to inject an API route, but there is no existing queue mechanism. Uh oh! The API could be used to store the records in a database table, but then I would need a Koha-based worker to access that database table and apply all the Koha-specific rules to the data. At this point it would be nice to have RabbitMQ for the queue, and then the Koha plugin could provide a Koha worker I suppose, which a sysadmin could manually start. I suppose we don't have to have RabbitMQ. The Koha worker could just tap into the database directly (until RabbitMQ is available). So in the end I suppose really 3 parts: 1. Koha plugin (Web functionality) 2. Koha plugin (Import Worker functionality) 3. Standalone OAI-PMH harvester Alternatively, the import API could handle all the Koha-related processing. I could do some tests to see how fast the web API could process the data. The OAI-PMH download will probably always be faster than the upload, so the OAI-PMH harvester would need to have its own internal queue for the downloaded records, but that could keep the Koha side of things slimmer. Plus... if Koha did implement a message queue like RabbitMQ, that change could be done transparently in the Koha background without affecting the OAI-PMH harvester. Ok so... 1. Koha plugin (Web UI to interact with OAI-PMH harvester) 2. Koha plugin (Import API to receive harvested OAI-PMH records) 3. Standalone OAI-PMH harvester I think that this makes sense. People could use the plugin, and then if they're liking it, then we could try again to get it into Koha master. I am actually interested in rewriting the OAI-PMH harvester in Golang to take advantage of its concurrent programming strengths. By using the Koha plugin to provide/consume APIs, we're able to use the best tools for the job for the actual OAI-PMH work. Note too that the OAI-PMH harvesting itself isn't actually Koha-specific. The only Koha-specific aspects are the scheduling and the record import. There's no real need to have the OAI-PMH code in the Koha codebase. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #290 from David Cook --- I've looked more at Bug 22417, and it's got me thinking. The OAI-PMH harvester has a few core needs: 1. Instant communication between Web UI and OAI-PMH harvester to coordinate harvesting tasks 2. Ability to schedule tasks 3. Ability to execute and repeat download tasks in parallel in the background 4. Ability to save downloaded records 5. Ability to import records into Koha The first is achieved by exchanging JSON messages over a Unix socket. (I'd actually like to change this to a TCP socket and use JSON messages over a HTTP API. Using HTTP would make it easier to communicate over a network, simplify the client code using standard communications mechanisms, and has authentication methods which could help to secure the OAI-PMH harvester. I could also provide a Docker image that contains the OAI-PMH harvester in a separate Docker container from the Koha application server.) This works fairly well and is easy enough to achieve. The second is provided by a bespoke scheduler using POE timers built into the OAI-PMH harvester. Given the granularity of the scheduling, I don't see this changing any time soon. In theory, a generic Koha task scheduler could replace this functionality, but that seems unlikely any time soon. This is arguably one of the most complex parts of the OAI-PMH harvester. Currently, the OAI-PMH uses an in-memory queue for download tasks, and a database queue for import tasks. I thought a lot about how RabbitMQ might be used to replace these queues. It could be useful to replace the in-memory download queue, and the download workers could be split out of the existing OAI-PMH harvester. As for the import tasks, the download workers need to save the records and enqueue an import task ASAP. At the moment, they save the records to disk, and add an import task to the database with a pointer to the records on disk. It works well enough, but it assumes that you have the disk space, and that the import worker has access to the local disk. I've been thinking it would be better to either 1) save the records to the database and enqueue a RabbitMQ message with a pointer to the database, or 2) send the records in a RabbitMQ message. I think the 1st option is probably better, because there is increased visibility. You can't see all the messages in a RabbitMQ queue. That all said, saving to disk is going to be faster than sending the data over a network. (However, in the past I think that I've sent individual records over the network. In this case, I'd be sending whole batches of records at once.) But for a download worker to send records, it would need credentials for either the database or RabbitMQ... so I'm thinking that perhaps it would be better to use an import API with an API key, although that would involve receiving all the data over the network and then sending it to the database. Also slow. But haven't tested the actual speeds. The import API would save the data to the database, and then enqueue an import task. Of course, this would just be a re-working of what's already here. The benefits of re-working the queues and workers are arguable at this point, although there is certainly benefits of changing from a bespoke client/server communication protocol to HTTP over TCP. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Sebastian Hierl changed: What|Removed |Added CC|s.hi...@aarome.org | -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #289 from David Cook --- So need to add "Import history" datatables use on http://localhost:8081/cgi-bin/koha/tools/oai-pmh-harvester/dashboard.pl Also need to look at Mirko's packaging issue. Also need to look at adding unit tests (although might be worthwhile to work on RabbitMQ first...) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #85228|0 |1 is obsolete|| --- Comment #286 from David Cook --- Created attachment 99743 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99743=edit Bug 10662: (follow-up) Template corrections and improvements This patch makes a number of corrections and improvements to the OAI harvester templates: - Add missing DataTables CSS include - Replace YUI grid with Bootstrap - Correct style of inline dialogs (.dialog.alert, .dialog.message) - Correct class of form hint and error messages - Format dates in saved and submitted requests tables - Add title-string sorting to DataTables configuration - Add delete confirmation to saved and submitted tables - Disable sorting on action columns - Style action links inside tables as buttons - Removed commented markup - Add missing JavaScript include (tools-menu.js) to highlight active section in sidebar menu - Add CodeMirror styling to record view page (CodeMirror XML mode file is added to enable this) - Remove invalid
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #85229|0 |1 is obsolete|| --- Comment #287 from David Cook --- Created attachment 99744 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99744=edit Bug 10662: (QA follow-up) Make atomic update consistent with kohastructrure. Remove utf8 charset Signed-off-by: Josef Moravec -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #288 from David Cook --- Created attachment 99745 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99745=edit Bug 10662: Strip UTC designators from header_datestamp -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #85227|0 |1 is obsolete|| --- Comment #285 from David Cook --- Created attachment 99742 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99742=edit Bug 10662: (QA follow-up) Enhance marc matchers description Signed-off-by: Josef Moravec -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #85225|0 |1 is obsolete|| --- Comment #283 from David Cook --- Created attachment 99740 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99740=edit Bug 10662: (QA follow-up) provide DBIC schema files DBIC schema files Signed-off-by: Josef Moravec -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #85226|0 |1 is obsolete|| --- Comment #284 from David Cook --- Created attachment 99741 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99741=edit Bug 10662: (QA follow-up) Fix plural in pod and use statements Signed-off-by: Josef Moravec -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #85224|0 |1 is obsolete|| --- Comment #282 from David Cook --- Created attachment 99739 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99739=edit Bug 10662: Build OAI-PMH Harvesting Client This patch adds an OAI-PMH harvesting client to Koha. The client runs as a daemon in the background. Users interact with the client via the Koha web user interface, which communicates with the daemon via a unix socket using a simple JSON-based protocol. The harvester ingests MARCXML. You can harvest other metadata formats, but you must use a XSLT to transform them into MARCXML, if you want them to be imported into Koha. You can supply your own download and import modules via the oai-pmh-harvester.yaml configuration file, but the default modules supplied in this patch should be good enough for your purposes. If they're not, raise a Bugzilla issue. There is a cleanup_database.pl addition, because high volume harvesting will cause the oai_harvester_import_queue table to fill quickly. This table is not required for adding/updating records. It's mostly just for general monitoring and audit purposes. Signed-off-by: Andreas Hedström Mace Signed-off-by: Josef Moravec -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #281 from David Cook --- Hmm running out of time today. Having KohaTable/DataTable issues... Uncaught TypeError: Cannot read property 'fnInit' of undefined at tb (datatables.min_19.1200030.js:90) at nb (datatables.min_19.1200030.js:72) at ha (datatables.min_19.1200030.js:87) at e (datatables.min_19.1200030.js:132) at HTMLTableElement. (datatables.min_19.1200030.js:132) at Function.each (jquery-2.2.3.min_19.1200030.js:2) at a.fn.init.each (jquery-2.2.3.min_19.1200030.js:2) at a.fn.init.n [as dataTable] (datatables.min_19.1200030.js:122) at KohaTable (dashboard.pl:885) at HTMLDocument. (dashboard.pl:919) I must be missing something that has changed with the use of KohaTable... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #280 from David Cook --- Ok rebased against master and running into that timestamp problem again, so I'll look at fixing that now. Once I have that fixed, I might actually go test Jonathan's work on RabbitMQ, since it would be great to replace my totally bespoke POE::Component::JobQueue implementation with a Koha Community RabbitMQ implementation. After reviewing my code, I'd be able to cut the vast majority of my code if we were using RabbitMQ instead. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #279 from David Cook --- Added the updated instructions for koha-testing-docker, since that's the tool I'll be using, and I think it's the tool a lot of you are now using too. I'll post some updated patches once I've worked out the latest kink. (I'm also mindful of some issues that Mirko flagged with the package related code.) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #278 from David Cook --- To setup koha-testing-docker: 0. Apply patches (e.g. git bz apply 10662) 1. apt-get install libpoe-perl libpoe-component-jobqueue-perl 2. In your browser, go to localhost:8081 and check that you get to the login page 3. cd /kohadevbox/koha 4. kshell -c "perl installer/data/mysql/updatedatabase.pl" 5. curl https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78583 > /kohadevbox/koha/oai.yml 6. sudo vi /etc/koha/sites/kohadev/koha-conf.xml Add the following before : /kohadevbox/koha/oai.yml 7. restart_all 8. KOHA_CONF=/etc/koha/sites/kohadev/koha-conf.xml PERL5LIB=/kohadevbox/koha perl misc/harvesterd.pl --log-level DEBUG To test: 1) Start OAI-PMH harvester daemon according to the above koha-testing-docker instructions (kohadevbox can be used instead but will need some modifications) 2) Go to /cgi-bin/koha/tools/tools-home.pl 3) Click on "OAI-PMH harvester" (/cgi-bin/koha/tools/oai-pmh-harvester/dashboard.pl) 4) At the toolbar choose "New request" 5) Give the request a "Name" (e.g. Test)(this is just used for Koha and not sent in the request), add "URL" for an OAI-PMH repository (e.g. http:///cgi-bin/koha/oai.pl), and fill in the OAI-PMH parameters as desired/required (an explanation of the protocol is at http://www.openarchives.org/OAI/openarchivesprotocol.html or see attached screenshot oai_request_example.jpg). The remaining values can be kept unchanged. 6) Click "Test parameters" to make sure your inputs are all valid. 7) Click "Save" 8) Click "Actions" and choose "Submit" 9) If the submission is successful, go to the "Submitted requests" tab 10) Click "Actions" and choose "Start" 11) Go to the "Import history" tab 12) Click "Refresh import history" to update the table and see your results. If nothing is coming up, you may need to change your request as the OAI-PMH repository may not have any results for that particular request. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #277 from David Cook --- Ok so I've rebased against master and now I'm testing in koha-testing-docker. I've run into a hiccup, which relates to the DB being stricter with timestamp/datetime fields I think. I'll have to fix that. Anyway, going to have to wait for another day. In any case, I really want to add unit tests for this. I think it's going to be challenging since the core app is a daemon running the POE event framework... but I'll do my best. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #276 from David Cook --- (In reply to Mirko Tietgen from comment #275) > (In reply to David Cook from comment #273) > > > It sounds like you've got the code dependencies but you'll need to set up > > the harvester daemon and run it as per > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662#c198. > > Thanks, that helped. Some comments: > - The Koha user can't create a directory in "/var/spool/koha/$instance/". We > have the DB backups there and the folder belongs to root. I think > /var/spool/koha/$instance/OAI" belonging to the Koha user would work better I think I've already taken that into account? When you do koha-create, it should use root access to create the relevant directory in /var/spool/koha/$instance/ and assign the permissions to the Koha user. I know I've already looked at this but maybe I'll need to look again. I don't think anyone else has had a problem with it? > - I have a problem related to the schema change in bug 22155 > > >[2019-05-02 10:06:00][DEBUG] [download][pid 2866][STDERR] Creating child > >process to download and feed parent process parser. at > >Koha/OAI/Harvester/Downloader.pm line 287. > >[2019-05-02 10:06:00][DEBUG] [download][pid 2866][STDERR] Creating parent > >process parser. at Koha/OAI/Harvester/Downloader.pm line 293. > >[2019-05-02 10:06:01][DEBUG] Registering POE::Session=ARRAY(0x91f7df8) as > >import for cf4d071b-76a9-4fff-b13d-c5a8bd27ed4a > >[2019-05-02 10:06:01][DEBUG] Child pid 3032 started as wheel 126 > >[2019-05-02 10:06:01][DEBUG] [import][pid 3032][STDERR] DBD::mysql::st > >execute failed: Unknown column 'me.marcflavour' in 'field list' [for > >Statement "SELECT `me`.`id`, `me`.`biblionumber`, `me`.`format`, > >`me`.`marcflavour`, `me`.`metadata`, `me`.`timestamp` FROM `biblio_metadata` > >`me` WHERE ( ( `me`.`biblionumber` = ? AND `me`.`format` = ? AND > >`me`.`marcflavour` = ? ) )" with ParamValues: 0='1271', 1='marcxml', > >2='MARC21'] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1832. > >[2019-05-02 10:06:01][DEBUG] [import][pid 3032][STDERR] > >DBIx::Class::Storage::DBI::_dbh_execute(): Unknown column 'me.marcflavour' > >in 'field list' at /usr/share/koha/lib/Koha/Objects.pm line 92 > >[2019-05-02 10:06:01][DEBUG] [import][pid 3032] closed all pipes > >[2019-05-02 10:06:01][DEBUG] [import][pid 3032] exited with status 0 Ah, interesting. I'll keep that in mind for when I find some time to work on this. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Blou changed: What|Removed |Added CC|bouzid.ferg...@inlibro.com, | |philippe.blo...@inlibro.com | -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #275 from Mirko Tietgen --- (In reply to David Cook from comment #273) > It sounds like you've got the code dependencies but you'll need to set up > the harvester daemon and run it as per > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662#c198. Thanks, that helped. Some comments: - The Koha user can't create a directory in "/var/spool/koha/$instance/". We have the DB backups there and the folder belongs to root. I think /var/spool/koha/$instance/OAI" belonging to the Koha user would work better - I have a problem related to the schema change in bug 22155 >[2019-05-02 10:06:00][DEBUG] [download][pid 2866][STDERR] Creating child >process to download and feed parent process parser. at >Koha/OAI/Harvester/Downloader.pm line 287. >[2019-05-02 10:06:00][DEBUG] [download][pid 2866][STDERR] Creating parent >process parser. at Koha/OAI/Harvester/Downloader.pm line 293. >[2019-05-02 10:06:01][DEBUG] Registering POE::Session=ARRAY(0x91f7df8) as >import for cf4d071b-76a9-4fff-b13d-c5a8bd27ed4a >[2019-05-02 10:06:01][DEBUG] Child pid 3032 started as wheel 126 >[2019-05-02 10:06:01][DEBUG] [import][pid 3032][STDERR] DBD::mysql::st execute >failed: Unknown column 'me.marcflavour' in 'field list' [for Statement "SELECT >`me`.`id`, `me`.`biblionumber`, `me`.`format`, `me`.`marcflavour`, >`me`.`metadata`, `me`.`timestamp` FROM `biblio_metadata` `me` WHERE ( ( >`me`.`biblionumber` = ? AND `me`.`format` = ? AND `me`.`marcflavour` = ? ) )" >with ParamValues: 0='1271', 1='marcxml', 2='MARC21'] at >/usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1832. >[2019-05-02 10:06:01][DEBUG] [import][pid 3032][STDERR] >DBIx::Class::Storage::DBI::_dbh_execute(): Unknown column 'me.marcflavour' in >'field list' at /usr/share/koha/lib/Koha/Objects.pm line 92 >[2019-05-02 10:06:01][DEBUG] [import][pid 3032] closed all pipes >[2019-05-02 10:06:01][DEBUG] [import][pid 3032] exited with status 0 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #274 from David Cook --- And thanks again, Mirko! I really wish that I had time to work on the tests right now, but I don't. Already burning the candle at both ends... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #273 from David Cook --- (In reply to Mirko Tietgen from comment #272) > I applied the patches, created a request, tested it ok, submitted and then … > nothing happens. "OAI-PMH harvester offline" > Is there something else I have to do? > Thanks for taking a look at this, Mirko! You might want to take a look at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662#c198. I'm not 100% sure what you've done so far, but basically you need to start up the OAI-PMH harvester daemon, which runs the code that does the real heavy lifting. > I can't choose "Record type: Authorities", is that intentional? > Yeah, at this point I've only built in support for bibliographic records. In theory, authorities should be easy enough, but I can see people trying to download both bibliographic and authority records from a different system and expecting them to stay linked, which they won't, so that's a problem for another day... > I manually installed libpoe-perl and libpoe-component-jobqueue-perl. Are > there other dependencies I mossed? It sounds like you've got the code dependencies but you'll need to set up the harvester daemon and run it as per https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662#c198. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #272 from Mirko Tietgen --- I applied the patches, created a request, tested it ok, submitted and then … nothing happens. "OAI-PMH harvester offline" Is there something else I have to do? I can't choose "Record type: Authorities", is that intentional? I manually installed libpoe-perl and libpoe-component-jobqueue-perl. Are there other dependencies I mossed? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #271 from Josef Moravec --- (In reply to David Cook from comment #269) > (In reply to Josef Moravec from comment #266) > > David, do you think it is possible for you to write test as soon as it could > > get into 19.05 release? It would be great to have it in 19.05... > > Do we know the timelines for the 19.05 release yet? I haven't seen anything > online. Not yet, but Nick announced he is going to think out and publish terms soon. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #270 from Magnus Enger --- See the section on “Unit tests” in the Coding guidelines: https://wiki.koha-community.org/wiki/Coding_Guidelines#PERL17:_Unit_tests_are_required_.28updated_Apr_26.2C_2017.29 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #269 from David Cook --- (In reply to Josef Moravec from comment #266) > David, do you think it is possible for you to write test as soon as it could > get into 19.05 release? It would be great to have it in 19.05... Do we know the timelines for the 19.05 release yet? I haven't seen anything online. I'm skeptical about having the time to write them before the 19.05 release. A heavy workload at the office and pressing matters at home mean time is quite short at the moment. That said, can you tell me more about the test coverage we're looking for? I could go to very time-consuming lengths testing the client-server relationship and POE event handling for the OAI-PMH harvester server. Or is it more important to first focus on functions that directly affect Koha? Much of the code is quite independent from Koha, so less vulnerable to breaking due to changes in the rest of Koha. I'd be inclined to focus most on adding tests for modules that reference the database schema or core APIs (like C4::Biblio). Does that make sense? I know we'd like 100% test coverage, but I'm curious what the minimum requirements are for the tests. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #268 from Andreas Hedström Mace --- Very very awesome, yes! =) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #267 from Magnus Enger --- (In reply to Josef Moravec from comment #266) > David, do you think it is possible for you to write test as soon as it could > get into 19.05 release? It would be great to have it in 19.05... It would be awesome, indeed! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #266 from Josef Moravec --- (In reply to David Cook from comment #265) > (In reply to Josef Moravec from comment #264) > > I tested again. I have to say it is looking good. So there is the only last > > thing, but big one: the tests... > > Awesome! I am very busy at the moment, but I'll get to the tests when I have > some time. David, do you think it is possible for you to write test as soon as it could get into 19.05 release? It would be great to have it in 19.05... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #265 from David Cook --- (In reply to Josef Moravec from comment #264) > I tested again. I have to say it is looking good. So there is the only last > thing, but big one: the tests... Awesome! I am very busy at the moment, but I'll get to the tests when I have some time. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Josef Moravec changed: What|Removed |Added Status|Signed Off |Failed QA --- Comment #264 from Josef Moravec --- I tested again. I have to say it is looking good. So there is the only last thing, but big one: the tests... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Josef Moravec changed: What|Removed |Added Attachment #84321|0 |1 is obsolete|| --- Comment #261 from Josef Moravec --- Created attachment 85227 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=85227=edit Bug 10662: (QA follow-up) Enhance marc matchers description Signed-off-by: Josef Moravec -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Josef Moravec changed: What|Removed |Added Attachment #84320|0 |1 is obsolete|| --- Comment #260 from Josef Moravec --- Created attachment 85226 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=85226=edit Bug 10662: (QA follow-up) Fix plural in pod and use statements Signed-off-by: Josef Moravec -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Josef Moravec changed: What|Removed |Added Attachment #85152|0 |1 is obsolete|| --- Comment #262 from Josef Moravec --- Created attachment 85228 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=85228=edit Bug 10662: (follow-up) Template corrections and improvements This patch makes a number of corrections and improvements to the OAI harvester templates: - Add missing DataTables CSS include - Replace YUI grid with Bootstrap - Correct style of inline dialogs (.dialog.alert, .dialog.message) - Correct class of form hint and error messages - Format dates in saved and submitted requests tables - Add title-string sorting to DataTables configuration - Add delete confirmation to saved and submitted tables - Disable sorting on action columns - Style action links inside tables as buttons - Removed commented markup - Add missing JavaScript include (tools-menu.js) to highlight active section in sidebar menu - Add CodeMirror styling to record view page (CodeMirror XML mode file is added to enable this) - Remove invalid
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #263 from Josef Moravec --- Created attachment 85229 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=85229=edit Bug 10662: (QA follow-up) Make atomic update consistent with kohastructrure. Remove utf8 charset Signed-off-by: Josef Moravec -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Josef Moravec changed: What|Removed |Added Attachment #84319|0 |1 is obsolete|| --- Comment #259 from Josef Moravec --- Created attachment 85225 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=85225=edit Bug 10662: (QA follow-up) provide DBIC schema files DBIC schema files Signed-off-by: Josef Moravec -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Josef Moravec changed: What|Removed |Added Attachment #84318|0 |1 is obsolete|| --- Comment #258 from Josef Moravec --- Created attachment 85224 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=85224=edit Bug 10662: Build OAI-PMH Harvesting Client This patch adds an OAI-PMH harvesting client to Koha. The client runs as a daemon in the background. Users interact with the client via the Koha web user interface, which communicates with the daemon via a unix socket using a simple JSON-based protocol. The harvester ingests MARCXML. You can harvest other metadata formats, but you must use a XSLT to transform them into MARCXML, if you want them to be imported into Koha. You can supply your own download and import modules via the oai-pmh-harvester.yaml configuration file, but the default modules supplied in this patch should be good enough for your purposes. If they're not, raise a Bugzilla issue. There is a cleanup_database.pl addition, because high volume harvesting will cause the oai_harvester_import_queue table to fill quickly. This table is not required for adding/updating records. It's mostly just for general monitoring and audit purposes. Signed-off-by: Andreas Hedström Mace Signed-off-by: Josef Moravec -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #257 from Andreas Hedström Mace --- Tested again, and the functionality works perfectly. As far as I can tell all the stylings are now correct. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Andreas Hedström Mace changed: What|Removed |Added Attachment #84322|0 |1 is obsolete|| --- Comment #256 from Andreas Hedström Mace --- Created attachment 85152 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=85152=edit Bug 10662: (follow-up) Template corrections and improvements This patch makes a number of corrections and improvements to the OAI harvester templates: - Add missing DataTables CSS include - Replace YUI grid with Bootstrap - Correct style of inline dialogs (.dialog.alert, .dialog.message) - Correct class of form hint and error messages - Format dates in saved and submitted requests tables - Add title-string sorting to DataTables configuration - Add delete confirmation to saved and submitted tables - Disable sorting on action columns - Style action links inside tables as buttons - Removed commented markup - Add missing JavaScript include (tools-menu.js) to highlight active section in sidebar menu - Add CodeMirror styling to record view page (CodeMirror XML mode file is added to enable this) - Remove invalid
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Andreas Hedström Mace changed: What|Removed |Added Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #82219|0 |1 is obsolete|| --- Comment #255 from David Cook --- Created attachment 84322 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=84322=edit Bug 10662: (follow-up) Template corrections and improvements This patch makes a number of corrections and improvements to the OAI harvester templates: - Add missing DataTables CSS include - Replace YUI grid with Bootstrap - Correct style of inline dialogs (.dialog.alert, .dialog.message) - Correct class of form hint and error messages - Format dates in saved and submitted requests tables - Add title-string sorting to DataTables configuration - Add delete confirmation to saved and submitted tables - Disable sorting on action columns - Style action links inside tables as buttons - Removed commented markup - Add missing JavaScript include (tools-menu.js) to highlight active section in sidebar menu - Add CodeMirror styling to record view page (CodeMirror XML mode file is added to enable this) - Remove invalid
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81864|0 |1 is obsolete|| --- Comment #254 from David Cook --- Created attachment 84321 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=84321=edit Bug 10662: (QA follow-up) Enhance marc matchers description -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81862|0 |1 is obsolete|| --- Comment #252 from David Cook --- Created attachment 84319 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=84319=edit Bug 10662: (QA follow-up) provide DBIC schema files DBIC schema files -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81863|0 |1 is obsolete|| --- Comment #253 from David Cook --- Created attachment 84320 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=84320=edit Bug 10662: (QA follow-up) Fix plural in pod and use statements -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81861|0 |1 is obsolete|| --- Comment #251 from David Cook --- Created attachment 84318 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=84318=edit Bug 10662: Build OAI-PMH Harvesting Client This patch adds an OAI-PMH harvesting client to Koha. The client runs as a daemon in the background. Users interact with the client via the Koha web user interface, which communicates with the daemon via a unix socket using a simple JSON-based protocol. The harvester ingests MARCXML. You can harvest other metadata formats, but you must use a XSLT to transform them into MARCXML, if you want them to be imported into Koha. You can supply your own download and import modules via the oai-pmh-harvester.yaml configuration file, but the default modules supplied in this patch should be good enough for your purposes. If they're not, raise a Bugzilla issue. There is a cleanup_database.pl addition, because high volume harvesting will cause the oai_harvester_import_queue table to fill quickly. This table is not required for adding/updating records. It's mostly just for general monitoring and audit purposes. Signed-off-by: Andreas Hedström Mace -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Ed Veal changed: What|Removed |Added CC||ev...@mckinneytexas.org -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #250 from David Cook --- Ready and waiting for testers/QA. Not sure where we're at in terms of sign offs at the moment. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81902|0 |1 is obsolete|| --- Comment #249 from David Cook --- Created attachment 82219 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82219=edit Bug 10662: (follow-up) Template corrections and improvements This patch makes a number of corrections and improvements to the OAI harvester templates: - Add missing DataTables CSS include - Replace YUI grid with Bootstrap - Correct style of inline dialogs (.dialog.alert, .dialog.message) - Correct class of form hint and error messages - Format dates in saved and submitted requests tables - Add title-string sorting to DataTables configuration - Add delete confirmation to saved and submitted tables - Disable sorting on action columns - Style action links inside tables as buttons - Removed commented markup - Add missing JavaScript include (tools-menu.js) to highlight active section in sidebar menu - Add CodeMirror styling to record view page (CodeMirror XML mode file is added to enable this) - Remove invalid
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #248 from David Cook --- Thanks to Owen for all that work! It looks so much better now! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #247 from David Cook --- The patch applies fine to master for me, and I'm having no trouble loading the database. Andreas, try running "reset_all" in your kohadevbox. I'm guessing you're getting a conflict with an existing database. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #246 from Andreas Hedström Mace --- Updatedatabase.pl fails for me. DEV atomic update: bug_10662.sql C4::Installer::load_sql returned the following errors while attempting to load /home/vagrant/kohaclone/installer/data/mysql/atomicupdate/bug_10662.sql: (No actual error shown) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Andreas Hedström Mace changed: What|Removed |Added Status|Patch doesn't apply |Needs Signoff --- Comment #245 from Andreas Hedström Mace --- Never mind my previous comment, I didn't have the latest master. Testing now. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Andreas Hedström Mace changed: What|Removed |Added Status|Needs Signoff |Patch doesn't apply --- Comment #244 from Andreas Hedström Mace --- Patch doesn't apply for me: Applying: Bug 10662: Build OAI-PMH Harvesting Client Using index info to reconstruct a base tree... M C4/Installer/PerlDependencies.pm M admin/columns_settings.yml M debian/scripts/koha-create M installer/data/mysql/kohastructure.sql M koha-tmpl/intranet-tmpl/prog/en/includes/tools-menu.inc M koha-tmpl/intranet-tmpl/prog/en/modules/tools/tools-home.tt Falling back to patching base and 3-way merge... Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/tools/tools-home.tt Auto-merging koha-tmpl/intranet-tmpl/prog/en/includes/tools-menu.inc Auto-merging installer/data/mysql/kohastructure.sql CONFLICT (content): Merge conflict in installer/data/mysql/kohastructure.sql Auto-merging debian/scripts/koha-create Auto-merging admin/columns_settings.yml CONFLICT (content): Merge conflict in admin/columns_settings.yml Auto-merging C4/Installer/PerlDependencies.pm Failed to merge in the changes. Patch failed at 0001 Bug 10662: Build OAI-PMH Harvesting Client -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Owen Leonard changed: What|Removed |Added Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #243 from Owen Leonard --- Created attachment 81902 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=81902=edit Bug 10662: (follow-up) Template corrections and improvements This patch makes a number of corrections and improvements to the OAI harvester templates: - Add missing DataTables CSS include - Replace YUI grid with Bootstrap - Correct style of inline dialogs (.dialog.alert, .dialog.message) - Correct class of form hint and error messages - Format dates in saved and submitted requests tables - Add title-string sorting to DataTables configuration - Add delete confirmation to saved and submitted tables - Disable sorting on action columns - Style action links inside tables as buttons - Removed commented markup - Add missing JavaScript include (tools-menu.js) to highlight active section in sidebar menu - Add CodeMirror styling to record view page (CodeMirror XML mode file is added to enable this) - Remove invalid
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662
--- Comment #242 from Owen Leonard ---
(In reply to David Cook from comment #241)
> - DataTable styling
I'm preparing a template follow-up to this, but the only thing that is a
blocker interface-wise is the DataTables styling. The problem is this is
missing from the template:
[% Asset.css("css/datatables.css") | $raw %]
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #241 from David Cook --- OK. I have fixed: - column collation in installer/data/mysql/atomicupdate/bug_10662.sql and installer/data/mysql/kohastructure.sql - fixed the "Refresh import history" button/ajax KohaTable/DataTable - fixed UTF-8 encoding problem in client-server communications - separated out the Koha/Schema/Result/*.pm schema files I think the only things I haven't changed are: - SQL statement handle I re-use - DataTable styling - matcher_id foreign key issue -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81855|0 |1 is obsolete|| -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81856|0 |1 is obsolete|| -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81784|0 |1 is obsolete|| -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81790|0 |1 is obsolete|| --- Comment #240 from David Cook --- Created attachment 81864 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=81864=edit Bug 10662: (QA follow-up) Enhance marc matchers description -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81783|0 |1 is obsolete|| --- Comment #237 from David Cook --- Created attachment 81861 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=81861=edit Bug 10662: Build OAI-PMH Harvesting Client This patch adds an OAI-PMH harvesting client to Koha. The client runs as a daemon in the background. Users interact with the client via the Koha web user interface, which communicates with the daemon via a unix socket using a simple JSON-based protocol. The harvester ingests MARCXML. You can harvest other metadata formats, but you must use a XSLT to transform them into MARCXML, if you want them to be imported into Koha. You can supply your own download and import modules via the oai-pmh-harvester.yaml configuration file, but the default modules supplied in this patch should be good enough for your purposes. If they're not, raise a Bugzilla issue. There is a cleanup_database.pl addition, because high volume harvesting will cause the oai_harvester_import_queue table to fill quickly. This table is not required for adding/updating records. It's mostly just for general monitoring and audit purposes. Signed-off-by: Andreas Hedström Mace -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook changed: What|Removed |Added Attachment #81789|0 |1 is obsolete|| --- Comment #239 from David Cook --- Created attachment 81863 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=81863=edit Bug 10662: (QA follow-up) Fix plural in pod and use statements -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #238 from David Cook --- Created attachment 81862 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=81862=edit Bug 10662: (QA follow-up) provide DBIC schema files DBIC schema files -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #236 from David Cook --- (In reply to David Cook from comment #224) > Comment on attachment 81789 [details] [review] > Bug 10662: (QA follow-up) Fix plural in pod and use statements > > Review of attachment 81789 [details] [review]: > - > > ::: Koha/OAI/Harvester/Import/Record.pm > @@ +29,4 @@ > > > > use Koha::OAI::Harvester::Import::MARCXML; > > use Koha::OAI::Harvester::Biblios; > > +use Koha::OAI::Harvester::Histories; > > This actually should be Koha::OAI::Harvester::History and not > Koha::OAI::Harvester::Histories. This change breaks the harvester. Actually, since Histories uses History, it's probably fine... but I don't actually use Histories in Record.pm. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #235 from David Cook --- (In reply to Josef Moravec from comment #209) > Created attachment 81786 [details] > Encoding problem + datatable > > In submitted requests table I encountered a encoding problem: > > "Knihovna ÃstÃ" should be "Knihovna Ústí". > > Also, the heading of datatable is usually formatted in one line, see patron > circulation history for example. Ok I've fixed that. I was reversing the Encode module encode/decode functions and not reading the output of Devel::Peek correctly. Now I can see both Knihovna Ústí and 我不好 coming back from the harvester as well. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #234 from David Cook --- Yep... put in some Chinese characters and now that decode isn't working either. Let's try this again... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662
--- Comment #233 from David Cook ---
(In reply to Josef Moravec from comment #209)
> Created attachment 81786 [details]
> Encoding problem + datatable
>
> In submitted requests table I encountered a encoding problem:
>
> "Knihovna ÃstÃ" should be "Knihovna Ústí".
>
> Also, the heading of datatable is usually formatted in one line, see patron
> circulation history for example.
I am stumped by this one.
I've added a Encode::Decode("UTF-8",$json_message") to the client used by the
web page, and that gets the characters to render as Knihovna Ústí on the web
page... but when I look at the actual hex code in the variable... it's not
valid UTF8. It's Windows 1252/Latin-1.
The hex is 4b6e69686f766e6120da7374ed, and the internal representation in Perl
is PV = 0xa6bce80 "Knihovna \303\232st\303\255"\0 [UTF8 "Knihovna
\x{da}st\x{ed}"].
So DA and ED are Unicode code points that match Ú and í
(https://www.utf8-chartable.de/unicode-utf8-table.pl).
However, DA and ED are also the Latin-1 hex for Ú and í
(https://en.wikipedia.org/wiki/Windows-1252)(https://en.wikipedia.org/wiki/ISO/IEC_8859-1).
I think maybe I need to try with some Chinese characters that don't exist in
Latin-1...
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 10662] Build OAI-PMH Harvesting Client
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #232 from David Cook --- (In reply to David Cook from comment #226) > Interesting... > > I can't even submit a task with a name of "Knihovna Ústí". The harvester is > failing to add it. I'll look into it. Ah, I see the problem I was having. Just me being silly. Now I see the problem. I have some ideas with this one... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
