[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon--- via Koha-bugs]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

--- Comment #24 from Katrin Fischer  ---
Pushed for 24.05!

Well done everyone, thank you!

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon--- via Koha-bugs]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Katrin Fischer  changed:

   What|Removed |Added

 Version(s)||24.05.00
released in||
 Status|Passed QA   |Pushed to main

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon--- via Koha-bugs]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

--- Comment #23 from Katrin Fischer  ---
I do like the documentation and the --confirm :)

I haven't tested this deeply, but it comes with a test run mode and is also
self contained. It kinda feels like we should be able to have tests for this...
but I know it's just a CLI script.

I notice it takes a while, even on my small sample database.

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon--- via Koha-bugs]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

--- Comment #22 from David Cook  ---
Ohhh I kept meaning to look at this. 

I wrote an authority de-duplicator years ago, and I'm planning on reviewing it
soon. 

Maybe I can replace my local one with this one. I'll have to take a look...

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon--- via Koha-bugs]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

David Nind  changed:

   What|Removed |Added

  Text to go in the||This new command line
  release notes||script
   ||(misc/maintenance/dedup_aut
   ||horities.pl) is a tool for
   ||helping to automatically
   ||remove duplicate authority
   ||records.
   ||
   ||Features:
   ||-
   ||Choose a method(s) to
   ||select which authority
   ||record to keep when
   ||duplicates are found.
   ||Methods include:
   ||  . date:
   ||keep the most recent
   ||authority (based on 005
   ||field)
   ||  . used: keep the
   ||most used authority
   ||  .
   ||ppn: PPN (UNIMARC only),
   ||keep the authority with a
   ||ppn (when some authorities
   ||don't have one, based on
   ||009 field)
   ||- Use a SQL
   ||WHERE statement to limit
   ||the authority records
   ||checked for deduplication
   ||-
   ||Check only specified
   ||authority types
   ||- Increase
   ||the level of detail shown
   ||using the --verbose option
   ||- Changes are only made
   ||when the --confirm option
   ||is used
   ||
   ||Examples:
   ||-
   ||Methods - for the
   ||authorities that have a
   ||PPN, keep the most recent,
   ||and if two (or more) have
   ||the same date in 005, keep
   ||the most used: --method ppn
   ||--method date --method used
   ||
   ||- SQL WHERE statement -
   ||only look at records with
   ||an auth_id less than 5,000:
   ||--where="authid < 5000"
   ||-
   ||Limit deduplication to
   ||specific authority types:
   ||--authtypecode PERSO_NAME
   ||
   ||See the script help for the
   ||options available and usage
   ||examples
   ||(misc/maintenance/dedup_aut
   ||horities.pl --help).

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon--- via Koha-bugs]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Tomás Cohen Arazi  changed:

   What|Removed |Added

 QA Contact|testo...@bugs.koha-communit |tomasco...@gmail.com
   |y.org   |
 CC||tomasco...@gmail.com

--- Comment #21 from Tomás Cohen Arazi  ---
This is a nice new feature, self-contained on a new script.

I have some doubts about the output format, but I think we want to listen from
real-life users and enhance this later with good feedback from users.

Nice job, Nick!!!

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon--- via Koha-bugs]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

--- Comment #20 from Tomás Cohen Arazi  ---
Created attachment 163031
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163031=edit
Bug 13706: Minor tidy/spelling issues

Signed-off-by: Tomas Cohen Arazi 

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon--- via Koha-bugs]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Tomás Cohen Arazi  changed:

   What|Removed |Added

 Attachment #157965|0   |1
is obsolete||

--- Comment #19 from Tomás Cohen Arazi  ---
Created attachment 163030
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163030=edit
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying
field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements

Signed-off-by: David Nind 
Signed-off-by: Tomas Cohen Arazi 

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon--- via Koha-bugs]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Tomás Cohen Arazi  changed:

   What|Removed |Added

 Status|Signed Off  |Passed QA

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

--- Comment #18 from David Nind  ---
It seems to work with my limited testing - if this isn't sufficient, please
change the status.

The only thing I found confusing was the message shown when run "Deleted XX
authorities" - I'm assuming there is some logic here that I don't quite get.

Basic testing notes (using KTD):

1. Go to Authorities
2. Duplicate an existing personal name authority that is used in a record
3. Note the original and duplicate authority number
4. Run the script: misc/maintenance/dedup_authorities.pl -v -a PERSO_NAME -m
date -c
5. Check the authorities in the staff interface - one is deleted and one is
kept

I did some other testing, notes as follows.

FYI 
===

Number of terms for authority types - from the script and search results in the
staff interface:

Chronological (CHRON_TERM): Script: 0 ; Search result: 0
Corporate Name (CORPO_NAME): Script: 88 ; Search result: 88
Genre/Form Term (GENRE/FORM): Script: 49 ; Search result: 49
Geographic Name (GEOGR_NAME): Script: 142 ; Search result: 142
Meeting Name (MEETI_NAME): Script: 3 ; Search result: 3
Personal Name (PERSO_NAME): Script: 650 ; Search result: 650
Topical Term (TOPIC_TERM): Script: 663 ; Search result: 663
Uniform Title (UNIF_TITLE): Script: 111 ; Search result: 111

Default/No authority type selected: Script: 1706 ; Search result: 1706

Script result - showing authority records by type:

kohadev-koha@kohadevbox:koha(bz13706)$ misc/maintenance/dedup_authorities.pl -v
-m date
RUNNING IN TEST MODE, NO CHANGES WILL BE MADE
Fetching authtypecodes...
Fetching authtypecodes done.
Deduping authtype '' 
Fetching authorities for ''... 0 authorities found
End of deduping for authtype ''
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'CHRON_TERM' 
Fetching authorities for 'CHRON_TERM'... 0 authorities found
End of deduping for authtype 'CHRON_TERM'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'CORPO_NAME' 
Fetching authorities for 'CORPO_NAME'... 88 authorities found
End of deduping for authtype 'CORPO_NAME'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'GENRE/FORM' 
Fetching authorities for 'GENRE/FORM'... 49 authorities found
End of deduping for authtype 'GENRE/FORM'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'GEOGR_NAME' 
Fetching authorities for 'GEOGR_NAME'... 142 authorities found
Progression for authtype 'GEOGR_NAME': 100/142 (70.42%)
End of deduping for authtype 'GEOGR_NAME'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'MEETI_NAME' 
Fetching authorities for 'MEETI_NAME'... 3 authorities found
End of deduping for authtype 'MEETI_NAME'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'PERSO_NAME' 
Fetching authorities for 'PERSO_NAME'... 650 authorities found
Malformed authority record, no heading at
misc/maintenance/dedup_authorities.pl line 172.
Progression for authtype 'PERSO_NAME': 100/650 (15.38%)
Progression for authtype 'PERSO_NAME': 200/650 (30.77%)
Progression for authtype 'PERSO_NAME': 300/650 (46.15%)
Progression for authtype 'PERSO_NAME': 400/650 (61.54%)
Progression for authtype 'PERSO_NAME': 500/650 (76.92%)
Malformed authority record, blank heading at
misc/maintenance/dedup_authorities.pl line 176.
End of deduping for authtype 'PERSO_NAME'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'TOPIC_TERM' 
Fetching authorities for 'TOPIC_TERM'... 663 authorities found
Progression for authtype 'TOPIC_TERM': 100/663 (15.08%)
Progression for authtype 'TOPIC_TERM': 200/663 (30.17%)
Progression for authtype 'TOPIC_TERM': 300/663 (45.25%)
Progression for authtype 'TOPIC_TERM': 400/663 (60.33%)
Progression for authtype 'TOPIC_TERM': 500/663 (75.41%)
End of deduping for authtype 'TOPIC_TERM'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'UNIF_TITLE' 
Fetching authorities for 'UNIF_TITLE'... 111 authorities found
End of deduping for authtype 'UNIF_TITLE'
Updated 0 biblios
Deleted 0 authorities
No biblios to update


Testing using Genre/Form Term
=

Summary
~~~

Total before script run: 49 terms

Manually went through results in staff interface to identify duplicates:

Commedy films. 982 (deleted), 1586 (kept)
Feature films. 625 (deleted), 650 (deleted), 654 (deleted), 822 (deleted), 984
(kept), 987 (deleted)
Fiction films. 823 (deleted), 985 (kept), 988 (deleted)
Foreign films. 626 (kept), 988 (deleted)
Historical fiction. 1018 (kept), 1019 (deleted)
Video recordings for the hearing impaired. 986 (kept), 989 (deleted)
Summary = 17 terms, should only be 6 = 11 should be deleted

Expected number of terms to be deleted: 11 (result would be 38 terms left)

Results from running the script - shows 14 deleted (3 shown as deleted twice):
987, 822, 625, 625, 650, 654, 655, 988, 823, 823, 982, 982, 989, 1019

End result is the same, not sure why showing as it does.

Search results after in the staff interface: 38 
Script results after: 38

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

David Nind  changed:

   What|Removed |Added

 Attachment #157935|0   |1
is obsolete||

--- Comment #17 from David Nind  ---
Created attachment 157965
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157965=edit
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying
field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements

Signed-off-by: David Nind 

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

David Nind  changed:

   What|Removed |Added

 Status|Needs Signoff   |Signed Off

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Nick Clemens  changed:

   What|Removed |Added

 Attachment #127263|0   |1
is obsolete||

--- Comment #16 from Nick Clemens  ---
Created attachment 157935
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157935=edit
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying
field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Nick Clemens  changed:

   What|Removed |Added

 Status|Failed QA   |Needs Signoff

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Enica Davis  changed:

   What|Removed |Added

 CC||en...@bywatersolutions.com

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

David Nind  changed:

   What|Removed |Added

 Status|Needs Signoff   |Failed QA
 CC||da...@davidnind.com

--- Comment #15 from David Nind  ---
I had a go at testing, but I got some errors when running the script:

- with the help option:

misc/maintenance/dedup_authorities.pl --help
Undefined subroutine ::pod2usage called at
misc/maintenance/dedup_authorities.pl line 99.

- duplicated an authority record, the attempted to run using the date option:

misc/maintenance/dedup_authorities.pl -v -c -m date
Fetching authtypecodes...
The method Koha::Authority::Types->authtypecode is not covered by tests!

Trace begun at /kohadevbox/koha/Koha/Objects.pm line 572
Koha::Objects::AUTOLOAD('Koha::Authority::Types=HASH(0x564621d97918)') called
at misc/maintenance/dedup_authorities.pl line 132

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Katrin Fischer  changed:

   What|Removed |Added

 Status|ASSIGNED|Needs Signoff

--- Comment #14 from Katrin Fischer  ---
I came here looking through old bugs - a new past time of mine. I am setting to
NSO then.

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

--- Comment #13 from Nick Clemens  ---
We use this in production as-is - if you like it Katrin, feel free to sign-off
- I would like it all to be covered by tests but don't know when I will have a
chance to work on that

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

--- Comment #12 from Katrin Fischer  ---
Is this a forgotten one? Still sounds interesting!

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Agnes Rivers-Moore  changed:

   What|Removed |Added

 CC||a...@hanover.ca

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Juliet Heltibridle  changed:

   What|Removed |Added

 CC||jheltibri...@rcplib.org

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Nick Clemens  changed:

   What|Removed |Added

 Attachment #124659|0   |1
is obsolete||

--- Comment #11 from Nick Clemens  ---
Created attachment 127263
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127263=edit
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying
field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Cab Vinton  changed:

   What|Removed |Added

 CC||bibli...@gmail.com

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Nick Clemens  changed:

   What|Removed |Added

 Attachment #122049|0   |1
is obsolete||

--- Comment #10 from Nick Clemens  ---
Created attachment 124659
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=124659=edit
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying
field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Nick Clemens  changed:

   What|Removed |Added

 CC||n...@bywatersolutions.com
   Assignee|alex.arn...@biblibre.com|n...@bywatersolutions.com
 Status|Failed QA   |ASSIGNED

--- Comment #9 from Nick Clemens  ---
Stealing this one unless Alex or Julian object

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Nick Clemens  changed:

   What|Removed |Added

  Attachment #35885|0   |1
is obsolete||
  Attachment #40069|0   |1
is obsolete||
 Attachment #121920|0   |1
is obsolete||

--- Comment #8 from Nick Clemens  ---
Created attachment 122049
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=122049=edit
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying
field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

--- Comment #7 from Nick Clemens  ---
Created attachment 121920
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=121920=edit
Bug 13706: Update some code

This updates a few things:
- ZebraIndex module does not exist
- merge calls ModBiblio, so the records are reindexed during merge
- DelAuthority used to remove records - no reindex needed after

Remaining issues:
- SimpleSearch doesn't work - only 'he' has an entry in ccl.properties
- abbreviations for indexes (he-main etc) do not exist anywhere
- There should be an option to use 'Match' and automatically generate the
search term from the auth record (search_form in Headings.pm) based on the
authtype
- The default options are unimarc centric
- this should be in module and testable

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

David Cook  changed:

   What|Removed |Added

 CC||jcam...@cpbibliography.com

--- Comment #6 from David Cook  ---
*** Bug 7419 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

David Cook  changed:

   What|Removed |Added

 CC||dc...@prosentient.com.au

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Holly  changed:

   What|Removed |Added

 CC||h...@interleaf.ie

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706
Bug 13706 depends on bug 11700, which changed state.

Bug 11700 Summary: C4::AuthoritiesMarc::merge enhancements/fixes
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11700

   What|Removed |Added

 Status|In Discussion   |RESOLVED
 Resolution|--- |DUPLICATE

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Marjorie Barry-Vila  changed:

   What|Removed |Added

 CC||marjorie.barry-v...@ccsr.qc
   ||.ca

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Mirko Tietgen  changed:

   What|Removed |Added

 CC||mi...@abunchofthings.net
 Status|Needs Signoff   |Failed QA

--- Comment #5 from Mirko Tietgen  ---
No test plan available.

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Marcel de Rooy  changed:

   What|Removed |Added

 CC||m.de.r...@rijksmuseum.nl

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Zeno Tajoli  changed:

   What|Removed |Added

 CC||z.taj...@cineca.it
   Patch complexity|--- |Small patch

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Alex Arnaud alex.arn...@biblibre.com changed:

   What|Removed |Added

   Assignee|julian.maur...@biblibre.com |alex.arn...@biblibre.com

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Alex Arnaud alex.arn...@biblibre.com changed:

   What|Removed |Added

 CC||alex.arn...@biblibre.com
 Status|Failed QA   |Needs Signoff

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

--- Comment #3 from Alex Arnaud alex.arn...@biblibre.com ---
Created attachment 40068
  --
http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=40068action=edit
[Follow up] Bug 13706 - Fix QA failure

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Alex Arnaud alex.arn...@biblibre.com changed:

   What|Removed |Added

  Attachment #40068|0   |1
is obsolete||

--- Comment #4 from Alex Arnaud alex.arn...@biblibre.com ---
Created attachment 40069
  --
http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=40069action=edit
[Follow up] Bug 13706 - Fix QA failure

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Bernardo Gonzalez Kriegel bgkrie...@gmail.com changed:

   What|Removed |Added

 CC||bgkrie...@gmail.com
 Status|Needs Signoff   |Failed QA

--- Comment #2 from Bernardo Gonzalez Kriegel bgkrie...@gmail.com ---
On top of 13705.

1) No license
2) koha-qa complais

 FAIL   misc/migration_tools/dedup_authorities.pl
   FAIL   pod
*** ERROR: Apparent command =cut not preceded by blank line  in
file misc/migration_tools/dedup_authorities.pl
*** WARNING: =head2 without preceding higher level  in file
misc/migration_tools/dedup_authorities.pl
   OK forbidden patterns
   FAIL   critic
Variable declared in conditional statement at line 429, column
5. Declare variables outside of the condition.
Variable declared in conditional statement at line 435, column
5. Declare variables outside of the condition.
   FAIL   valid
my variable $string masks earlier declaration in same scope 
given is experimental 
when is experimental

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Julian Maurice julian.maur...@biblibre.com changed:

   What|Removed |Added

 Depends on||11700

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Julian Maurice julian.maur...@biblibre.com changed:

   What|Removed |Added

 Status|ASSIGNED|Needs Signoff

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

--- Comment #1 from Julian Maurice julian.maur...@biblibre.com ---
Created attachment 35885
  --
http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=35885action=edit
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query, depending on --match parameter(s)
  2.2) run the query, retrieve the results
  2.3) for each result check if result is really a duplicate. check is
  customizable with option --check.
  2.4) among real duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) index updated biblios and authorities, and remove from index
 removed authorities

Use --help for more informations on options

Script is in misc/migration_tools/

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[Koha-bugs] [Bug 13706] Deduping authorities script (dedup_authorities.pl)

[bugzilla-daemon]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13706

Julian Maurice julian.maur...@biblibre.com changed:

   What|Removed |Added

 Depends on||13705

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/