Re: [dspace-tech] Re: Notes on PostgreSQL connection pooling with a Tomcat JNDI resource

[Alan Orth]

Is there some kind of database transaction leaking issue with DSpace,
particularly the XMLUI? Basically, I find that it doesn't matter what I my
pool size is. Eventually it always gets full. Sometimes it takes a few
days, sometimes it happens a few times in one day. Today I saw this in
dspace.log.2018-02-07:

org.apache.tomcat.jdbc.pool.PoolExhaustedException:
[http-bio-127.0.0.1-8443-exec-328] Timeout: Pool empty. Unable to fetch a
connection in 5 seconds, none available[size:250; busy:250; idle:0;
lastwait:5000].

The pool size is 250, and yet PostgreSQL activity shows literally all of
the 250 connections as either idle or idle in transaction:

$ psql -c 'select * from pg_stat_activity' | grep -c "PostgreSQL JDBC"
250
$ psql -c 'select * from pg_stat_activity' | grep "PostgreSQL JDBC" | grep
-c idle
250
$ psql -c 'select * from pg_stat_activity' | grep "PostgreSQL JDBC" | grep
-c "idle in transaction"
187

What is going on? I'm going crazy restarting the server and staring at log
files trying to figure out what is going on. We're using DSpace 5.5 with
PostgreSQL 9.5.10 and the latest PostgreSQL JDBC driver from
jdbc.postgresql.org (42.2.1) on fast hardware with plenty of memory and
CPU. But DSpace. Always. Crashes. Every. Day. It seems DS-3551 might help,
but we can't move to DSpace 5.7 yet, and our 5.5 branch doesn't build if I
cherry-pick the commits for DS-3551[0].

[0] https://jira.duraspace.org/browse/DS-3551

Let's get to the bottom of this,

On Thu, Feb 1, 2018 at 3:43 PM Mark H. Wood  wrote:

> If you have stuck connections, you may want to try some of the pool
> provider's connection testing parameters.  I've had good results using
> 'validationQuery="SELECT 1"' with 'testOnBorrow="true"'.
>
> Also I've found that older PostgreSQL drivers seem to see this problem
> more frequently, so I try to keep Tomcat updated with the latest released
> DBMS drivers.
>
> --
> You received this message because you are subscribed to the Google Groups
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dspace-tech+unsubscr...@googlegroups.com.
> To post to this group, send email to dspace-tech@googlegroups.com.
> Visit this group at https://groups.google.com/group/dspace-tech.
> For more options, visit https://groups.google.com/d/optout.
>
-- 

Alan Orth
alan.o...@gmail.com
https://picturingjordan.com
https://englishbulgaria.net
https://mjanja.ch

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] Re: SECURITY ALERT: ImageMagick vulnerability may affect DSpace 5.x sites that use ImageMagick Media Filters

[Tim Donohue]

Hello Yanan,

According to the bug report for that ImageMagick security issue, it looks
like this issue has been fixed

 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-3714

On the page above, I see a notice that says it only affected "ImageMagick
before 6.9.3-10 and 7.x before 7.0.1-1"

So, it sounds like it is fixed as long as you have a more recent version of
ImageMagick running.

- Tim

On Tue, Feb 6, 2018 at 9:11 PM Yanan Z  wrote:

> Kia ora,
>
> At Lincoln University (NZ), we are planning to install ImageMagick
> Thumbnails for our dspace instance. We are currently on Dspace v5.6. If we
> install the latest version of ImageMagick ie.,
> ImageMagick-7.0.7-22-Q16-x64
> https://www.imagemagick.org/script/download.php, does anyone know if we
> still need to be concerned about this vulnerability?
> https://www.imagemagick.org/discourse-server/viewtopic.php?f=4=29588
>
> Ngā mihi nui (Many thanks in advance),
> Yanan
>
> *Yanan Zhao*
>
> *Digital Services Analyst*
>
>
>
> *Library, Teaching and Learning, Te Wharepūrākau*
>
> *P O Box 85064*
>
> *Lincoln University*
>
> *Lincoln 7647*
>
> *Canterbury*
>
> *New Zealand*
>
>
>
> *p* +64 3 423 0340 <+64%203-423%200340>
>
> *e* *yanan.z...@lincoln.ac.nz * | *w*
> ltl.lincoln.ac.nz
>
>
> On Saturday, May 14, 2016 at 2:23:02 AM UTC+12, Tim Donohue wrote:
>
>> Hi,
>>
>> This vulnerability appears in ImageMagick and doesn't actually appear
>> anywhere in the DSpace code itself. However, if you are using the
>> ImageMagick Thumbnails, then you would be affected by these
>> vulnerabilities. This is because you will have had to install ImageMagick
>> on your server in order to use the Thumbnail creation tools:
>>
>> https://wiki.duraspace.org/display/DSDOC5x/ImageMagick+Media+Filters
>>
>> So, to answer your questions:
>>
>> * You only need to be concerned about this vulnerability if you actually
>> have *installed* ImageMagick (http://www.imagemagick.org/), as it's a
>> separate installation from DSpace and does NOT come bundled with DSpace.
>>
>> * There's no need to remove the ImageMagick configuration lines from your
>> configuration file. They won't be used unless they are uncommented and
>> ImageMagick is installed.
>> - Tim
>>
>> On 5/10/2016 9:27 AM, Feed My Lambs Esq. wrote:
>>
> Thanks for the announcement of this vulnerability, Tim.
>>
>> I found the plugin addition in dspace.cfg
>> under plugin.named.org.dspace.app.mediafilter.FormatFilter = ...
>>   org.dspace.app.mediafilter.ImageMagickImageThumbnailFilter =
>> ImageMagick Image Thumbnail, \
>>   org.dspace.app.mediafilter.ImageMagickPdfThumbnailFilter = ImageMagick
>> PDF Thumbnail
>>
>> but this line is still commented out:
>> # org.dspace.app.mediafilter.ImageMagickThumbnailFilter.ProcessStarter =
>> /usr/bin
>> (which is how I found it in our Windows server)
>>
>> I'm assuming that means we aren't using this plugin (and therefore not
>> vulnerable).
>>
>> I also tried to find the software installed in our Windows "Program
>> Files" directories but didn't see it.
>>
>> I realize I may be overthinking things but just wanted to make sure.
>> Thank you for confirming!
>>
>> Lastly, should I delete / comment out the ImageMagick lines under the
>> FormatFilter I mentioned above? Thanks
>> --
>> You received this message because you are subscribed to the Google Groups
>> "DSpace Technical Support" group.
>>
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to dspace-tech...@googlegroups.com.
>> To post to this group, send email to dspac...@googlegroups.com.
>>
>>
>> Visit this group at https://groups.google.com/group/dspace-tech.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> Tim Donohue
>> Technical Lead for DSpace & DSpaceDirect
>> DuraSpace.org | DSpace.org | DSpaceDirect.org
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dspace-tech+unsubscr...@googlegroups.com.
> To post to this group, send email to dspace-tech@googlegroups.com.
> Visit this group at https://groups.google.com/group/dspace-tech.
> For more options, visit https://groups.google.com/d/optout.
>
-- 
Tim Donohue
Technical Lead for DSpace & DSpaceDirect
DuraSpace.org | DSpace.org | DSpaceDirect.org

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] find number of items submitted in year 2016

[Terry Brady]

If you have the DSpace 6 REST API enabled, you can run a report to retrieve
this information.

Here is an example on demo.dspace.org pulling items from 2016, 2017, and
2018.

https://demo.dspace.org/rest/static/reports/query.html?query_field[]=dc.date.accessioned_op[]=matches_val[]=%5E201%5B678%5D.*[]==100=0=parentCollection%2Cmetadata=none_fields[]=dc.date.accessioned

On Wed, Feb 7, 2018 at 5:49 AM, Lewatle Johannes Phaladi  wrote:

> Dear All,
>
> I am using DSpace 6.2 on both xmlui and jspui interface, I would like to
> know how can I see total number of items submitted in the year 2016 and
> 2017.
>
> When I check *dc. date. accessioned metadata *field might assist, I am
> using out of box DSpace interface which doesn't have   *dc. date.
> accessioned  *under browse, how can I see number of items submitted as
> per the year the item was submitted, using any method that will help.
>
> Regards,
> Lewatle
>
> --
> You received this message because you are subscribed to the Google Groups
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dspace-tech+unsubscr...@googlegroups.com.
> To post to this group, send email to dspace-tech@googlegroups.com.
> Visit this group at https://groups.google.com/group/dspace-tech.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Terry Brady
Applications Programmer Analyst
Georgetown University Library Information Technology
https://github.com/terrywbrady/info
425-298-5498 (Seattle, WA)

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

[dspace-tech] Re: DSpace Developer Show and Tell Meetings

[Terry Brady]

Based on the folks who expressed interest in the Developer Show and Tell
Meeting, I have created a Doodle Poll to find the best date and time for
our initial meeting.

If you are interested in attending, please indicate the times that would
work for you.  The agenda and Doodle Poll link are available here:

https://wiki.duraspace.org/display/DSPACE/DSpace+Developer+Show+and+Tell+Meetings#DSpaceDeveloperShowandTellMeetings-NextMeeting

Thank you also for the proposed agenda items.  I hope to incorporate all of
the suggestions in subsequent meetings.

Terry

On Wed, Jan 31, 2018 at 12:58 PM, Terry Brady 
wrote:

> I would like to schedule a couple of developer show and tell meetings for
> DSpace developers to share knowledge about development tools, testing
> frameworks and deployment environments.
>
> Details are posted below.  Please  add your name to the page if this would
> be of interest to you.  Also, please note the topics that you need to know
> more about to support your software development efforts.
>
> https://wiki.duraspace.org/display/DSPACE/DSpace+Developer+Show+and+Tell+
> Meetings
>
> Terry
> --
> Terry Brady
> Applications Programmer Analyst
> Georgetown University Library Information Technology
> https://github.com/terrywbrady/info
> 425-298-5498 <(425)%20298-5498> (Seattle, WA)
>



-- 
Terry Brady
Applications Programmer Analyst
Georgetown University Library Information Technology
https://github.com/terrywbrady/info
425-298-5498 (Seattle, WA)

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] find number of items submitted in year 2016

[Claudia Jürgen]

Hello Lewatle,

you may query the database, this is our query for accessioned (5.4) but should 
work:
SELECT to_char(date_trunc('year', t1.ts), '') AS year, count(*) FROM 
(SELECT to_timestamp(text_value, '-MM-DD') AS ts FROM metadatavalue WHERE 
metadata_field_id = 12) t1 GROUP BY date_trunc('year', t1.ts) order by year;

Just replace the metadata_field_id with your correct id.

Hope this helps

Claudia Jürgen

Am 07.02.2018 um 14:49 schrieb Lewatle Johannes Phaladi:

Dear All,

I am using DSpace 6.2 on both xmlui and jspui interface, I would like to
know how can I see total number of items submitted in the year 2016 and
2017.

When I check *dc. date. accessioned metadata *field might assist, I am
using out of box DSpace interface which doesn't have   *dc. date.
accessioned  *under browse, how can I see number of items submitted as per
the year the item was submitted, using any method that will help.

Regards,
Lewatle




--
Claudia Juergen
Eldorado

Technische Universität Dortmund
Universitätsbibliothek
Vogelpothsweg 76
44227 Dortmund

Tel.: +49 231-755 40 43
Fax: +49 231-755 40 32
claudia.juer...@tu-dortmund.de
www.ub.tu-dortmund.de

Wichtiger Hinweis: Die Information in dieser E-Mail ist vertraulich. Sie ist 
ausschließlich für den Adressaten bestimmt. Sollten Sie nicht der für diese 
E-Mail bestimmte Adressat sein, unterrichten Sie bitte den Absender und 
vernichten Sie diese Mail. Vielen Dank.
Unbeschadet der Korrespondenz per E-Mail, sind unsere Erklärungen 
ausschließlich final rechtsverbindlich, wenn sie in herkömmlicher Schriftform 
(mit eigenhändiger Unterschrift) oder durch Übermittlung eines solchen 
Schriftstücks per Telefax erfolgen.

Important note: The information included in this e-mail is confidential. It is 
solely intended for the recipient. If you are not the intended recipient of 
this e-mail please contact the sender and delete this message. Thank you. 
Without prejudice of e-mail correspondence, our statements are only legally 
binding when they are made in the conventional written form (with personal 
signature) or when such documents are sent by fax.

--
You received this message because you are subscribed to the Google Groups "DSpace 
Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

[dspace-tech] find number of items submitted in year 2016

[Lewatle Johannes Phaladi]

Dear All,

I am using DSpace 6.2 on both xmlui and jspui interface, I would like to 
know how can I see total number of items submitted in the year 2016 and 
2017.

When I check *dc. date. accessioned metadata *field might assist, I am 
using out of box DSpace interface which doesn't have   *dc. date. 
accessioned  *under browse, how can I see number of items submitted as per 
the year the item was submitted, using any method that will help.

Regards,
Lewatle  

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] Updating 5.6 to 6.2 - two questions

[Evgeni Dimitrov]

Yes, I repeated everything from the beginning and all worked fine.

Best regards
Evgeni

On Tuesday, February 6, 2018 at 11:59:17 PM UTC+2, Tim Donohue wrote:
>
> Hi Evgeni,
>
> Yes, that section of the documentation is actually asking you to run the 
> v5.6 "update-sequences.sql" against your 5.6-compatible database before 
> upgrading it. It's not required, but sometimes if your database sequences 
> get out of "sync", then the database upgrade could fail.
>
> Obviously, as the database changes in new releases, sometimes the 
> "update-sequences.sql" script needs to also change.  So, the issue here 
> seems to be that you ran the v6.2 "update-sequences.sql" against a 5.6 
> compatible database, and there are changes in the v6.2 
> "update-sequences.sql" that are not compatible with 5.6.
>
> Hopefully that makes more sense,
>
> Tim
>
>
> On Tue, Feb 6, 2018 at 2:46 PM Evgeni Dimitrov  > wrote:
>
>> Thank you Tim,
>>
>> I followed the "Upgrade steps":
>> "In some scenarios, if your database's "sequences" are outdated, 
>> inconsistent or incorrect, a
>> database migration error may occur (in your DSpace logs). In order to 
>> AVOID this scenario, you
>> may wish to manually run the "update-sequences.sql" script PRIOR to 
>> upgrade."
>>
>> So, I did run "update-sequences.sql" AFTER setting 6.2 in [dspace-source] 
>> and running mvn and ant, but BEFORE the DB migration...
>>
>> Or may be this "PRIOR to upgrade" means to run the "update-sequences.sql" 
>> script of the 5.6 version before even starting the upgrade to 6.2?
>>
>> Best regards
>> Evgeni
>>
>>
>> On Tuesday, February 6, 2018 at 7:13:39 PM UTC+2, Tim Donohue wrote:
>>
>>> Hi Evgeni,
>>>
>>> It sounds to me like your DSpace 6.2 upgrade didn't fully succeed *OR* 
>>> you haven't yet run all the database migrations.  You might want to check 
>>> your DSpace logs (dspace.log) and/or the output of "[dspace]/bin/dspace 
>>> database status".  If migrations are listed as PENDING, that means they 
>>> haven't yet been run, and your database is not fully upgraded yet.  You can 
>>> manually run "./dspace database migrate" to run all Pending migrations (and 
>>> you should then watch the dspace.log file to see that they successfully are 
>>> applied). 
>>>
>>> 1) The "handle_id_seq" is created in one of the migrations that upgrade 
>>> your database to 6.x.  Specifically, this migration: 
>>> https://github.com/DSpace/DSpace/blob/dspace-6_x/dspace-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2016.07.26__DS-3277_fix_handle_assignment.sql#L15
>>>   
>>>  So, it sounds to me like this migration did not run yet, or failed in some 
>>> way.  Once this migration runs, the "update-sequences.sql" should work 
>>> again.
>>>
>>> 2) The registry updates should be triggered automatically (and you 
>>> should find them noted in your dspace.log file, if they were executed).  
>>> The database migrations and registry updates both should appear in your 
>>> dspace.log file.  If you have concerns though, you *can* manually run those 
>>> registry updates. Manually running them won't hurt anything, as it simply 
>>> checks if all the registry entries already exist...if they already do exist 
>>> in your database, then no changes will be made.
>>>
>>> In general, during a DSpace upgrade, you should keep a close eye on the 
>>> DSpace log files. That's where you'll see more information on whether a 
>>> migration failed, which migrations succeeded, and whether or not your 
>>> registry updates automatically.
>>>
>>> Good luck,
>>>
>>> Tim
>>>
>>> On Tue, Feb 6, 2018 at 9:12 AM Evgeni Dimitrov  
>>> wrote:
>>>
>> I updated a small test repository from 5.6 to 6.2. Quite smoothly and all 
 works in the end, but two questions appeared:

 1. Running update-sequences.sql there was an error message:

 psql -U dspace -f 
 D:\DSPACE.5.6.n\DSPACE\etc\postgres\update-sequences.sql dspace56n

 ...

  setval
 
 (1 row)

 psql:D:/DSPACE.5.6.n/DSPACE/etc/postgres/update-sequences.sql:63: 
 ERROR:  relation "handle_id_seq" does not exist
 LINE 1: SELECT setval('handle_id_seq', max(handle_id)) FROM handle;
   ^
  setval
 
  42
 (1 row)

 Is this an error in the script or an error in the database? Is not 
 update-sequences.sql supposed to fix errors, not to report errors?

 2. In the "Upgrade Steps" there is "The database migration should also 
 trigger your metadata/file registries to be automatically updated (based 
 on 
 the config files in [dspace]/config/registries/). However, if this update 
 was NOT triggered, you can also manually run these registry updates (they 
 will not harm existing registry contents)."

 How can I know was the update of "metadata/file registries" triggered 
 or not? What does registry-loader do? What if I run