[dspace-tech] Re: create postgresSQL 'dspace' database error

[Evelthon Prodromou]

Hello,

Did you switch to the postgres user to create the DB? If yes, did you use 
"su postgres" or "su - postgres" ? If you used the first, and you were 
logged in as root, you are probably still in the root account.

Make sure you are in the right user account by using 


*whoami.*
Evelthon


On Wednesday, August 10, 2016 at 5:06:17 AM UTC+3, Nor Hazira Mat Noh (HQ) 
wrote:
>
> i have problem while creating the database...
>
> Error that appear:
> could not change directory to "/root": Permission denied
> createdb: database creation failed: ERROR:  new encoding (UTF8) is 
> incompatible with the encoding of the template database (SQL_ASCII)
> HINT:  Use the same encoding as in the template database, or use template0 
> as template.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] New installation of DSpace 5.5 with migrated data rendering pages after ~40 seconds

[Andrea Bollini]

Dear Alan,

on DSpace-CRIS we have make the indexing process multi-thread and for 
our experience this improve the performance a lot, 10x or more depending 
on the number of threads used and the server configuration.

See

https://github.com/4Science/DSpace/commit/6206ca6f7980cdae31d5bd69c450d706f8518dfb

https://github.com/4Science/DSpace/blob/dspace-5_x_x-cris/dspace-api/src/main/java/org/dspace/discovery/SolrServiceImpl.java#L476

https://github.com/4Science/DSpace/blob/dspace-5_x_x-cris/dspace-api/src/main/java/org/dspace/discovery/SolrServiceImpl.java#L2585


Running SOLR on a separate tomcat, or better on a dedicated server bring 
also big improvements. When you use multiple threads you need to be sure 
to have enough database connections to serve all the threads and the 
running webapps.

If you are available to test this improvement on a plan DSpace 5.x, we 
can prepare a pull request (against DSpace 5.x) and if the feedback is 
good we can also port it to DSpace 6.x

BTW, the DSpace-CRIS enhancement also introduce the ability to (re)index 
a single object so can also replace the pull request
https://github.com/DSpace/DSpace/pull/1469

Best,
Andrea

Il 12/08/2016 15:33, Alan Orth ha scritto:
> Evelthon,
>
> Interesting observation about the indexing speed. Just yesterday I
> posted a message about Java JVM settings for Solr/Lucene to this
> mailing list. I'm sure there is room for improvements in Solr
> performance if you're willing to monitor, tweak, monitor, tweak, etc.
> I've stayed away from JVM tuning for the most part. Here's the link I
> posted yesterday, from a Solr developer, where he recommends some JVM
> settings as well as Java 8 (which wasn't the case when I checked this
> wiki last year):
>
> https://wiki.apache.org/solr/ShawnHeisey
>
> For what it's worth, our indexing takes ~60 minutes for 55,000 items,
> and we're on a Linode VPS where we have an SSD and plenty of CPU cores
> and memory — I hate to think how long it takes on less performant
> hardware.
>
> Regards,
>
>
> On Fri, Aug 12, 2016 at 10:56 AM, Evelthon Prodromou
>  wrote:
>> Hello Alan,
>>
>> Basically to finish the initial indexing, media-filter faster. Probably an
>> overkill.
>>
>>
>> Thanks.
>>
>> On Friday, August 12, 2016 at 10:20:49 AM UTC+3, Alan Orth wrote:
>>> I'm glad you solved it, Evelthon.
>>>
>>> I guess it depends on your OS and how you have Tomcat running. In
>>> Ubuntu we set JAVA_OPTS in /etc/default/tomcat7, but CentOS's Tomcat
>>> is surely different. By the way, there's more discussion about tuning
>>> DSpace (including JAVA_OPTS and CATALINA_OPTS) on the wiki:
>>>
>>> https://wiki.duraspace.org/display/DSDOC5x/Performance+Tuning+DSpace
>>>
>>> I still wonder why your JVM settings are so highly tweaked. Most
>>> people don't need to adjust those, and unless you really know you need
>>> them, I'd say to leave them off. Remember, "premature optimization is
>>> the root of all evil" ;)
>>>
>>> Cheers,
>>>
>>> On Fri, Aug 12, 2016 at 9:59 AM, Evelthon Prodromou
>>>  wrote:
 Hello Luigi,

 CATALINA_OPTS did it. All works good now. Seems like solr was choaking
 and
 caused slow loading on the UI.

 Curious though, shouldn't it fallback to java_opts?


 In any case, thank you.

 Evelthon


 On Thursday, August 11, 2016 at 1:01:34 PM UTC+3, Luigi Andrea
 Pascarelli
 wrote:
> Hi,
>
> as far as I know DSpace doesn't need a big amount of memory. For the
> first
> step you can try to setup CATALINA_OPTS to
>
> CATALINA_OPTS: -Xms1024m -Xmx2048m -XX:MaxPermSize=256m
> -Dfile.encoding=UTF-8
>
> And as Alan highlighted you could use JAVA_OPTS with less memory.
>
> Second step you could try to check if the I/O is the real issue. Maybe
> POSTGRES and SOLR and TOMCAT write and read on the same node and so you
> can
> benefits from separate them (or one of them) on differents volumes.
>
> Let me know.
>
> Regards,
>
> Luigi Andrea
>
>
> Il 11/08/2016 11:46, Evelthon Prodromou ha scritto:
>
> The server has 32GB, postgresql on different box. I don't think it's a
> RAM
> issue.
>
> I am wondering if it is a solr issue. I don't have tomcat running as
> user
> dspace. Instead, I change ownership in [dspace]/solr to dspace:tomcat
> and
> gave rw rights to both user and group.
>
> Thanks
>
>
>
> On Thursday, August 11, 2016 at 12:33:05 PM UTC+3, Alan Orth wrote:
>> Hi,
>>
>>  From your JAVA_OPTS I see you are allocating 4096 + 2048 megabytes of
>> RAM to Tomcat right from the start. How much memory does your server
>> have? This means your host must have AT LEAST 6GB of RAM just for
>> Tomcat, let alone PostgreSQL, Solr, and the rest of the operating
>> system. I wouldn't be surprised if you are encountering poor

Re: [dspace-tech] New installation of DSpace 5.5 with migrated data rendering pages after ~40 seconds

[Alan Orth]

Evelthon,

Interesting observation about the indexing speed. Just yesterday I
posted a message about Java JVM settings for Solr/Lucene to this
mailing list. I'm sure there is room for improvements in Solr
performance if you're willing to monitor, tweak, monitor, tweak, etc.
I've stayed away from JVM tuning for the most part. Here's the link I
posted yesterday, from a Solr developer, where he recommends some JVM
settings as well as Java 8 (which wasn't the case when I checked this
wiki last year):

https://wiki.apache.org/solr/ShawnHeisey

For what it's worth, our indexing takes ~60 minutes for 55,000 items,
and we're on a Linode VPS where we have an SSD and plenty of CPU cores
and memory — I hate to think how long it takes on less performant
hardware.

Regards,


On Fri, Aug 12, 2016 at 10:56 AM, Evelthon Prodromou
 wrote:
> Hello Alan,
>
> Basically to finish the initial indexing, media-filter faster. Probably an
> overkill.
>
>
> Thanks.
>
> On Friday, August 12, 2016 at 10:20:49 AM UTC+3, Alan Orth wrote:
>>
>> I'm glad you solved it, Evelthon.
>>
>> I guess it depends on your OS and how you have Tomcat running. In
>> Ubuntu we set JAVA_OPTS in /etc/default/tomcat7, but CentOS's Tomcat
>> is surely different. By the way, there's more discussion about tuning
>> DSpace (including JAVA_OPTS and CATALINA_OPTS) on the wiki:
>>
>> https://wiki.duraspace.org/display/DSDOC5x/Performance+Tuning+DSpace
>>
>> I still wonder why your JVM settings are so highly tweaked. Most
>> people don't need to adjust those, and unless you really know you need
>> them, I'd say to leave them off. Remember, "premature optimization is
>> the root of all evil" ;)
>>
>> Cheers,
>>
>> On Fri, Aug 12, 2016 at 9:59 AM, Evelthon Prodromou
>>  wrote:
>> > Hello Luigi,
>> >
>> > CATALINA_OPTS did it. All works good now. Seems like solr was choaking
>> > and
>> > caused slow loading on the UI.
>> >
>> > Curious though, shouldn't it fallback to java_opts?
>> >
>> >
>> > In any case, thank you.
>> >
>> > Evelthon
>> >
>> >
>> > On Thursday, August 11, 2016 at 1:01:34 PM UTC+3, Luigi Andrea
>> > Pascarelli
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> as far as I know DSpace doesn't need a big amount of memory. For the
>> >> first
>> >> step you can try to setup CATALINA_OPTS to
>> >>
>> >> CATALINA_OPTS: -Xms1024m -Xmx2048m -XX:MaxPermSize=256m
>> >> -Dfile.encoding=UTF-8
>> >>
>> >> And as Alan highlighted you could use JAVA_OPTS with less memory.
>> >>
>> >> Second step you could try to check if the I/O is the real issue. Maybe
>> >> POSTGRES and SOLR and TOMCAT write and read on the same node and so you
>> >> can
>> >> benefits from separate them (or one of them) on differents volumes.
>> >>
>> >> Let me know.
>> >>
>> >> Regards,
>> >>
>> >> Luigi Andrea
>> >>
>> >>
>> >> Il 11/08/2016 11:46, Evelthon Prodromou ha scritto:
>> >>
>> >> The server has 32GB, postgresql on different box. I don't think it's a
>> >> RAM
>> >> issue.
>> >>
>> >> I am wondering if it is a solr issue. I don't have tomcat running as
>> >> user
>> >> dspace. Instead, I change ownership in [dspace]/solr to dspace:tomcat
>> >> and
>> >> gave rw rights to both user and group.
>> >>
>> >> Thanks
>> >>
>> >>
>> >>
>> >> On Thursday, August 11, 2016 at 12:33:05 PM UTC+3, Alan Orth wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> From your JAVA_OPTS I see you are allocating 4096 + 2048 megabytes of
>> >>> RAM to Tomcat right from the start. How much memory does your server
>> >>> have? This means your host must have AT LEAST 6GB of RAM just for
>> >>> Tomcat, let alone PostgreSQL, Solr, and the rest of the operating
>> >>> system. I wouldn't be surprised if you are encountering poor
>> >>> performance due to swapping.
>> >>>
>> >>> For reference, we run a fairly large DSpace instance with ~55,000
>> >>> items and a decent amount of traffic and these are our JAVA_OPTS:
>> >>>
>> >>> -Djava.awt.headless=true -Xms3072m -Xmx3072m -XX:MaxPermSize=256m
>> >>> -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8
>> >>>
>> >>> Our server has 8GB of physical memory. Unless you know you need all
>> >>> those JVM tweaks, I'd start by simplifying your JAVA_OPTS to something
>> >>> more simple (for testing at least).
>> >>>
>> >>> Regards,
>> >>>
>> >>> On Wed, Aug 10, 2016 at 7:03 PM, Evelthon Prodromou
>> >>>  wrote:
>> >>> > Hello,
>> >>> >
>> >>> >
>> >>> >  I seem to be having an issue with tomcat. It takes ~38+ seconds to
>> >>> > load
>> >>> > pages. I believe it's tomcat since i notice shell scripts (
>> >>> > [dspace]/bin/dspace ) executing slow when tomcat is started, and
>> >>> > very
>> >>> > fast
>> >>> > (normal I presume) when tomcat is stopped.
>> >>> >
>> >>> >
>> >>> > The system  is a new installation of DSpace 5.5 on CentOS7. Data and
>> >>> > sql
>> >>> > migrated from an 1.7.0 installation.
>> >>> >
>> >>> > tomcat.conf includes the following JAVA_OPTS
>> >>> >
>> >>> > JAVA_OPTS="-Xmx4096m -Xms4096m 

Re: [dspace-tech] New installation of DSpace 5.5 with migrated data rendering pages after ~40 seconds

[Alan Orth]

I'm glad you solved it, Evelthon.

I guess it depends on your OS and how you have Tomcat running. In
Ubuntu we set JAVA_OPTS in /etc/default/tomcat7, but CentOS's Tomcat
is surely different. By the way, there's more discussion about tuning
DSpace (including JAVA_OPTS and CATALINA_OPTS) on the wiki:

https://wiki.duraspace.org/display/DSDOC5x/Performance+Tuning+DSpace

I still wonder why your JVM settings are so highly tweaked. Most
people don't need to adjust those, and unless you really know you need
them, I'd say to leave them off. Remember, "premature optimization is
the root of all evil" ;)

Cheers,

On Fri, Aug 12, 2016 at 9:59 AM, Evelthon Prodromou
 wrote:
> Hello Luigi,
>
> CATALINA_OPTS did it. All works good now. Seems like solr was choaking and
> caused slow loading on the UI.
>
> Curious though, shouldn't it fallback to java_opts?
>
>
> In any case, thank you.
>
> Evelthon
>
>
> On Thursday, August 11, 2016 at 1:01:34 PM UTC+3, Luigi Andrea Pascarelli
> wrote:
>>
>> Hi,
>>
>> as far as I know DSpace doesn't need a big amount of memory. For the first
>> step you can try to setup CATALINA_OPTS to
>>
>> CATALINA_OPTS: -Xms1024m -Xmx2048m -XX:MaxPermSize=256m
>> -Dfile.encoding=UTF-8
>>
>> And as Alan highlighted you could use JAVA_OPTS with less memory.
>>
>> Second step you could try to check if the I/O is the real issue. Maybe
>> POSTGRES and SOLR and TOMCAT write and read on the same node and so you can
>> benefits from separate them (or one of them) on differents volumes.
>>
>> Let me know.
>>
>> Regards,
>>
>> Luigi Andrea
>>
>>
>> Il 11/08/2016 11:46, Evelthon Prodromou ha scritto:
>>
>> The server has 32GB, postgresql on different box. I don't think it's a RAM
>> issue.
>>
>> I am wondering if it is a solr issue. I don't have tomcat running as user
>> dspace. Instead, I change ownership in [dspace]/solr to dspace:tomcat and
>> gave rw rights to both user and group.
>>
>> Thanks
>>
>>
>>
>> On Thursday, August 11, 2016 at 12:33:05 PM UTC+3, Alan Orth wrote:
>>>
>>> Hi,
>>>
>>> From your JAVA_OPTS I see you are allocating 4096 + 2048 megabytes of
>>> RAM to Tomcat right from the start. How much memory does your server
>>> have? This means your host must have AT LEAST 6GB of RAM just for
>>> Tomcat, let alone PostgreSQL, Solr, and the rest of the operating
>>> system. I wouldn't be surprised if you are encountering poor
>>> performance due to swapping.
>>>
>>> For reference, we run a fairly large DSpace instance with ~55,000
>>> items and a decent amount of traffic and these are our JAVA_OPTS:
>>>
>>> -Djava.awt.headless=true -Xms3072m -Xmx3072m -XX:MaxPermSize=256m
>>> -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8
>>>
>>> Our server has 8GB of physical memory. Unless you know you need all
>>> those JVM tweaks, I'd start by simplifying your JAVA_OPTS to something
>>> more simple (for testing at least).
>>>
>>> Regards,
>>>
>>> On Wed, Aug 10, 2016 at 7:03 PM, Evelthon Prodromou
>>>  wrote:
>>> > Hello,
>>> >
>>> >
>>> >  I seem to be having an issue with tomcat. It takes ~38+ seconds to
>>> > load
>>> > pages. I believe it's tomcat since i notice shell scripts (
>>> > [dspace]/bin/dspace ) executing slow when tomcat is started, and very
>>> > fast
>>> > (normal I presume) when tomcat is stopped.
>>> >
>>> >
>>> > The system  is a new installation of DSpace 5.5 on CentOS7. Data and
>>> > sql
>>> > migrated from an 1.7.0 installation.
>>> >
>>> > tomcat.conf includes the following JAVA_OPTS
>>> >
>>> > JAVA_OPTS="-Xmx4096m -Xms4096m -XX:MaxPermSize=2048m
>>> > -Dfile.encoding=UTF-8
>>> > -XX:MaxHeapFreeRatio=70 -XX:+UseConcMarkSweepGC
>>> > -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled"
>>> >
>>> > index-discovery was executed and discovery facets show up.
>>> >
>>> >
>>> > Someone please point me to the right direction to investigate.
>>> >
>>> >
>>> > Thank you,
>>> >
>>> >
>>> > Evelthon
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "DSpace Technical Support" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an
>>> > email to dspace-tech...@googlegroups.com.
>>> > To post to this group, send email to dspac...@googlegroups.com.
>>> > Visit this group at https://groups.google.com/group/dspace-tech.
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> Alan Orth
>>> alan...@gmail.com
>>> https://englishbulgaria.net
>>> https://alaninkenya.org
>>> https://mjanja.ch
>>> "In heaven all the interesting people are missing." ―Friedrich Nietzsche
>>> GPG public key ID: 0x8cb0d0acb5cd81ec209c6cdfbd1a0e09c2f836c0
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "DSpace Technical Support" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to dspace-tech...@googlegroups.com.
>> To post to this group, send email to 

Re: [dspace-tech] New installation of DSpace 5.5 with migrated data rendering pages after ~40 seconds

[Evelthon Prodromou]

Hello Luigi,

CATALINA_OPTS did it. All works good now. Seems like solr was choaking and 
caused slow loading on the UI.

Curious though, shouldn't it fallback to java_opts?


In any case, thank you.

Evelthon

On Thursday, August 11, 2016 at 1:01:34 PM UTC+3, Luigi Andrea Pascarelli 
wrote:
>
> Hi,
>
> as far as I know DSpace doesn't need a big amount of memory. For the first 
> step you can try to setup CATALINA_OPTS to
>
> CATALINA_OPTS: -Xms1024m -Xmx2048m -XX:MaxPermSize=256m 
> -Dfile.encoding=UTF-8
>
> And as Alan highlighted you could use JAVA_OPTS with less memory. 
>
> Second step you could try to check if the I/O is the real issue. Maybe 
> POSTGRES and SOLR and TOMCAT write and read on the same node and so you can 
> benefits from separate them (or one of them) on differents volumes.
>
> Let me know.
>
> Regards,
>
> Luigi Andrea
>
>
> Il 11/08/2016 11:46, Evelthon Prodromou ha scritto:
>
> The server has 32GB, postgresql on different box. I don't think it's a RAM 
> issue.
>
> I am wondering if it is a solr issue. I don't have tomcat running as user 
> dspace. Instead, I change ownership in [dspace]/solr to dspace:tomcat and 
> gave rw rights to both user and group.
>
> Thanks 
>
>
>
> On Thursday, August 11, 2016 at 12:33:05 PM UTC+3, Alan Orth wrote: 
>>
>> Hi, 
>>
>> From your JAVA_OPTS I see you are allocating 4096 + 2048 megabytes of 
>> RAM to Tomcat right from the start. How much memory does your server 
>> have? This means your host must have AT LEAST 6GB of RAM just for 
>> Tomcat, let alone PostgreSQL, Solr, and the rest of the operating 
>> system. I wouldn't be surprised if you are encountering poor 
>> performance due to swapping. 
>>
>> For reference, we run a fairly large DSpace instance with ~55,000 
>> items and a decent amount of traffic and these are our JAVA_OPTS: 
>>
>> -Djava.awt.headless=true -Xms3072m -Xmx3072m -XX:MaxPermSize=256m 
>> -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 
>>
>> Our server has 8GB of physical memory. Unless you know you need all 
>> those JVM tweaks, I'd start by simplifying your JAVA_OPTS to something 
>> more simple (for testing at least). 
>>
>> Regards, 
>>
>> On Wed, Aug 10, 2016 at 7:03 PM, Evelthon Prodromou 
>>  wrote: 
>> > Hello, 
>> > 
>> > 
>> >  I seem to be having an issue with tomcat. It takes ~38+ seconds to 
>> load 
>> > pages. I believe it's tomcat since i notice shell scripts ( 
>> > [dspace]/bin/dspace ) executing slow when tomcat is started, and very 
>> fast 
>> > (normal I presume) when tomcat is stopped. 
>> > 
>> > 
>> > The system  is a new installation of DSpace 5.5 on CentOS7. Data and 
>> sql 
>> > migrated from an 1.7.0 installation. 
>> > 
>> > tomcat.conf includes the following JAVA_OPTS 
>> > 
>> > JAVA_OPTS="-Xmx4096m -Xms4096m -XX:MaxPermSize=2048m 
>> -Dfile.encoding=UTF-8 
>> > -XX:MaxHeapFreeRatio=70 -XX:+UseConcMarkSweepGC 
>> > -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled" 
>> > 
>> > index-discovery was executed and discovery facets show up. 
>> > 
>> > 
>> > Someone please point me to the right direction to investigate. 
>> > 
>> > 
>> > Thank you, 
>> > 
>> > 
>> > Evelthon 
>> > 
>> > -- 
>> > You received this message because you are subscribed to the Google 
>> Groups 
>> > "DSpace Technical Support" group. 
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> an 
>> > email to dspace-tech...@googlegroups.com. 
>> > To post to this group, send email to dspac...@googlegroups.com. 
>> > Visit this group at https://groups.google.com/group/dspace-tech. 
>> > For more options, visit https://groups.google.com/d/optout. 
>>
>>
>>
>> -- 
>> Alan Orth 
>> alan...@gmail.com 
>> https://englishbulgaria.net 
>> https://alaninkenya.org 
>> https://mjanja.ch 
>> "In heaven all the interesting people are missing." ―Friedrich Nietzsche 
>> GPG public key ID: 0x8cb0d0acb5cd81ec209c6cdfbd1a0e09c2f836c0 
>>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to dspace-tech...@googlegroups.com .
> To post to this group, send email to dspac...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/dspace-tech.
> For more options, visit https://groups.google.com/d/optout.
>
>
> -- 
> Luigi Andrea Pascarelli
>
> DSpace Committer and DSpace-CRIS Lead Developer
>
> 4Science,  www.4science.it (an Itway Group Company)
>
> office: Via Edoardo D'Onofrio 304, 00155 Roma, Italy
> tel: +39 333 934 1782
> skype: l_a_p82
> linkedin: luigiandreapascarelli
>
> 
> Salvate un albero. Non stampate questa mail se non necessario.
> Save a tree. Don't print this e-mail unless it's really necessary.
>
> DISCLAIMER: Le informazioni contenute in questo messaggio sono confidenziali, 
> possono essere protette da leggi locali, 
> e devono essere utilizzate esclusivamente