Re: No space left on device - When I execute suggester component.

2017-12-19 Thread Jamie Jackson 
Have you checked disk consumption on the box yet?

On Dec 19, 2017 8:46 PM, "Fiz Newyorker"  wrote:

> Hello Solr Experts,
>
> Good Evening !
>
> I have an  issue , Whenever I run the Auto Suggest component ,
>
> http://lapp81.rno.com:8989/solr/Data/suggest?suggest=
> true=true=fuzzySuggester=json&
> suggest.q=Jerry
>
>
> I am getting the following Error .
>
> {"responseHeader":{"status":500,"QTime":341753},"error":{"msg":"No
> space left on device","trace":"java.io.IOException: No space left on
> device\n\tat sun.nio.ch.FileDispatcherImpl.write0(Native Method)\n\tat
> sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)\n\tat
> sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)\n\tat
> sun.nio.ch.IOUtil.write(IOUtil.java:65)\n\tat
> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)\n\tat
> java.nio.channels.Channels.writeFullyImpl(Channels.java:78)\n\tat
> java.nio.channels.Channels.writeFully(Channels.java:101)\n\tat
> java.nio.channels.Channels.access$000(Channels.java:61)\n\tat
> java.nio.channels.Channels$1.write(Channels.java:174)\n\tat
> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.
> write(FSDirectory.java:419)\n\tat
> java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)\n\tat
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)\
> n\tat
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)\n\tat
> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(
> OutputStreamIndexOutput.java:53)\n\tat
> org.apache.lucene.util.OfflineSorter$ByteSequencesWriter.write(
> OfflineSorter.java:529)\n\tat
> org.apache.lucene.search.suggest.analyzing.AnalyzingSuggester.build(
> AnalyzingSuggester.java:480)\n\tat
> org.apache.lucene.search.suggest.Lookup.build(Lookup.java:190)\n\tat
> org.apache.solr.spelling.suggest.SolrSuggester.build(
> SolrSuggester.java:181)\n\tat
> org.apache.solr.handler.component.SuggestComponent.
> prepare(SuggestComponent.java:179)\n\tat
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(
> SearchHandler.java:269)\n\tat
> org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:173)\n\tat
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)\n\tat
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)\n\tat
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)\n\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:347)\n\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:298)\n\tat
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1691)\n\tat
> org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:582)\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)\n\tat
> org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)\n\tat
> org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)\n\tat
> org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1180)\n\tat
> org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:512)\n\tat
> org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)\n\tat
> org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1112)\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)\n\tat
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:213)\n\tat
> org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)\n\tat
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)\n\tat
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
> RewriteHandler.java:335)\n\tat
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)\n\tat
> org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat
> org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)\n\tat
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)\n\tat
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)\n\tat
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:148)\n\tat
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> ExecuteProduceConsume.java:136)\n\tat
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:671)\n\tat
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>

Re: How to restart solr in docker?

2017-12-15 Thread Jamie Jackson 
The usual procedure with containers is to restart or recreate the container.

On Fri, Dec 15, 2017 at 11:55 AM, Buckler, Christine <
christine.buck...@nordstrom.com> wrote:

> What is the command for restarting solr on a docker image? I have modified
> the solrconfig.xml to add the suggest plugin and now I want to update to
> reflect this change. I was able to stop solr using “$ docker exec -it
> my_solr stop all” but now I can’t figure out how to restart.
>
> Thanks,
> Christine Buckler
>
>

Problem using user.timezone in DIH

2017-09-22 Thread Jamie Jackson 
Solr 6.6.0

I'm getting a weird issue and a misleading error when I try to use the
user.timezone property in a data import handler configuration which has
child entities:

Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
connect: Access denied for user '{}'@'111.222.111.222' (using password: YES)

I was trying this in a query for a delta import:

CONVERT_TZ('${dataimporter.last_index_time}','${user.timezone}','${
custom.dataimporter.datasource.tz}')

It works great for cores with no child entities in the DIH config.

However, if I have any child entities, it dies. Here's a failing
configuration and its stack trace:
https://gist.github.com/jamiejackson/1ec544e6b87607098fbb1077a4eece30

I isolated it down to the use of user.timezone. Even if I simply add the
following in a SQL _comment_, it causes the problem. For example:

select /* ${user.timezone} */ column_a, column_b, ...

Bug?

Thanks,
Jamie

Re: [bulk]: Re: Dates and DataImportHandler

2017-09-21 Thread Jamie Jackson 
That's what I ended up doing too, for my MariaDB (MySQL derivative) DB:

CONVERT_TZ('${dataimporter.last_index_time}','${user.timezone}','${
custom.dataimporter.datasource.tz}')

user.timezone is Solr's time zone and custom.dataimporter.datasource.tz is
a property I set on startup.

The other option was to change Solr's timezone to match the DB, but I
thought that was even more of a hack.

On Thu, Sep 21, 2017 at 7:08 AM, Mannott, Birgit <b.mann...@klopotek.de>
wrote:

> As far as I understood, you can use the locale so that DIH saves the last
> index time for  the given time zone and not for UTC. So if you set the
> locale according to the timezone of your DB you don't need to convert dates
> for comparison.
>
> But for me it's not working because every time I include settings for the
> propertyWriter (concrete data doesn't matter - I tried a lot),  the null
> pointer exception appears. So I had to find a workaround and convert now
> the time saved by the DIH from UTC to the timezone of my DB when comparing
> for delta imports. But it's an ugly workaround I don't like.
>
> So, if anyone has an idea why this NPE occurs it would be great. Do I
> perhaps have to add something to solrconfig.xml?
>
> Thanks,
> Birgit
>
>
>
> -Original Message-
> From: Jamie Jackson [mailto:jamieja...@gmail.com]
> Sent: Tuesday, September 19, 2017 6:54 PM
> To: solr-user@lucene.apache.org
> Subject: [bulk]: Re: [bulk]: Dates and DataImportHandler
>
> FWIW, I know mine worked, so maybe try:
>
>  type="SimplePropertiesWriter" />
>
> I can't conceive of what the locale would possibly do when a dateFormat is
> specified, so I omitted the attribute. (Maybe one can specify dateFormat
> *or *locale--it seems like specifying both would cause a clash.) For what
> it's worth, the format you're trying to write seems identical to the
> default*, so I'm not sure what benefit you're getting by using that
> propertyWriter.
>
> *It's identical to *my* default, anyway. Maybe the default changes based
> on one's system configuration, I don't know. This stuff isn't very well
> documented.
>
> On Tue, Sep 19, 2017 at 7:22 AM, Mannott, Birgit <b.mann...@klopotek.de>
> wrote:
>
> > Hi,
> >
> > I have a similar problem. I try to change the timezone for the
> > last_index_time by setting
> >
> >  > type="SimplePropertiesWriter" locale="en_US" />
> >
> > in the  section of my data-config.xml file.
> >
> > But when doing this I always get a NullPointerException on Delta Import:
> >
> > 2017-09-15 14:04:00.825 INFO  (Thread-2938) [   x:mex_prd_dev1100-ap]
> > o.a.s.h.d.DataImporter Starting Delta Import
> > 2017-09-15 14:04:00.827 ERROR (Thread-2938) [   x:mex_prd_dev1100-ap]
> > o.a.s.h.d.DataImporter Delta Import Failed
> > org.apache.solr.handler.dataimport.DataImportHandlerException: Unable
> > to PropertyWriter implementation:SimplePropertiesWriter
> > at org.apache.solr.handler.dataimport.DataImporter.
> > createPropertyWriter(DataImporter.java:330)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > doDeltaImport(DataImporter.java:439)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > runCmd(DataImporter.java:476)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > lambda$runAsync$0(DataImporter.java:457)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.NullPointerException
> > at java.text.SimpleDateFormat.(SimpleDateFormat.java:
> > 598)
> > at org.apache.solr.handler.dataimport.
> > SimplePropertiesWriter.init(SimplePropertiesWriter.java:100)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > createPropertyWriter(DataImporter.java:328)
> > ... 4 more
> >
> > Has anyone an idea what is wrong or missing?
> >
> > Thanks,
> > Birgit
> >
> >
> >
> > -Original Message-
> > From: Jamie Jackson [mailto:jamieja...@gmail.com]
> > Sent: Tuesday, September 19, 2017 3:42 AM
> > To: solr-user@lucene.apache.org
> > Subject: [bulk]: Dates and DataImportHandler
> >
> > Hi folks,
> >
> > My DB server is on America/Chicago time. Solr (on Docker) is running
> > on UTC. Dates coming from my (MariaDB) data source seem to get
> > translated properly into the Solr index without me doing anything
> special.
> >
> > However when doing delta imports using last_index_time (
> > http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport
> > ), I can't seem

Re: [bulk]: Dates and DataImportHandler

2017-09-19 Thread Jamie Jackson 
FWIW, I know mine worked, so maybe try:



I can't conceive of what the locale would possibly do when a dateFormat is
specified, so I omitted the attribute. (Maybe one can specify dateFormat
*or *locale--it seems like specifying both would cause a clash.) For what
it's worth, the format you're trying to write seems identical to the
default*, so I'm not sure what benefit you're getting by using that
propertyWriter.

*It's identical to *my* default, anyway. Maybe the default changes based on
one's system configuration, I don't know. This stuff isn't very well
documented.

On Tue, Sep 19, 2017 at 7:22 AM, Mannott, Birgit <b.mann...@klopotek.de>
wrote:

> Hi,
>
> I have a similar problem. I try to change the timezone for the
> last_index_time by setting
>
>  type="SimplePropertiesWriter" locale="en_US" />
>
> in the  section of my data-config.xml file.
>
> But when doing this I always get a NullPointerException on Delta Import:
>
> 2017-09-15 14:04:00.825 INFO  (Thread-2938) [   x:mex_prd_dev1100-ap]
> o.a.s.h.d.DataImporter Starting Delta Import
> 2017-09-15 14:04:00.827 ERROR (Thread-2938) [   x:mex_prd_dev1100-ap]
> o.a.s.h.d.DataImporter Delta Import Failed
> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> PropertyWriter implementation:SimplePropertiesWriter
> at org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWriter(DataImporter.java:330)
> at org.apache.solr.handler.dataimport.DataImporter.
> doDeltaImport(DataImporter.java:439)
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:476)
> at org.apache.solr.handler.dataimport.DataImporter.
> lambda$runAsync$0(DataImporter.java:457)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at java.text.SimpleDateFormat.(SimpleDateFormat.java:
> 598)
> at org.apache.solr.handler.dataimport.
> SimplePropertiesWriter.init(SimplePropertiesWriter.java:100)
> at org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWriter(DataImporter.java:328)
>     ... 4 more
>
> Has anyone an idea what is wrong or missing?
>
> Thanks,
> Birgit
>
>
>
> -Original Message-
> From: Jamie Jackson [mailto:jamieja...@gmail.com]
> Sent: Tuesday, September 19, 2017 3:42 AM
> To: solr-user@lucene.apache.org
> Subject: [bulk]: Dates and DataImportHandler
>
> Hi folks,
>
> My DB server is on America/Chicago time. Solr (on Docker) is running on
> UTC. Dates coming from my (MariaDB) data source seem to get translated
> properly into the Solr index without me doing anything special.
>
> However when doing delta imports using last_index_time (
> http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport ), I
> can't seem to get the date, which Solr provides, to be understood by the DB
> as being UTC (and translated back, accordingly). In other words, the DB
> thinks the Solr UTC date is local, so it thinks the date is ahead by six
> hours.
>
> '${dataimporter.request.clean}' != 'false'
>
> or dt > '${dataimporter.last_index_time}'
>
> I came up with this workaround, which seems to work:
>
> '${dataimporter.request.clean}' != 'false'
>
> /* ${user.timezone} is UTC, and the ${custom.dataimporter.datasource.tz}
> property is set to America/Chicago */
>
> or dt > CONVERT_TZ('${dataimporter.last_index_time}','${user.
> timezone}','${
> custom.dataimporter.datasource.tz}')
>
> However, isn't there a way for this translation to happen more naturally?
>
> I thought maybe I could do something like this:
>
> 
> dateFormat="-MM-dd HH:mm:ssZ"
>
> type="SimplePropertiesWriter"
>
> />
>
> The above did set the property as expected (with a trailiing `+`), but
> that didn't seem to help the DB understand/translate the date.
>
> Thanks,
> Jamie
>

Dates and DataImportHandler

2017-09-18 Thread Jamie Jackson 
Hi folks,

My DB server is on America/Chicago time. Solr (on Docker) is running on
UTC. Dates coming from my (MariaDB) data source seem to get translated
properly into the Solr index without me doing anything special.

However when doing delta imports using last_index_time (
http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport ), I
can't seem to get the date, which Solr provides, to be understood by the DB
as being UTC (and translated back, accordingly). In other words, the DB
thinks the Solr UTC date is local, so it thinks the date is ahead by six
hours.

'${dataimporter.request.clean}' != 'false'

or dt > '${dataimporter.last_index_time}'

I came up with this workaround, which seems to work:

'${dataimporter.request.clean}' != 'false'

/* ${user.timezone} is UTC, and the ${custom.dataimporter.datasource.tz}
property is set to America/Chicago */

or dt > CONVERT_TZ('${dataimporter.last_index_time}','${user.timezone}','${
custom.dataimporter.datasource.tz}')

However, isn't there a way for this translation to happen more naturally?

I thought maybe I could do something like this:



The above did set the property as expected (with a trailiing `+`), but
that didn't seem to help the DB understand/translate the date.

Thanks,
Jamie

Re: Scheduling Data Import Handler (DIH) Delta Imports

2016-11-28 Thread Jamie Jackson 
One last bump before I get crackin'...

On Mon, Nov 21, 2016 at 11:54 AM, Jamie Jackson <jamieja...@gmail.com>
wrote:

> Hi Folks,
>
> I have DIH cores that are being indexed by my Lucee application. That
> works, but I'd like to make some improvements:
>
>
>- Make a standalone scheduler that's not part of a larger application.
>(FYI, I want to Dockerize the import-triggering service.)
>- Prevent import requests from stacking up. Some of my cores' delta
>imports run every 15 seconds, and they do so blindly/ignorantly. If there
>is contention, very occasionally, import jobs will run long and stack up,
>so I want to make the scheduler nicer/more intelligent. Maybe the service
>would check the import status to see if there's a job already running
>before requesting a new one.
>
> I can write such a thing myself, but does anybody have a Linux or
> cross-platform solution written already?
>
> Thanks,
> Jamie
>

Scheduling Data Import Handler (DIH) Delta Imports

2016-11-21 Thread Jamie Jackson 
Hi Folks,

I have DIH cores that are being indexed by my Lucee application. That
works, but I'd like to make some improvements:


   - Make a standalone scheduler that's not part of a larger application.
   (FYI, I want to Dockerize the import-triggering service.)
   - Prevent import requests from stacking up. Some of my cores' delta
   imports run every 15 seconds, and they do so blindly/ignorantly. If there
   is contention, very occasionally, import jobs will run long and stack up,
   so I want to make the scheduler nicer/more intelligent. Maybe the service
   would check the import status to see if there's a job already running
   before requesting a new one.

I can write such a thing myself, but does anybody have a Linux or
cross-platform solution written already?

Thanks,
Jamie

Re: Problem with Password Decryption in Data Import Handler

2016-11-03 Thread Jamie Jackson 
You were right, Fuad. There was a flaw in my script (inconsistent naming of
the `plain_db_pwd` variable.

Thanks for figuring that out.

For posterity, here's the fixed script:


encrypt_key=your_encryption_key
plain_db_pwd=your_db_password
cred_dir=/your/credentials/directory

cd "${cred_dir}
echo -n "${encrypt_key}" > encrypt.key
echo -n "${plain_db_pwd}" | openssl enc -aes-128-cbc -a -salt -k
"${encrypt_key}"
#==#

Then, in the DIH config:
 encryptKeyFile="/your/credentials/directory/encrypt.key"

I have another, semi-related, issue that I'll bring up in another thread.

Thanks,
Jamie


On Wed, Nov 2, 2016 at 6:26 PM, Fuad Efendi <f...@efendi.ca> wrote:

> Then I can only guess that in current configuration decrypted password is
> empty string.
>
> Try to manually replace some characters in encpwd.txt file to see if you
> get different errors; try to delete this file completely to see if you get
> different errors. Try to add new line in this file; try to change password
> in config file.
>
>
>
> On November 2, 2016 at 5:23:33 PM, Jamie Jackson (jamieja...@gmail.com)
> wrote:
>
> I should have mentioned that I verified connectivity with plain passwords:
>
> From the same machine that Solr's running on:
>
> solr@000650cbdd5e:/opt/solr$ mysql -uroot -pOakton153 -h local.mysite.com
> mysite -e "select 'foo' as bar;"
> +-+
> | bar |
> +-+
> | foo |
> +-+
>
> Also, if I add the plain-text password to the config, it connects fine:
>
>  driver="org.mariadb.jdbc.Driver"
> url="jdbc:mysql://local.mysite.com:3306/mysite"
> user="root"
> password="Oakton153"
> />
>
>
> So that is why I claim to have a problem with encryptKeyFile, specifically,
> because I've eliminated general connectivity/authentication problems.
>
> Thanks,
> Jamie
>
> On Wed, Nov 2, 2016 at 4:58 PM, Fuad Efendi <f...@efendi.ca> wrote:
>
> > In MySQL, this command will explicitly allow to connect from
> > remote ICZ2002912 host, check MySQL documentation:
> >
> > GRANT ALL ON mysite.* TO 'root’@'ICZ2002912' IDENTIFIED BY ‘Oakton123’;
> >
> >
> >
> > On November 2, 2016 at 4:41:48 PM, Fuad Efendi (f...@efendi.ca) wrote:
> >
> > This is the root of the problem:
> > "Access denied for user 'root'@'ICZ2002912' (using password: NO) “
> >
> >
> > First of all, ensure that plain (non-encrypted) password settings work
> for
> > you.
> >
> > Check that you can connect using MySQL client from ICZ2002912 to your
> > MySQL & Co. instance
> >
> > I suspect you need to allow MySQL & Co. to accept connections
> > from ICZ2002912. Plus, check DNS resolution, etc.
> >
> >
> > Thanks,
> >
> >
> > --
> > Fuad Efendi
> > (416) 993-2060
> > http://www.tokenizer.ca
> > Recommender Systems
> >
> >
> > On November 2, 2016 at 2:37:08 PM, Jamie Jackson (jamieja...@gmail.com)
> > wrote:
> >
> > I'm at a brick wall. Here's the latest status:
> >
> > Here are some sample commands that I'm using:
> >
> > *Create the encryptKeyFile and encrypted password:*
> >
> >
> > encrypter_password='this_is_my_encrypter_password'
> > plain_db_pw='Oakton153'
> >
> > cd /var/docker/solr_stage2/credentials/
> > echo -n "${encrypter_password}" > encpwd.txt
> > echo -n "${plain_db_pwd}" > plaindbpwd.txt
> > openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
> > "${encrypter_password}"
> >
> > rm plaindbpwd.txt
> >
> > That generated this as the password, by the way:
> >
> > U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=
> >
> > *Configure DIH configuration:*
> >
> > 
> >
> >  > driver="org.mariadb.jdbc.Driver"
> > url="jdbc:mysql://local.mysite.com:3306/mysite"
> > user="root"
> > password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
> > encryptKeyFile="/opt/solr/credentials/encpwd.txt"
> > />
> > ...
> >
> >
> > By the way, /var/docker/solr_stage2/credentials/ is mapped to
> > /opt/solr/credentials/ in the docker container, so that's why the paths
> > *seem* different (but aren't, really).
> >
> >
> > *Authentication error when data import is run:*
> >
> > Exception while processing:

Re: Problem with Password Decryption in Data Import Handler

2016-11-02 Thread Jamie Jackson 
I should have mentioned that I verified connectivity with plain passwords:

>From the same machine that Solr's running on:

solr@000650cbdd5e:/opt/solr$ mysql -uroot -pOakton153 -h local.mysite.com
mysite -e "select 'foo' as bar;"
+-+
| bar |
+-+
| foo |
+-+

Also, if I add the plain-text password to the config, it connects fine:




So that is why I claim to have a problem with encryptKeyFile, specifically,
because I've eliminated general connectivity/authentication problems.

Thanks,
Jamie

On Wed, Nov 2, 2016 at 4:58 PM, Fuad Efendi <f...@efendi.ca> wrote:

> In MySQL, this command will explicitly allow to connect from
> remote ICZ2002912 host, check MySQL documentation:
>
> GRANT ALL ON mysite.* TO 'root’@'ICZ2002912' IDENTIFIED BY ‘Oakton123’;
>
>
>
> On November 2, 2016 at 4:41:48 PM, Fuad Efendi (f...@efendi.ca) wrote:
>
> This is the root of the problem:
> "Access denied for user 'root'@'ICZ2002912' (using password: NO) “
>
>
> First of all, ensure that plain (non-encrypted) password settings work for
> you.
>
> Check that you can connect using MySQL client from ICZ2002912 to your
> MySQL & Co. instance
>
> I suspect you need to allow MySQL & Co. to accept connections
> from ICZ2002912. Plus, check DNS resolution, etc.
>
>
> Thanks,
>
>
> --
> Fuad Efendi
> (416) 993-2060
> http://www.tokenizer.ca
> Recommender Systems
>
>
> On November 2, 2016 at 2:37:08 PM, Jamie Jackson (jamieja...@gmail.com)
> wrote:
>
> I'm at a brick wall. Here's the latest status:
>
> Here are some sample commands that I'm using:
>
> *Create the encryptKeyFile and encrypted password:*
>
>
> encrypter_password='this_is_my_encrypter_password'
> plain_db_pw='Oakton153'
>
> cd /var/docker/solr_stage2/credentials/
> echo -n "${encrypter_password}" > encpwd.txt
> echo -n "${plain_db_pwd}" > plaindbpwd.txt
> openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
> "${encrypter_password}"
>
> rm plaindbpwd.txt
>
> That generated this as the password, by the way:
>
> U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=
>
> *Configure DIH configuration:*
>
> 
>
>  driver="org.mariadb.jdbc.Driver"
> url="jdbc:mysql://local.mysite.com:3306/mysite"
> user="root"
> password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
> encryptKeyFile="/opt/solr/credentials/encpwd.txt"
> />
> ...
>
>
> By the way, /var/docker/solr_stage2/credentials/ is mapped to
> /opt/solr/credentials/ in the docker container, so that's why the paths
> *seem* different (but aren't, really).
>
>
> *Authentication error when data import is run:*
>
> Exception while processing: question document :
> SolrInputDocument(fields:
> []):org.apache.solr.handler.dataimport.DataImportHandlerException:
> Unable to execute query: select 'foo' as bar; Processing
> Document # 1
> at org.apache.solr.handler.dataimport.DataImportHandlerException.
> wrapAndThrow(DataImportHandlerException.java:69)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.(JdbcDataSource.java:323)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getData(JdbcDataSource.java:283)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getData(JdbcDataSource.java:52)
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> initQuery(SqlEntityProcessor.java:59)
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> nextRow(SqlEntityProcessor.java:73)
> at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(
> EntityProcessorWrapper.java:244)
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:475)
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:414)
> at org.apache.solr.handler.dataimport.DocBuilder.
> doFullDump(DocBuilder.java:329)
> at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:232)
> at org.apache.solr.handler.dataimport.DataImporter.
> doFullImport(DataImporter.java:416)
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:480)
> at org.apache.solr.handler.dataimport.DataImporter$1.run(
> DataImporter.java:461)
> Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
> connect: Access denied for user 'root'@'ICZ2002912' (using password:
> NO)
> at org.mariadb.jdbc.internal.util.ExceptionMapper.get(
> ExceptionMapper.java:123)
> at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(
> ExceptionMapper.java:71)
> at org.mariadb.jdbc.Driver.connect(Driver.java:109)
> at org.apache.solr.handler.dataimport.JdbcDataSource$1.

Re: Problem with Password Decryption in Data Import Handler

2016-11-02 Thread Jamie Jackson 
I'm at a brick wall. Here's the latest status:

Here are some sample commands that I'm using:

*Create the encryptKeyFile and encrypted password:*


encrypter_password='this_is_my_encrypter_password'
plain_db_pw='Oakton153'

cd /var/docker/solr_stage2/credentials/
echo -n "${encrypter_password}" > encpwd.txt
echo -n "${plain_db_pwd}" > plaindbpwd.txt
openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
"${encrypter_password}"

rm plaindbpwd.txt

That generated this as the password, by the way:

U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=

*Configure DIH configuration:*




...


By the way, /var/docker/solr_stage2/credentials/ is mapped to
/opt/solr/credentials/ in the docker container, so that's why the paths
*seem* different (but aren't, really).


*Authentication error when data import is run:*

Exception while processing: question document :
SolrInputDocument(fields:
[]):org.apache.solr.handler.dataimport.DataImportHandlerException:
Unable to execute query: select 'foo' as bar;Processing
Document # 1
at 
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:323)
at 
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:283)
at 
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:52)
at 
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
at 
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at 
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at 
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at 
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at 
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at 
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at 
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)
Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
connect: Access denied for user 'root'@'ICZ2002912' (using password:
NO)
at 
org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
at 
org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:71)
at org.mariadb.jdbc.Driver.connect(Driver.java:109)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:192)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
at 
org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:503)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:313)
... 12 more
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
not connect: Access denied for user 'root'@'ICZ2002912' (using
password: NO)
at 
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:524)
at 
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:472)
at 
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:374)
at 
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:763)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)
at org.mariadb.jdbc.Driver.connect(Driver.java:104)
... 16 more



On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <jamieja...@gmail.com> wrote:

> It happens to be ten characters.
>
> On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <arafa...@gmail.com
> > wrote:
>
>> How long is the encryption key (file content)? Because the code I am
>> looking at seems to expect it to be at most 100 characters.
>>
>> Regards,
>>Alex.
>> 
>> Newsletter and resources for Solr beginners and intermediates:
>> http://www.solr-start.com/
>>
>>
>> On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com>
>> wrote:
>> > I haven't tried this but is it possible there is a new line at the end
>> in
>> > the file?
>> >
>> > If you did something like echo "" > file.txt then there would be a new
>> > line. Use echo -n "" > fi

Re: Problem with Password Decryption in Data Import Handler

2016-10-06 Thread Jamie Jackson 
It happens to be ten characters.

On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <arafa...@gmail.com>
wrote:

> How long is the encryption key (file content)? Because the code I am
> looking at seems to expect it to be at most 100 characters.
>
> Regards,
>Alex.
> 
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com> wrote:
> > I haven't tried this but is it possible there is a new line at the end in
> > the file?
> >
> > If you did something like echo "" > file.txt then there would be a new
> > line. Use echo -n "" > file.txt
> >
> > Also you should be able to check how many characters are in the file.
> >
> > Kevin Risden
> >
> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com>
> wrote:
> >
> >> Hi Folks,
> >>
> >> (Using Solr 5.5.3.)
> >>
> >> As far as I know, the only place where encrypted password use is
> documented
> >> is in
> >> https://cwiki.apache.org/confluence/display/solr/
> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> >> under the "Configuring the DIH Configuration File", in a comment in the
> >> sample XML file:
> >>
> >> 
> >>
> >> Anyway, I can encrypt just fine:
> >>
> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> >> enter aes-128-cbc encryption password:
> >> Verifying - enter aes-128-cbc encryption password:
> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> >>
> >>
> >> I can also decrypt just fine from the command line.
> >>
> >> However, if I use the encrypted password and encryptKeyFile in the
> config
> >> file, I end up with an error: "String length must be a multiple of
> four."
> >>
> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
> >>
> >> How do I get this working?
> >>
> >> Thanks,
> >> Jamie
> >>
>

Re: Problem with Password Decryption in Data Import Handler

2016-10-06 Thread Jamie Jackson 
I tried it both ways yesterday--with a newline and without.

On Thu, Oct 6, 2016 at 12:26 PM, Kevin Risden <compuwizard...@gmail.com>
wrote:

> I haven't tried this but is it possible there is a new line at the end in
> the file?
>
> If you did something like echo "" > file.txt then there would be a new
> line. Use echo -n "" > file.txt
>
> Also you should be able to check how many characters are in the file.
>
> Kevin Risden
>
> On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com>
> wrote:
>
> > Hi Folks,
> >
> > (Using Solr 5.5.3.)
> >
> > As far as I know, the only place where encrypted password use is
> documented
> > is in
> > https://cwiki.apache.org/confluence/display/solr/
> > Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> > under the "Configuring the DIH Configuration File", in a comment in the
> > sample XML file:
> >
> > 
> >
> > Anyway, I can encrypt just fine:
> >
> > $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> > enter aes-128-cbc encryption password:
> > Verifying - enter aes-128-cbc encryption password:
> > U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> >
> >
> > I can also decrypt just fine from the command line.
> >
> > However, if I use the encrypted password and encryptKeyFile in the config
> > file, I end up with an error: "String length must be a multiple of four."
> >
> > https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
> >
> > How do I get this working?
> >
> > Thanks,
> > Jamie
> >
>

Problem with Password Decryption in Data Import Handler

2016-10-05 Thread Jamie Jackson 
Hi Folks,

(Using Solr 5.5.3.)

As far as I know, the only place where encrypted password use is documented
is in
https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
under the "Configuring the DIH Configuration File", in a comment in the
sample XML file:



Anyway, I can encrypt just fine:

$ openssl enc -aes-128-cbc -a -salt -in stgps.txt
enter aes-128-cbc encryption password:
Verifying - enter aes-128-cbc encryption password:
U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=


I can also decrypt just fine from the command line.

However, if I use the encrypted password and encryptKeyFile in the config
file, I end up with an error: "String length must be a multiple of four."

https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9

How do I get this working?

Thanks,
Jamie

Re: JNDI settings

2016-09-27 Thread Jamie Jackson 
In 4, with a separate servlet container, I used externally-defined,
container-managed jndi data sources. However, I'm in the middle of an
upgrade to 5 (before moving on to 6), and I had to add the DIH lib to each
core's solrconfig.xml and add connection details to each core's DIH config
xml. See this:
https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler

On Sep 12, 2016 11:29 AM, "Aristedes Maniatis"  wrote:

> I am using Solr 5.5 and wanting to add JNDI settings to Solr (for data
> import). I'm new to Solr Cloud setup (previously I was running Solr running
> as a custom bundled war) so I can't figure where to put the JNDI settings
> with user/pass themselves.
>
> I don't want to add it to jetty.xml because that's part of the packaged
> application which will be upgraded from time to time.
>
> Should it go into solr.xml inside the solr.home directory? If so, what's
> the right syntax there?
>
>
> Ari
>
>
> --
> -->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>

Solrj: Cannot retry request with a non-repeatable request entity.

2014-10-03 Thread Jamie Jackson 
Please help me get over this obstacle:

Stack trace and (ColdFusion) snippet: https://gist.github.com/jamiejackson

(This is an attempt to get Solrj to perform a deleteByQuery
with ConcurrentUpdateSolrServer using basic authentication. I am a noob
with the entire Solrj and HTTPClient stack. )

Please let me know if you need more info.

FWIW, here's a message I got on the HttpClient mailing list:

Jamie
This question should have probably been addressed to Solrj community.
From the HC standpoint you have three options
(1) make request entity repeatable
(2) force authentication by doing a GET prior to doing PUT or POST
(3) use 'expect-continue' handshake on the client side if the handshake
is supported by the server side.

Oleg


You'll notice that I tried to set expect-continue (commented out in my
snipppet), but that didn't seem to work.

Thanks,
Jamie