Re: SOLR last modified different than filesystem last modified

[Toke Eskildsen]

sat  wrote:
> What do you mean by updating the viewer ?

Whatever you are using to viewing the file timestamps seems to adjust for your 
local timezone (which is normally a fine thing). How you tell your viewer to 
show UTC instead, I don't know. If you use a command line tool under Linux, I 
would suspect it to be very easy. If you use the in-build graphical file 
explorer, I suspect the only way to do so is by adjusting timezone settings for 
the whole system. Etc.

- Toke Eskildsen

Fwd: atomic update in solr with pysolr(python client)

[sara hajili]

hi
have any idea about implement atomic update with pysolr.?
i found this in solrJ:
HttpSolrClient client = new HttpSolrClient("http://localhost:8983/solr;);

// create the document
SolrInputDocument sdoc = new SolrInputDocument();
sdoc.addField("id","book1");
Map fieldModifier = new HashMap<>(1);
fieldModifier.put("add","Cyberpunk");
sdoc.addField("cat", fieldModifier);  // add the map as the field value

client.add( sdoc );  // send it to the solr server

client.close();  // shutdown client before we exit

how i implement this in pysolr?

conn = pysolr.Solr('http://localhost:8983/solr/post')

docs=[{'id':'post1','title':'this is a test','member_id':{"inc":12}}]

conn.add(docs)
this throw an error in :

member_id':{"inc":12}

any idea to implement atomic update?to increament just member_id field

Re: frequently update field

[Upayavira]

On Sat, Sep 5, 2015, at 06:16 AM, sara hajili wrote:
> hi
> i am new in solr, i face to a problem and need any solution to solve
> that.
> i have a field that this field need to update frequently.
> "image i need to index all post of member of a social app"
> in this case i need to store and index all posts field like caption ,
> image, title,comments ,etc
> but question is about some field like
> "like_count,repost_count,comment_count" this field frequenly changed and
> i
> need to update that but other like caption ,title are not as the same of
> like count field.
> so what is the best solution to handle this frequntly update..
> i found that in solr 4 people used external file.
> but now in solr 5.x i see that atomic update appear.
> atomic update is substitute of extenal file?and what is best approach in
> this case?
> "i really worry about cost of re indexing docs when update like count"

You are right to worry about the re-indexing cost.

You need to think about what you want to do with the field once it is
indexed.

Do you want to filter based upon it? Do you want to score based upon it?
Do you want to get the value back in a search result?

The first two can be done, the third hasn't yet been implemented in
Solr. 

Upayavira

ghostly config issues

[Mark Fenbers]

The log data is from solr.log.  There are a couple of puzzling items.

1. On line 2015-09-05 19:19:56.678, it shows a "lib" subdir
   (/localapps/dev/EventLog/lib) which doesn't exist and isn't
   specified anywhere that I can find (lots of "find | grep"
   commands).  I did, at one point, specify this in a version of
   solrconfig.xml that I was experimenting with, but have since removed
   that long ago.  The fact that this still appears is strange, as if
   it is using an old cache or something.
2. On line 2015-09-05 19:19:57.455, it shows a .../conf/conf/... and I
   can't figure out where the double "conf" is coming from.
   /localapps/dev/EventLog/solr/conf/conf/solrconfig.xml doesn't exist,
   and solrconfig.xml resides in
   /localapps/dev/EventLog/solr/conf/ (just one conf) where it belongs!

Can someone give me clues how to uncover these ghostly configuration 
settings?

Mark


2015-09-05 19:19:54.416 INFO  (main) [   ] o.e.j.u.log Logging 
initialized @902ms
2015-09-05 19:19:54.815 INFO  (main) [   ] o.e.j.s.Server 
jetty-9.2.11.v20150529
2015-09-05 19:19:54.851 WARN  (main) [   ] o.e.j.s.h.RequestLogHandler 
!RequestLog
2015-09-05 19:19:54.856 INFO  (main) [   ] o.e.j.d.p.ScanningAppProvider 
Deployment monitor [file:/localapps/dev/solr-5.3.0/server/contexts/] at 
interval 0
2015-09-05 19:19:56.027 INFO  (main) [   ] 
o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not 
find org.apache.jasper.servlet.JspServlet
2015-09-05 19:19:56.057 WARN  (main) [   ] o.e.j.s.SecurityHandler 
ServletContext@o.e.j.w.WebAppContext@51cc87e3{/solr,file:/localapps/dev/solr-5.3.0/server/solr-webapp/webapp/,STARTING}{/localapps/dev/solr-5.3.0/server/solr-webapp/webapp} 
has uncovered http methods for path: /
2015-09-05 19:19:56.076 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter 
SolrDispatchFilter.init(): WebAppClassLoader=784350225@2ec03c11
2015-09-05 19:19:56.110 INFO  (main) [   ] o.a.s.c.SolrResourceLoader 
JNDI not configured for solr (NoInitialContextEx)
2015-09-05 19:19:56.111 INFO  (main) [   ] o.a.s.c.SolrResourceLoader 
using system property solr.solr.home: /localapps/dev/EventLog/
2015-09-05 19:19:56.116 INFO  (main) [   ] o.a.s.c.SolrResourceLoader 
new SolrResourceLoader for directory: '/localapps/dev/EventLog/'
2015-09-05 19:19:56.449 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading 
container configuration from /localapps/dev/EventLog/solr.xml
2015-09-05 19:19:56.621 INFO  (main) [   ] o.a.s.c.CoresLocator 
Config-defined core root directory: /localapps/dev/EventLog
2015-09-05 19:19:56.677 INFO  (main) [   ] o.a.s.c.CoreContainer New 
CoreContainer 748968642
2015-09-05 19:19:56.678 INFO  (main) [   ] o.a.s.c.CoreContainer Loading 
cores into CoreContainer [instanceDir=/localapps/dev/EventLog/]
2015-09-05 19:19:56.678 INFO  (main) [   ] o.a.s.c.CoreContainer loading 
shared library: /localapps/dev/EventLog/lib
2015-09-05 19:19:56.678 WARN  (main) [   ] o.a.s.c.SolrResourceLoader 
Can't find (or read) directory to add to classloader: lib (resolved as: 
/localapps/dev/EventLog/lib).
2015-09-05 19:19:56.721 INFO  (main) [   ] 
o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
60,connTimeout : 6,maxConnectionsPerHost : 20,maxConnections : 
1,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 
5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
2015-09-05 19:19:57.120 INFO  (main) [   ] o.a.s.u.UpdateShardHandler 
Creating UpdateShardHandler HTTP client with params: 
socketTimeout=60=6=true
2015-09-05 19:19:57.129 INFO  (main) [   ] o.a.s.l.LogWatcher SLF4J impl 
is org.slf4j.impl.Log4jLoggerFactory
2015-09-05 19:19:57.132 INFO  (main) [   ] o.a.s.l.LogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2015-09-05 19:19:57.139 INFO  (main) [   ] o.a.s.c.CoreContainer 
Security conf doesn't exist. Skipping setup for authorization module.
2015-09-05 19:19:57.140 INFO  (main) [   ] o.a.s.c.CoreContainer No 
authentication plugin used.
2015-09-05 19:19:57.225 INFO  (main) [   ] o.a.s.c.CoresLocator Looking 
for core definitions underneath /localapps/dev/EventLog
2015-09-05 19:19:57.409 INFO  (main) [   ] o.a.s.c.SolrCore Created 
CoreDescriptor: {name=EventLog, config=solrconfig.xml, transient=false, 
schema=schema.xml, loadOnStartup=true, 
configSetProperties=configsetprops.json, 
instanceDir=/localapps/dev/EventLog/solr/conf, 
absoluteInstDir=/localapps/dev/EventLog/solr/conf/, 
coreNodeName=core_node5, dataDir=/localapps/dev/EventLog}
2015-09-05 19:19:57.411 INFO  (main) [   ] o.a.s.c.CoresLocator Found 
core EventLog in /localapps/dev/EventLog/solr/conf/
2015-09-05 19:19:57.415 INFO  (main) [   ] o.a.s.c.CoresLocator Found 1 
core definitions
2015-09-05 19:19:57.425 INFO  (coreLoadExecutor-6-thread-1) [   ] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/localapps/dev/EventLog/solr/conf/'
2015-09-05 19:19:57.426 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter 

Re: Search opening hours

[O. Klein]

Darren,

So I created worldBounds="0 0 10080 0"

Didn't get polygons to work (needed at least 4 points that closed the
polygon), but I figured I only needed a line and check if a point was on
there. So I am using linestring now

Was wondering if this would work for Monday problem. Will the line be drawn
through the boundary and start at front again?






--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-opening-hours-tp4225250p4227330.html
Sent from the Solr - User mailing list archive at Nabble.com.

Could we merge solr logo into our logo?

[Shi Jinghai]

Hi there,

We are integrating solr into our project (Apache OFBiz project). I’m not sure 
whether we could change the logo in solr admin page. I merged OFBiz and solr 
logo together, here it is:
https://issues.apache.org/jira/secure/attachment/12753546/ofbiz-solr-logo-screenshot.jpg
 


Could we use solr logo this way?

TIA,

Shi Jinghai

Re: Could we merge solr logo into our logo?

[Mikhail Khludnev]

solr/webapp/web/css/styles/common.css

refers to

solr/webapp/web/img/solr.svg

by
#header #solr
{
  background-image: url( ../../img/solr.svg );

On Sat, Sep 5, 2015 at 3:08 PM, Shi Jinghai  wrote:

> Hi there,
>
> We are integrating solr into our project (Apache OFBiz project). I’m not
> sure whether we could change the logo in solr admin page. I merged OFBiz
> and solr logo together, here it is:
>
> https://issues.apache.org/jira/secure/attachment/12753546/ofbiz-solr-logo-screenshot.jpg
> <
> https://issues.apache.org/jira/secure/attachment/12753546/ofbiz-solr-logo-screenshot.jpg
> >
>
> Could we use solr logo this way?
>
> TIA,
>
> Shi Jinghai




-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

Re: frequently update field

[Jack Krupansky]

The standard recommendation is to create a proof of concept implementation
and see how well it performs.

The external file approach is intended for bulk update, such as when the
pricing for many products will be updated all at once.

Atomic update is recommended for incremental updates.

Atomic update does depend of setting all source fields to stored since the
entire document must be first read from the stored values before updating
the selected fields.

If storing all source fields is prohibitive, then they must be stored in an
external database so that the full documents can be reindexed when updating
is required.

As with any database question, the first thing you must do is identify your
access patterns - how much data will you be updating and with what
frequency.

Generally, atomic update is recommended when only a small fraction of the
data will be updated in some relatively small interval of time, such as
hundreds of documents per hour or dozens of documents per minute, or a
handful per second.


-- Jack Krupansky

On Sat, Sep 5, 2015 at 1:16 AM, sara hajili  wrote:

> hi
> i am new in solr, i face to a problem and need any solution to solve that.
> i have a field that this field need to update frequently.
> "image i need to index all post of member of a social app"
> in this case i need to store and index all posts field like caption ,
> image, title,comments ,etc
> but question is about some field like
> "like_count,repost_count,comment_count" this field frequenly changed and i
> need to update that but other like caption ,title are not as the same of
> like count field.
> so what is the best solution to handle this frequntly update..
> i found that in solr 4 people used external file.
> but now in solr 5.x i see that atomic update appear.
> atomic update is substitute of extenal file?and what is best approach in
> this case?
> "i really worry about cost of re indexing docs when update like count"
>

Re: Issue Using Solr 5.3 Authentication and Authorization Plugins

[Dan Davis]

Kevin & Noble,

I'll take it on to test this.   I've built from source before, and I've
wanted this authorization capability for awhile.

On Fri, Sep 4, 2015 at 9:59 AM, Kevin Lee  wrote:

> Noble,
>
> Does SOLR-8000 need to be re-opened?  Has anyone else been able to test
> the restart fix?
>
> At startup, these are the log messages that say there is no security
> configuration and the plugins aren’t being used even though security.json
> is in Zookeeper:
> 2015-09-04 08:06:21.205 INFO  (main) [   ] o.a.s.c.CoreContainer Security
> conf doesn't exist. Skipping setup for authorization module.
> 2015-09-04 08:06:21.205 INFO  (main) [   ] o.a.s.c.CoreContainer No
> authentication plugin used.
>
> Thanks,
> Kevin
>
> > On Sep 4, 2015, at 5:47 AM, Noble Paul  wrote:
> >
> > There are no download links for 5.3.x branch  till we do a bug fix
> release
> >
> > If you wish to download the trunk nightly (which is not same as 5.3.0)
> > check here
> https://builds.apache.org/job/Solr-Artifacts-trunk/lastSuccessfulBuild/artifact/solr/package/
> >
> > If you wish to get the binaries for 5.3 branch you will have to make it
> > (you will need to install svn and ant)
> >
> > Here are the steps
> >
> > svn checkout
> http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_3/
> > cd lucene_solr_5_3/solr
> > ant server
> >
> >
> >
> > On Fri, Sep 4, 2015 at 4:11 PM, davidphilip cherian
> >  wrote:
> >> Hi Kevin/Noble,
> >>
> >> What is the download link to take the latest? What are the steps to
> compile
> >> it, test and use?
> >> We also have a use case to have this feature in solr too. Therefore,
> wanted
> >> to test and above info would help a lot to get started.
> >>
> >> Thanks.
> >>
> >>
> >> On Fri, Sep 4, 2015 at 1:45 PM, Kevin Lee 
> wrote:
> >>
> >>> Thanks, I downloaded the source and compiled it and replaced the jar
> file
> >>> in the dist and solr-webapp’s WEB-INF/lib directory.  It does seem to
> be
> >>> protecting the Collections API reload command now as long as I upload
> the
> >>> security.json after startup of the Solr instances.  If I shutdown and
> bring
> >>> the instances back up, the security is no longer in place and I have to
> >>> upload the security.json again for it to take effect.
> >>>
> >>> - Kevin
> >>>
>  On Sep 3, 2015, at 10:29 PM, Noble Paul  wrote:
> 
>  Both these are committed. If you could test with the latest 5.3 branch
>  it would be helpful
> 
>  On Wed, Sep 2, 2015 at 5:11 PM, Noble Paul 
> wrote:
> > I opened a ticket for the same
> > https://issues.apache.org/jira/browse/SOLR-8004
> >
> > On Wed, Sep 2, 2015 at 1:36 PM, Kevin Lee  >
> >>> wrote:
> >> I’ve found that completely exiting Chrome or Firefox and opening it
> >>> back up re-prompts for credentials when they are required.  It was
> >>> re-prompting with the /browse path where authentication was working
> each
> >>> time I completely exited and started the browser again, however it
> won’t
> >>> re-prompt unless you exit completely and close all running instances
> so I
> >>> closed all instances each time to test.
> >>
> >> However, to make sure I ran it via the command line via curl as
> >>> suggested and it still does not give any authentication error when
> trying
> >>> to issue the command via curl.  I get a success response from all the
> Solr
> >>> instances that the reload was successful.
> >>
> >> Not sure why the pre-canned permissions aren’t working, but the one
> to
> >>> the request handler at the /browse path is.
> >>
> >>
> >>> On Sep 1, 2015, at 11:03 PM, Noble Paul 
> wrote:
> >>>
> >>> " However, after uploading the new security.json and restarting the
> >>> web browser,"
> >>>
> >>> The browser remembers your login , So it is unlikely to prompt for
> the
> >>> credentials again.
> >>>
> >>> Why don't you try the RELOAD operation using command line (curl) ?
> >>>
> >>> On Tue, Sep 1, 2015 at 10:31 PM, Kevin Lee
> 
> >>> wrote:
>  The restart issues aside, I’m trying to lockdown usage of the
> >>> Collections API, but that also does not seem to be working either.
> 
>  Here is my security.json.  I’m using the “collection-admin-edit”
> >>> permission and assigning it to the “adminRole”.  However, after
> uploading
> >>> the new security.json and restarting the web browser, it doesn’t seem
> to be
> >>> requiring credentials when calling the RELOAD action on the Collections
> >>> API.  The only thing that seems to work is the custom permission
> “browse”
> >>> which is requiring authentication before allowing me to pull up the
> page.
> >>> Am I using the permissions correctly for the
> RuleBasedAuthorizationPlugin?
> 
> 

Re: ghostly config issues

[Shawn Heisey]

On 9/5/2015 1:56 PM, Mark Fenbers wrote:
> The log data is from solr.log.  There are a couple of puzzling items.
> 
> 1. On line 2015-09-05 19:19:56.678, it shows a "lib" subdir
>(/localapps/dev/EventLog/lib) which doesn't exist and isn't
>specified anywhere that I can find (lots of "find | grep"
>commands).  I did, at one point, specify this in a version of
>solrconfig.xml that I was experimenting with, but have since removed
>that long ago.  The fact that this still appears is strange, as if
>it is using an old cache or something.
> 2. On line 2015-09-05 19:19:57.455, it shows a .../conf/conf/... and I
>can't figure out where the double "conf" is coming from.
>/localapps/dev/EventLog/solr/conf/conf/solrconfig.xml doesn't exist,
>and solrconfig.xml resides in
>/localapps/dev/EventLog/solr/conf/ (just one conf) where it belongs!

Your solr home is /localapps/dev/EventLog ... Solr automatically loads
any jar found in the lib directory in the solr home, so it is attempting
to use /localapps/dev/EventLog/lib for the classloader.

For the other things you noticed, I believe I know why that is happening
too.

This SHOULD be the structure of a core instanceDir, if "collection1" is
the name of that directory.  This is highly simplified and missing
things you would likely find in the directory structure:

collection1/
|-core.properties
|-conf/
|--solrconfig.xml
|--schema.xml
|--stopwords.txt
|-data/
|--index/
|---[Lucene index files go here]

The directory with the core.properties file is the instanceDir.  The
instanceDir is supposed to contain a conf directory and a data directory.

It appears that you have this of structure in your solr home (leaving a
lot of things out on this one):

solr/
|-conf/
|--core.properties
|--solrconfig.xml
|--schema.xml

This is why it is looking in a path that has "conf" twice -- it is going
to the instanceDir (where it found core.properties) and assuming that it
will find a conf directory there.  The conf directory is not supposed to
be the instanceDir.

Thanks,
Shawn