But we just saw from your logs that there were dcterm fields both on initial
ingest and via pid. So gsearch generates the expected index document, and I do
not think that solr would treat the same index document differently at two
different times. There must be something else going on.
Gert
On 19/08/2013, at 14.02, Alistair Young wrote:
> thanks for looking Gert. The problem seems to be, when indexing from scratch,
> after creating an empty index and restarting solr, the resource is indexed
> but only using its DC. So no searches on its dcterms work. It only indexes on
> DCTERMS when the resource is indexed via pid.
>
> Alistair
>
> --
> mov eax,1
> mov ebx,0
> int 80h
>
> From: Gert Schmeltz Pedersen <gerts...@gmail.com>
> Reply-To: "Support and info exchange list for Fedora users."
> <fedora-commons-users@lists.sourceforge.net>
> Date: Monday, 19 August 2013 12:35
> To: "Support and info exchange list for Fedora users."
> <fedora-commons-users@lists.sourceforge.net>
> Subject: Re: [fcrepo-user] gsearch not indexing dcterms on upload
>
> Thank you, Alistair. Sorry that I replied to your previous email, the later
> one was placed in my spam folder by my email client, and I only noticed it
> this morning. I include that email in the history below for completeness.
>
> These two log extracts
>
>
>> the initial ingest from all foxml files for one specific resource that shows
>> the behaviour - initial–ingest-fgs.log
>>
>> the log from indexing that resource from pid - index–from-pid.log
>
>
> both show the generated index document, and they are identical. As I said in
> the previous reply, I thought that the problem was that one of them would
> lack the dcterms fields. Since they are identical, which is as it should be,
> I think I need your explanation again about what the problem is.
>
> Gert
>
>
>
> On 19/08/2013, at 10.26, Alistair Young wrote:
>
>> the later email should contain that info Gert. Just wondering if the initial
>> indexing gets hold of a cached version of the resource (before the DCTERMS
>> namespace was renamed to be the same as the XSLT).
>>
>> thanks,
>>
>> Alistair
>>
>> --
>> mov eax,1
>> mov ebx,0
>> int 80h
>>
>> From: Gert Schmeltz Pedersen <gerts...@gmail.com>
>> Reply-To: "Support and info exchange list for Fedora users."
>> <fedora-commons-users@lists.sourceforge.net>
>> Date: Friday, 16 August 2013 09:25
>> To: "Support and info exchange list for Fedora users."
>> <fedora-commons-users@lists.sourceforge.net>
>> Subject: Re: [fcrepo-user] gsearch not indexing dcterms on upload
>>
>> Thank you, Alistair. Your log lines show me part of one index document
>> generation, which contains the dct (dcterms) fields. However, if I
>> understand you right, the problem is that you get an index document without
>> the dct fields, when you call the updateIndex in one of the other ways
>> (fromPid, fromFoxmlFiles, triggered by ingest/update). So, in order to
>> compare the successful case and the problematic case, I need to see also the
>> log lines from the problematic index document generation, and I need to see
>> all the log lines of both operations, that is, from the INFO log line
>> starting the operation to the INFO log line at the end of the operation.
>>
>> Gert
>
>
> On 15/08/2013, at 10.58, Alistair Young wrote:
>
>> I've dug deeper into the logs Gert and attached are:
>>
>> the initial ingest from all foxml files for one specific resource that shows
>> the behaviour - initial–ingest-fgs.log
>>
>> the log from indexing that resource from pid - index–from-pid.log
>>
>> and the corresponding solr log extracts for both indexing job -
>> solr–extract.log
>>
>> There appear to be two ingests initially for that resource into solr?
>> DEBUG - 2013-08-15 09:31:47.790
>> DEBUG - 2013-08-15 09:34:29.375
>> they're from the same indexing job, not run separately
>>
>> thanks,
>>
>> Alistair
>>
>> --
>>
>>
>> On 14/08/2013, at 17.52, Alistair Young wrote:
>>
>>> Hope this is of some use…
>>>
>>> Alistair
>>>
>>>
>>> -----------------
>>> mov eax,1
>>> mov ebx,0
>>> int 80
>>>
>>> From: Gert Schmeltz Pedersen <gerts...@gmail.com>
>>> Reply-To: "Support and info exchange list for Fedora users."
>>> <fedora-commons-users@lists.sourceforge.net>
>>> Date: Wednesday, 14 August 2013 16:15
>>> To: "Support and info exchange list for Fedora users."
>>> <fedora-commons-users@lists.sourceforge.net>
>>> Subject: Re: [fcrepo-user] gsearch not indexing dcterms on upload
>>>
>>> Let me see the log lines.
>>>
>>> Gert
>>>
>>>
>>> On 14/08/2013, at 17.04, Alistair Young wrote:
>>>
>>>> Having said that, I have to force index via pid to get it to index the
>>>> updated resources.
>>>>
>>>> Renamed all dcterms datastreams namespaces to be same as xslt in the
>>>> objectStore. Rebuilt the resourceIndex and database. Blew away the solr
>>>> index. Index from scratch and it won't index dcterms on the resources that
>>>> had their dcterms renamed. Have to force update on them by pid and then it
>>>> indexes their dcterms.
>>>>
>>>> Any idea why an initial index would fail while an individual index via pid
>>>> would work?
>>>>
>>>> Alistair
>>>>
>>>> -----------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>>
>>>> From: Alistair Young <alistair.yo...@uhi.ac.uk>
>>>> Reply-To: "Support and info exchange list for Fedora users."
>>>> <fedora-commons-users@lists.sourceforge.net>
>>>> Date: Wednesday, 14 August 2013 09:53
>>>> To: "Support and info exchange list for Fedora users."
>>>> <fedora-commons-users@lists.sourceforge.net>
>>>> Subject: Re: [fcrepo-user] gsearch not indexing dcterms on upload
>>>>
>>>> Thanks for that Gert. Finally tracked it down to different representations
>>>> of the same namespace in the repo. Some objects have their dcterms
>>>> namespace declared as per the xslt, others don't. I feel some metadata
>>>> munging coming on. Thanks for confirming the single xslt though. At the
>>>> end of this hoping to produce a tutorial on fedora/gsearch/solr. This
>>>> seems to be the last thorn in the bush.
>>>>
>>>> Cheers,
>>>>
>>>> Alistair
>>>>
>>>>
>>>> From: Gert Schmeltz Pedersen <gerts...@gmail.com>
>>>> Reply-To: "Support and info exchange list for Fedora users."
>>>> <fedora-commons-users@lists.sourceforge.net>
>>>> Date: Tuesday, 13 August 2013 20:51
>>>> To: "Support and info exchange list for Fedora users."
>>>> <fedora-commons-users@lists.sourceforge.net>
>>>> Subject: Re: [fcrepo-user] gsearch not indexing dcterms on upload
>>>>
>>>> No separate xslt, gsearch uses the same indexing xslt, whether triggered
>>>> by an ingest/update of the Fedora object, or by an updateIndex fromPid
>>>> operation, or by an updateIndex fromFoxmlFiles operation. If you get
>>>> different indexing documents in these three cases, then the way to find
>>>> out why, is to compare the log lines in debug mode. I may help, if you
>>>> send me those log lines.
>>>>
>>>> Gert
>>>>
>>>>
>>>> On 13/08/2013, at 18.47, Alistair Young wrote:
>>>>
>>>>> Nup , not that. Even uploading the DCTERMS first it doesn't index them.
>>>>> Is there a separate xslt it uses for updating a resource? The symptoms
>>>>> are identical to when the main xslt was lacking the dcterms namespace.
>>>>>
>>>>> Alistair
>>>>>
>>>>> -------------------
>>>>> Alistair Young
>>>>> Àrd Innleadair air Bathair-bog
>>>>> UHI@Sabhal Mòr Ostaig
>>>>>
>>>>> From: Alistair Young <alistair.yo...@uhi.ac.uk>
>>>>> Reply-To: "Support and info exchange list for Fedora users."
>>>>> <fedora-commons-users@lists.sourceforge.net>
>>>>> Date: Tuesday, 13 August 2013 16:44
>>>>> To: "Support and info exchange list for Fedora users."
>>>>> <fedora-commons-users@lists.sourceforge.net>
>>>>> Subject: Re: [fcrepo-user] gsearch not indexing dcterms on upload
>>>>>
>>>>> I think the problem is down to the multi part commit technique being used
>>>>> when uploading to Fedora:
>>>>>
>>>>> 1 – new object pid is generated by a POST
>>>>> 2 – the DC datastream is then PUT
>>>>> 3 - the DCTERMS datastream is then PUT
>>>>> 4 - the content datastream is then POST
>>>>>
>>>>> so I suspect 3 (or 4) is not getting to solr. gsearch indexes on 2 but
>>>>> not on 3 I'm assuming. All the datastreams are present in the foxml files
>>>>> when doing a new index from scratch so they work.
>>>>>
>>>>> Is it possible to turn off auto indexing on upload and instead force it
>>>>> via a REST call to gsearch once the complete resource is uploaded?
>>>>>
>>>>> Alistair
>>>>>
>>>>> --
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80h
>>>>>
>>>>> From: Alistair Young <alistair.yo...@uhi.ac.uk>
>>>>> Reply-To: "Support and info exchange list for Fedora users."
>>>>> <fedora-commons-users@lists.sourceforge.net>
>>>>> Date: Tuesday, 13 August 2013 16:10
>>>>> To: "Support and info exchange list for Fedora users."
>>>>> <fedora-commons-users@lists.sourceforge.net>
>>>>> Subject: [fcrepo-user] gsearch not indexing dcterms on upload
>>>>>
>>>>> I think I must be missing a file mod or something. Indexing from scratch,
>>>>> using from foxml files works fine and dcterms in all resources are
>>>>> indexed as the namespaces are in the xslt. When uploading a new resource
>>>>> to fedora it gets indexed but only on dc, not dcterms. Does gsearch use a
>>>>> different xslt when indexing from an upload rather than a clean start?
>>>>>
>>>>> Alistair
>>>>>
>>>>> --
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80h
>>>>> ------------------------------------------------------------------------------
>>>>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>>>> It's a free troubleshooting tool designed for production.
>>>>> Get down to code-level detail for bottlenecks, with <2% overhead.
>>>>> Download for free and get started troubleshooting in minutes.
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk_______________________________________________
>>>>> Fedora-commons-users mailing list
>>>>> Fedora-commons-users@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>>> It's a free troubleshooting tool designed for production.
>>>> Get down to code-level detail for bottlenecks, with <2% overhead.
>>>> Download for free and get started troubleshooting in minutes.
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk_______________________________________________
>>>> Fedora-commons-users mailing list
>>>> Fedora-commons-users@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>>>
>>> <fglogs.tar.gz>------------------------------------------------------------------------------
>>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>> It's a free troubleshooting tool designed for production.
>>> Get down to code-level detail for bottlenecks, with <2% overhead.
>>> Download for free and get started troubleshooting in minutes.
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk_______________________________________________
>>> Fedora-commons-users mailing list
>>> Fedora-commons-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>>
>> ------------------------------------------------------------------------------
>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
>> It's a free troubleshooting tool designed for production.
>> Get down to code-level detail for bottlenecks, with <2% overhead.
>> Download for free and get started troubleshooting in minutes.
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk_______________________________________________
>> Fedora-commons-users mailing list
>> Fedora-commons-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk_______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
