I set up an ES-2.0 plugin and pointed it at ES 2.0.2. Build fails pretty dramatically. I'd say that pretty much all the API's changed. I committed the code if you want to try for yourself.
You can find the plugin at: https://svn.apache.org/repos/asf/manifoldcf/integration/elasticsearch-2.0/trunk Karl On Tue, Aug 30, 2016 at 3:27 PM, Holtz, Michael (IT Consultant) < [email protected]> wrote: > Ah got it. I was wondering if it was something like that. > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:[email protected]] > *Sent:* Tuesday, August 30, 2016 2:55 PM > > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hi Mike, > > > > There are TWO versions for the plugin: (1) the version of ES it was > written for (1.5), and the functional compatibility with MCF itself (that's > where the 2.1 comes from). > > > > There is no version of the ES plugin that we built for ES 2.x. That's > what we're trying to create right now. > > > > So when you see: > > > > apache-manifoldcf-elasticsearch-1.5-plugin-2.2 > > > > ... the 1.5 means the ES version and the 2.2 is the actual version of the > ES-1.5 plugin. > > > > Karl > > > > > > On Tue, Aug 30, 2016 at 1:25 PM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > So the plugin install fails in Elastic Search 2.1.0 with the same error: > > > > > > C:\APPS\elasticsearch-2.1.0>bin\plugin install file:///C:/Users/holtzm/ > Downloads/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip > > -> Installing from file:/C:/Users/holtzm/Downloads/apache-manifoldcf- > elasticsearch-plugin-2.1-bin.zip... > > Trying file:/C:/Users/holtzm/Downloads/apache-manifoldcf- > elasticsearch-plugin-2.1-bin.zip ... > > Downloading .DONE > > Verifying file:/C:/Users/holtzm/Downloads/apache-manifoldcf- > elasticsearch-plugin-2.1-bin.zip checksums if available ... > > NOTE: Unable to verify checksum for downloaded plugin (unable to find > .sha1 or .md5 file to verify) > > ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in > plugin zip > > > > > > The documentation says for 2.1 though so I’m confused at this point. > > > > I will try adding the ‘plugin-descriptor.properties ‘ file > > > > > > Thanks, > > Mike H. > > > > > > *From:* Holtz, Michael (IT Consultant) > *Sent:* Tuesday, August 30, 2016 1:03 PM > > > *To:* [email protected] > *Subject:* RE: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > I think I’ll switch to Elastic Search version 2.1, just to see it work > first: > > > > *Plugin for Elastic Search (toolkit only)* > > Latest release (Apache ManifoldCF Plugin for Elastic Search, version 2.1, > 2015 Jul 25) > > > > > > > > > > > > BTW I was able to see my first use case work. It turns out that I needed > to explicitly include the custom meta data fields, “include all metadata” > didn’t work. > > > > > > That leaves me with #4 > > > > 1. Index custom meta data fields from SharePoint into the > ElasticSearch index *done* > > 2. Full text indexing of SharePoint content in ElasticSearch *done* > > 3. SharePoint security information indexed in ElasticSearch* done* > > 4. ElasticSearch Plugin to enforce SharePoint security settings for > search *todo* > > > > Thanks, > > Mike H. > > > > > > *From:* Karl Wright [mailto:[email protected] <[email protected]>] > *Sent:* Tuesday, August 30, 2016 12:34 PM > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > We'll probably need to build against ES 2.x, and will need a whole new > plugin. > > > > Karl > > > > > > On Tue, Aug 30, 2016 at 12:29 PM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > Actually I see a classpath exception when starting elastic search so it > doesn’t appear to be that simple. > > > > *From:* Karl Wright [mailto:[email protected]] > *Sent:* Tuesday, August 30, 2016 11:55 AM > > > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Ok, you just included this in the jar and all was OK? It should be > straightforward to set that up if so... > > > > Thanks! > > Karl > > > > > > On Tue, Aug 30, 2016 at 11:27 AM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > Karl, > > > > Ok, I will see what I can do. > > > > I just created the attached file for the 1.5 version of the plugin and was > able to install it. Let me know if you see any issues with it. > > > > I’m going to start with a clean version of the new 2.5 release for my use > cases just using the example setup, let me know if you don’t think that is > a good idea. > > > > Thanks, > > Mike H. > > > > > > *From:* Karl Wright [mailto:[email protected]] > *Sent:* Tuesday, August 30, 2016 11:17 AM > > > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > The 2.0 spec has clearly changed from that needed for the 1.5-level > plugins. > > > > The plugin code is likely to be pretty much unchanged but the packaging > will need to change. Now, I did not develop the 1.5-level plugin, and > don't know that much about ES in the first place, so if you want to > undertake the plugin transformation and contribute it, we'd be very > grateful. > > > > Karl > > > > > > > > On Tue, Aug 30, 2016 at 11:09 AM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > It looks like a ‘plugin-descriptor.properties’ file needs to be added to > the jar: > > > > https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/ > plugin-authors.html > > > > > > *From:* Holtz, Michael (IT Consultant) > *Sent:* Tuesday, August 30, 2016 10:53 AM > > > *To:* [email protected] > *Subject:* RE: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Karl, > > > > > > FYI, Now I’m seeing this: > > > > > > C:\APPS\elasticsearch-2.3.3>bin\plugin install file:///C:/Users/holtzm/ > workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/ > elasticsearch-1.5/target/elasticsearch-1.5-plugin-mcf- > 2.0.1-jar-with-dependencies.jar > > -> Installing from file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch-1.5/target/ > elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar... > > Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch-1.5/target/ > elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar ... > > Downloading ...........................DONE > > Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch-1.5/target/ > elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar checksums if > available ... > > NOTE: Unable to verify checksum for downloaded plugin (unable to find > .sha1 or .md5 file to verify) > > ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in > plugin zip > > > > Mike H. > > > > > > *From:* Holtz, Michael (IT Consultant) > *Sent:* Tuesday, August 30, 2016 10:48 AM > *To:* [email protected] > *Subject:* RE: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Karl, > > > > I got what your’re saying, I should use the 1.5 plugin. Got it. > > > > I will try that > > > > Thanks, > > Mike H. > > > > *From:* Holtz, Michael (IT Consultant) > *Sent:* Tuesday, August 30, 2016 10:47 AM > *To:* [email protected] > *Subject:* RE: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hmm that’s confusing, here is the documentation: > > > > *Apache ManifoldCF ElasticSearch plugin compatibility* > > *ManifoldCF versions* > > *Plugin version* > > 0.1.x-1.4.x > > 0.x > > 1.5.x > > 1.x > > >=1.6.x > > 2.x > > > > > > > > I’m running version 2.4 of manifold so based on this I went with the 2.1 > plugin. > > > > > > Where can I get the 1.0 plugin? I don’t see it on the site or in the 2.4 > build. > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:[email protected] <[email protected]>] > *Sent:* Tuesday, August 30, 2016 10:38 AM > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > You need the ES-1.5 plugin. That's the older (deprecated) 1.0 plugin. > > > > Karl > > > > > > On Tue, Aug 30, 2016 at 10:36 AM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > Elasticsearch 2.3.3 > > Plugin 2.1 > > > > Here is the current error > > > > C:\APPS\elasticsearch-2.3.3>bin\plugin install file:///C:/Users/holtzm/ > workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/ > elasticsearch/elasticsearch-plugin-mcf-2.1.jar > > -> Installing from file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch/elasticsearch- > plugin-mcf-2.1.jar... > > Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar > ... > > Downloading .DONE > > Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar > checksums if available ... > > NOTE: Unable to verify checksum for downloaded plugin (unable to find > .sha1 or .md5 file to verify) > > ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in > plugin zip > > > > > > > > > > > > *From:* Karl Wright [mailto:[email protected]] > *Sent:* Tuesday, August 30, 2016 10:22 AM > > > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hi Mike, > > > > If you are just trying things out you can use the single-process example > and it will make your life easier for experimentation. > > > > As far as the ES plugin, there are two: one for ES up to 1.5, and one for > 1.5 onward. Which did you try to install? > > > > Karl > > > > > > On Tue, Aug 30, 2016 at 9:56 AM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > Karl, > > > > You are correct. > > > > Before I setup zookeeper I would like to prove out a few use cases. > > > > 1. Index custom meta data fields from SharePoint into the > ElasticSearch index > > 2. Full text indexing of SharePoint content in ElasticSearch > > 3. SharePoint security information indexed in ElasticSearch > > 4. ElasticSearch Plugin to enforce SharePoint security settings for > search > > > > At this point I’ve only managed to see #2 and #3 work > > > > For #4 I was getting an error when installing the ElasticSearch plugin. > > I would like to see #1 work before I do anything else. Are there any > example configurations that show how to do this? > > > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:[email protected]] > *Sent:* Tuesday, August 30, 2016 9:49 AM > > > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hi Mike, > > > > This sounds like you're having a lock problem. > > > > Let me guess: you are using the multiprocess example with file-based > locking? If so, you need to stop everything and do the "lock clean" > procedure. I would also strongly recommend that you switch to the > zookeeper multiprocess example. > > > > Thanks, > > Karl > > > > > > On Tue, Aug 30, 2016 at 9:44 AM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > I currently cannot view the job. The UI is hanging when I click the “view” > link for the job. I’m wondering if that has anything to do with the > “Metadata Adjuster” I added? > > > > I did not schedule the job. > > > > At this point the job has finished running and the Document Status and > Queue status reports. I do not see any errors reported there. > > > > Also I still do not see my custom meta data fields from SharePoint in the > ElasticSearch index. > > > > We are hoping to use this in production, but at this point it is not > looking too good. > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:[email protected]] > *Sent:* Tuesday, August 30, 2016 9:09 AM > > > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > It is also important to know whether you have your job set up as > "continuous". If so, it will run forever. > > > > Karl > > > > > > On Tue, Aug 30, 2016 at 8:48 AM, Furkan KAMACI <[email protected]> > wrote: > > Hi Mike, > > > > Could you tell us what "Document Status" and "Queue Status" says as Karl > mentioned? > > > > Kind Regards, > > Furkan KAMACI > > > > On Tue, Aug 30, 2016 at 3:39 PM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > I’m looking at manifoldcf.log and I see no errors. > > > > I have only 13 files and they are all very tiny < 15kb .docx files. > > > > Mike H. > > > > *From:* Karl Wright [mailto:[email protected]] > *Sent:* Monday, August 29, 2016 5:26 PM > > > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > You should be looking at the ManifoldCF.log file, and the examples have > logging for errors at least as being the default. > > > > The only other possibility is that you have a very large document that is > taking a long time to transfer/index. > > > > Also, you can generally see what MCF is up to using the "document status" > and "queue status" reports. > > > > Karl > > > > > > On Mon, Aug 29, 2016 at 5:22 PM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > Karl, > > > > I see no errors in the log. > > > > I probably need to configure the logging I think. > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:[email protected]] > *Sent:* Monday, August 29, 2016 5:20 PM > > > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hi Mike, > > > > Have a look in the log. If there are errors there please send them to > us. Generally, "hangs" are due to MCF retrying documents that have errors. > > > > Karl > > > > > > On Mon, Aug 29, 2016 at 5:18 PM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > Karl, > > > > Yes, it shows up. I include all meta-data. > > > > I’m no trying to use a MetaData adjuster like this > > > > Parameter > > field1 “${Field 1}” > > > > And now I see “field1”=”” in Elastic Search. > > > > Also it seems each time I make a change the job will hang and I need to > restart everything including ElasticSearch to get it to complete. > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:[email protected]] > *Sent:* Monday, August 29, 2016 5:13 PM > *To:* [email protected] > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hi Mike, > > > > If I were you I'd try to independently verify that the custom field shows > up. Do you see it appear in the pulldown for selecting metadata? If not, > it probably is not accessible through the inspection methods that give us > access into SharePoint fields. > > > > If it appears there then we should look at what ES gets sent. It's > possible it appears but under a different field name. > > > > Karl > > > > > > On Mon, Aug 29, 2016 at 5:01 PM, Holtz, Michael (IT Consultant) < > [email protected]> wrote: > > Hi, > > > > I have created a custom content type with a custom field “Field 1”. > > > > I have configured the job to include all meta data. > > > > I do not see “Field 1” being populated in the Elastic Search index. > > > > > > Does anyone have any experience with this? > > > > Thanks in advance, > > Mike H > > > > > > > > > > > > > > > > > > > > > > > > >
