That's too bad - I would really like to have understood this issue. If you pick it up again in the future, please let us know.
Karl On Thu, Sep 16, 2010 at 11:41 AM, Martijn v Groningen <martijn.is.h...@gmail.com> wrote: > Hi Karl, > > Unfortunately in the environment where I'm currently working, is > installing a new Sharepoint instance is not an option. Due to time > pressure we dropped the requirement of crawling documents from > Sharepoint. > > Martijn > > On 15 September 2010 23:53, Karl Wright <daddy...@gmail.com> wrote: >> Any news on this issue? >> Karl >> >> On Mon, Sep 13, 2010 at 3:00 PM, Karl Wright <daddy...@gmail.com> wrote: >>> >>> I would expect that domain administrator privs would be sufficient. ;-) >>> >>> Unfortunately, SharePoint (and .NET services in general) often seem to >>> have unusual security problems with internal communication. I've seen cases >>> where some of SharePoint's own web services have this issue. Never was able >>> to figure out the problem. Perhaps a .NET guru could, but that's not me. >>> >>> Karl >>> >>> On Mon, Sep 13, 2010 at 2:55 PM, Martijn v Groningen >>> <martijn.is.h...@gmail.com> wrote: >>>> >>>> That could explain the error. I've installed and uninstalled the >>>> webservice extension a few times, but I know for sure that the last >>>> time I installed it as domain administrator. Last week I used a >>>> plain-vanilla Sharepoint (trail version), the webservice extension >>>> worked there without any problem. >>>> >>>> On 13 September 2010 19:30, Karl Wright <daddy...@gmail.com> wrote: >>>> > The key error is the following: >>>> > >>>> >>>>>>> >>>> > <soap:Fault><faultcode>soap:Client</faultcode><faultstring>The >>>> > request failed with HTTP status 401: >>>> > >>>> > Unauthorized.</faultstring><faultactor>http://[host]/_vti_bin/MCPermissions.asmx</faultactor><detail><Error><ErrorNumber>1000</ErrorNumber><ErrorMessage>The >>>> > request failed with HTTP status 401: >>>> > >>>> > Unauthorized.</ErrorMessage><ErrorSource>System.Web.Services</ErrorSource></Error></detail></soap:Fault> >>>> > <<<<<< >>>> > >>>> > Clearly the MCPermissions web service does not have sufficient >>>> > permissions >>>> > to perform its task. I don't recall ever having seen this before, but >>>> > perhaps during installation you were not logged in as a user that has >>>> > enough >>>> > permission to perform security lookups. >>>> > >>>> > Karl >>>> > >>>> > On Mon, Sep 13, 2010 at 12:15 PM, Martijn v Groningen >>>> > <martijn.is.h...@gmail.com> wrote: >>>> >> >>>> >> Hi Karl, >>>> >> >>>> >> Today I'm not at the environment where I can verify this, but I'll >>>> >> definitely check this. But I ran into another issue with the >>>> >> Sharepoint connector. In a another environment I installed the >>>> >> Metacarta Sharepoint webservice extensions, but by executing the >>>> >> following post: >>>> >> HTTP POST http://[host]/rotterdamn/_vti_bin/MCPermissions.asmx >>>> >> <?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope >>>> >> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; >>>> >> xmlns:xsd="http://www.w3.org/2001/XMLSchema"; >>>> >> >>>> >> >>>> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";><soapenv:Body><GetPermissionCollection >>>> >> >>>> >> >>>> >> xmlns="http://microsoft.com/sharepoint/webpartpages/";><objectName>/</objectName><objectType>Web</objectType></GetPermissionCollection></soapenv:Body></soapenv:Envelope> >>>> >> >>>> >> I get back the following response (http 500): >>>> >> <?xml version="1.0" encoding="utf-8"?><soap:Envelope >>>> >> xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"; >>>> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>>> >> >>>> >> >>>> >> xmlns:xsd="http://www.w3.org/2001/XMLSchema";><soap:Body><soap:Fault><faultcode>soap:Client</faultcode><faultstring>The >>>> >> request failed with HTTP status 401: >>>> >> >>>> >> >>>> >> Unauthorized.</faultstring><faultactor>http://[host]/_vti_bin/MCPermissions.asmx</faultactor><detail><Error><ErrorNumber>1000</ErrorNumber><ErrorMessage>The >>>> >> request failed with HTTP status 401: >>>> >> >>>> >> >>>> >> Unauthorized.</ErrorMessage><ErrorSource>System.Web.Services</ErrorSource></Error></detail></soap:Fault></soap:Body></soap:Envelope> >>>> >> >>>> >> I seems that I only get an error with the MCPermissions webservice >>>> >> call. Other calls such as GetListCollection work fine. I'm >>>> >> authentication with the domain administrator account. This environment >>>> >> has also Sharepoint 3.0 installed. I'm making these posts to >>>> >> Sharepoint with Firefox http poster plugin. Also the url in the >>>> >> response is without the subsite. >>>> >> >>>> >> Also important to note is that browsing to >>>> >> http://[host]/[subsite]/_vti_bin/MCPermissions.asmx shows the >>>> >> GetPermissionsCollection operation. That is what I checked after >>>> >> installing the webservice extension. You have a clue what might be >>>> >> wrong here? >>>> >> >>>> >> Thanks, >>>> >> >>>> >> Martijn >>>> >> >>>> >> On 13 September 2010 10:27, Karl Wright <daddy...@gmail.com> wrote: >>>> >> > Hi Martijn, >>>> >> > >>>> >> > For the 401 error, here's something also worth trying, to remove the >>>> >> > possibility that your error has anything to do with other recent >>>> >> > changes. >>>> >> > Can you check out the following: >>>> >> > >>>> >> > svn co -r987345 >>>> >> > https://svn.apache.org/repos/asf/incubator/lcf/trunk/modules/lib >>>> >> > >>>> >> > In the checkout lib area, you will see a jar called >>>> >> > commons-httpclient-lcf.jar. Replace commons-httpclient-acf.jar with >>>> >> > it >>>> >> > (renaming to commons-httpclient-acf.jar, of course), and try running >>>> >> > with >>>> >> > it. If your 401 error no longer happens, then it means something >>>> >> > was >>>> >> > messed >>>> >> > up, and I'll need to do some research. >>>> >> > >>>> >> > Thanks, >>>> >> > Karl >>>> >> > >>>> >> > On Sun, Sep 12, 2010 at 5:02 PM, Karl Wright <daddy...@gmail.com> >>>> >> > wrote: >>>> >> >> >>>> >> >> I confirmed that without any mappings set, the Solr Connector >>>> >> >> *should* >>>> >> >> just be passing the metadata through using the metadata's name as >>>> >> >> the >>>> >> >> Solr >>>> >> >> field name. >>>> >> >> >>>> >> >> For debugging, if you could post the Solr output from one update >>>> >> >> operation, I'd love to see if any metadata seems to be in it. >>>> >> >> Potentially >>>> >> >> it's there but the Solr schema is not right somehow - that should >>>> >> >> be >>>> >> >> the >>>> >> >> first thing we verify. >>>> >> >> >>>> >> >> Karl >>>> >> >> >>>> >> >> >>>> >> >> On Sun, Sep 12, 2010 at 4:50 PM, Martijn v Groningen >>>> >> >> <martijn.is.h...@gmail.com> wrote: >>>> >> >>> >>>> >> >>> Tomorrow I'll dive into code and do some more debugging. Last week >>>> >> >>> I >>>> >> >>> didn't specify any mappings in the mapping tab for the meta data >>>> >> >>> fields I selected in the metadata tab. But this shouldn't be the >>>> >> >>> problem, right? >>>> >> >>> >>>> >> >>> Thanks, >>>> >> >>> >>>> >> >>> Martijn >>>> >> >>> >>>> >> >>> On 12 September 2010 22:29, Karl Wright <daddy...@gmail.com> >>>> >> >>> wrote: >>>> >> >>> > Martijn, >>>> >> >>> > >>>> >> >>> > (1) The precise svn url for the acf version of httpclient is as >>>> >> >>> > follows. My >>>> >> >>> > apologies for any earlier confusion - I was away from my >>>> >> >>> > computer at >>>> >> >>> > the >>>> >> >>> > time. >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > https://svn.apache.org/repos/asf/incubator/lcf/upstream/commons-httpclient-3x >>>> >> >>> > >>>> >> >>> > (2) Each time the solr connector posts into Solr, you should see >>>> >> >>> > a >>>> >> >>> > set >>>> >> >>> > of >>>> >> >>> > argument names and values dumped to standard out (or the log). >>>> >> >>> > So >>>> >> >>> > it >>>> >> >>> > should >>>> >> >>> > be easy to see what is being sent, and whether the arguments in >>>> >> >>> > fact >>>> >> >>> > are the >>>> >> >>> > correct ones for the extracting update request handler, or not. >>>> >> >>> > Furthermore, the Solr output connector recently had a tab added >>>> >> >>> > which >>>> >> >>> > performs the mapping I alluded to. This mapping is designed to >>>> >> >>> > translate >>>> >> >>> > metadata coming from a connector like SharePoint, into fields >>>> >> >>> > that >>>> >> >>> > you >>>> >> >>> > presumably have in your Solr schema. However, if you don't set >>>> >> >>> > anything, >>>> >> >>> > the fields are not changed, and you should see an argument for >>>> >> >>> > every >>>> >> >>> > metadata field, something like: literal.xxx=yyy. >>>> >> >>> > >>>> >> >>> > If you have a document that you *know* has metadata, and you've >>>> >> >>> > specified >>>> >> >>> > that metadata in the job, and you run the job after you specify >>>> >> >>> > that >>>> >> >>> > metadata, but still see no literal.xxx=yyy corresponding to it >>>> >> >>> > in >>>> >> >>> > the >>>> >> >>> > Solr >>>> >> >>> > output, then we should spend some time chasing this problem >>>> >> >>> > down. >>>> >> >>> > Be >>>> >> >>> > wary >>>> >> >>> > because incremental crawling means you'll probably not see your >>>> >> >>> > document >>>> >> >>> > processed again unless you either change it in SharePoint, or >>>> >> >>> > delete >>>> >> >>> > and >>>> >> >>> > recreate the job. But be reassured that SharePoint metadata was >>>> >> >>> > covered by >>>> >> >>> > the old MetaCarta tests, and there have been no changes of any >>>> >> >>> > significance >>>> >> >>> > to the SharePoint connector since then, so I have no explanation >>>> >> >>> > why >>>> >> >>> > it >>>> >> >>> > would not work for you too. That's why I'm spending time trying >>>> >> >>> > to >>>> >> >>> > figure >>>> >> >>> > out if this is a Solr connector issue instead. >>>> >> >>> > >>>> >> >>> > Please let me know if this helps you, or whether you need to go >>>> >> >>> > deeper >>>> >> >>> > into >>>> >> >>> > debugging. >>>> >> >>> > >>>> >> >>> > Karl >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > On Sun, Sep 12, 2010 at 4:05 PM, Martijn v Groningen >>>> >> >>> > <martijn.is.h...@gmail.com> wrote: >>>> >> >>> >> >>>> >> >>> >> I didn't notice that I was under the upstream-changes >>>> >> >>> >> directory. >>>> >> >>> >> Thanks for pointing that out. >>>> >> >>> >> >>>> >> >>> >> In Solr I have a wildcard (*) dynamic field, so everything acf >>>> >> >>> >> sends >>>> >> >>> >> should end up in my index (or at least that is what I assume). >>>> >> >>> >> I >>>> >> >>> >> also >>>> >> >>> >> did some debugging in the Solr connecter and I noticed that no >>>> >> >>> >> metadata was send to Solr. I didn't create field mappings in my >>>> >> >>> >> acf >>>> >> >>> >> job. Do you always have to make mapping for metadata? >>>> >> >>> >> >>>> >> >>> >> Martijn >>>> >> >>> >> >>>> >> >>> >> On 12 September 2010 21:09, Karl Wright <daddy...@gmail.com> >>>> >> >>> >> wrote: >>>> >> >>> >> > The source for upstream changes is under >>>> >> >>> >> > lcf/upstream-changes/httpclient, not under trunk. >>>> >> >>> >> > >>>> >> >>> >> > As for the metadata, how are you determining that no metadata >>>> >> >>> >> > is >>>> >> >>> >> > being >>>> >> >>> >> > indexed? If this is Solr you are indexing into, have you set >>>> >> >>> >> > up >>>> >> >>> >> > the >>>> >> >>> >> > appropriate metadata/field mappings? >>>> >> >>> >> > >>>> >> >>> >> > Karl >>>> >> >>> >> > >>>> >> >>> >> > On 9/12/10, Martijn v Groningen <martijn.is.h...@gmail.com> >>>> >> >>> >> > wrote: >>>> >> >>> >> >> To authenticate with Share point I had to include the domain >>>> >> >>> >> >> as >>>> >> >>> >> >> well. >>>> >> >>> >> >> Also the ui reported an error if I didn't specify the >>>> >> >>> >> >> username >>>> >> >>> >> >> in a >>>> >> >>> >> >> domain / username format. Maybe this http client issue was >>>> >> >>> >> >> just >>>> >> >>> >> >> particular with the Sharepoint / Domain Controller >>>> >> >>> >> >> installation >>>> >> >>> >> >> I >>>> >> >>> >> >> was >>>> >> >>> >> >> working with. I also couldn't find the source of afc version >>>> >> >>> >> >> of >>>> >> >>> >> >> http >>>> >> >>> >> >> client. Is it hosted in another source repository? >>>> >> >>> >> >> >>>> >> >>> >> >> I still don't understand why for the documents I crawled, I >>>> >> >>> >> >> didn't >>>> >> >>> >> >> have any metadata associated with it. In the job >>>> >> >>> >> >> configuration I >>>> >> >>> >> >> was >>>> >> >>> >> >> able to choose which metadata I wanted to include. You have >>>> >> >>> >> >> an >>>> >> >>> >> >> idea >>>> >> >>> >> >> what might be the cause of this? >>>> >> >>> >> >> >>>> >> >>> >> >> Regards, >>>> >> >>> >> >> >>>> >> >>> >> >> Martijn >>>> >> >>> >> >> >>>> >> >>> >> >> On 12 September 2010 18:40, Karl Wright <daddy...@gmail.com> >>>> >> >>> >> >> wrote: >>>> >> >>> >> >>> Hi Martijn, >>>> >> >>> >> >>> >>>> >> >>> >> >>> The ACF version of httpclient has support for NTLMv1, >>>> >> >>> >> >>> NTLMv2, >>>> >> >>> >> >>> and >>>> >> >>> >> >>> NTLM2 >>>> >> >>> >> >>> protocols. The standard client does not. >>>> >> >>> >> >>> >>>> >> >>> >> >>> What this means practically for you depends on how the >>>> >> >>> >> >>> Windows >>>> >> >>> >> >>> domain >>>> >> >>> >> >>> controller you are working with is configured. You cannot >>>> >> >>> >> >>> use >>>> >> >>> >> >>> the >>>> >> >>> >> >>> off-the-shelf httpclient and still authenticate if the >>>> >> >>> >> >>> domain >>>> >> >>> >> >>> controller >>>> >> >>> >> >>> is >>>> >> >>> >> >>> configured to not allow LM connections, which is what >>>> >> >>> >> >>> Microsoft >>>> >> >>> >> >>> recommends >>>> >> >>> >> >>> people do. >>>> >> >>> >> >>> >>>> >> >>> >> >>> Since the ACF version of httpclient will always try to >>>> >> >>> >> >>> connect >>>> >> >>> >> >>> using >>>> >> >>> >> >>> NTLMv2, >>>> >> >>> >> >>> this means that you must be more rigorous about setting up >>>> >> >>> >> >>> your >>>> >> >>> >> >>> client >>>> >> >>> >> >>> machine. First, it must have a name, and it must have a >>>> >> >>> >> >>> machine >>>> >> >>> >> >>> account >>>> >> >>> >> >>> in >>>> >> >>> >> >>> the domain. Second, NTLMv2 is much more picky about how >>>> >> >>> >> >>> you >>>> >> >>> >> >>> specify >>>> >> >>> >> >>> user >>>> >> >>> >> >>> and domain. The end user documentation provides details >>>> >> >>> >> >>> that >>>> >> >>> >> >>> may >>>> >> >>> >> >>> be >>>> >> >>> >> >>> helpful >>>> >> >>> >> >>> to you in this regard. >>>> >> >>> >> >>> >>>> >> >>> >> >>> Thanks, >>>> >> >>> >> >>> Karl >>>> >> >>> >> >>> >>>> >> >>> >> >>> >>>> >> >>> >> >>> On Sun, Sep 12, 2010 at 5:00 AM, Martijn v Groningen >>>> >> >>> >> >>> <martijn.is.h...@gmail.com> wrote: >>>> >> >>> >> >>>> >>>> >> >>> >> >>>> Hi All, >>>> >> >>> >> >>>> >>>> >> >>> >> >>>> I've configured the Sharepoint connector (to connect to >>>> >> >>> >> >>>> sharepoint >>>> >> >>> >> >>>> 3.0), Solr connector and a job that adds documents into >>>> >> >>> >> >>>> Solr. >>>> >> >>> >> >>>> The >>>> >> >>> >> >>>> only >>>> >> >>> >> >>>> thing that I'm missing is the meta data from Sharepoint. >>>> >> >>> >> >>>> Per >>>> >> >>> >> >>>> document >>>> >> >>> >> >>>> I need to know which users can access it. In the metadata >>>> >> >>> >> >>>> tab >>>> >> >>> >> >>>> on >>>> >> >>> >> >>>> the >>>> >> >>> >> >>>> job page I've configured the metadata to be included, but >>>> >> >>> >> >>>> this >>>> >> >>> >> >>>> doesn't >>>> >> >>> >> >>>> end up in my Solr index. Does anybody know what I should >>>> >> >>> >> >>>> do to >>>> >> >>> >> >>>> also >>>> >> >>> >> >>>> have the metadata in my index? >>>> >> >>> >> >>>> >>>> >> >>> >> >>>> I also had another issue with the Sharepoint connector >>>> >> >>> >> >>>> which I >>>> >> >>> >> >>>> managed >>>> >> >>> >> >>>> to solve. But I'm curious to know if someone else >>>> >> >>> >> >>>> encountered >>>> >> >>> >> >>>> a >>>> >> >>> >> >>>> similar issue. >>>> >> >>> >> >>>> When I was setting up the sharepoint connecter I always >>>> >> >>> >> >>>> got a >>>> >> >>> >> >>>> 401 >>>> >> >>> >> >>>> message on the connectors page as status. I was sure I >>>> >> >>> >> >>>> entered >>>> >> >>> >> >>>> the >>>> >> >>> >> >>>> correct credentials. After some debugging I noticed that >>>> >> >>> >> >>>> the >>>> >> >>> >> >>>> NLTM >>>> >> >>> >> >>>> data >>>> >> >>> >> >>>> that was send to Solr was different then when I did a http >>>> >> >>> >> >>>> post >>>> >> >>> >> >>>> with >>>> >> >>> >> >>>> Firefox poster plugin to a Sharepoint webservice url (I >>>> >> >>> >> >>>> check >>>> >> >>> >> >>>> this >>>> >> >>> >> >>>> with Wireshark). After writing a little test case with >>>> >> >>> >> >>>> httpclient >>>> >> >>> >> >>>> used >>>> >> >>> >> >>>> in afc, I got the same 401 error. I then ran the test with >>>> >> >>> >> >>>> a >>>> >> >>> >> >>>> clean >>>> >> >>> >> >>>> http client (version 3.1), that ran as expected. I got a >>>> >> >>> >> >>>> response >>>> >> >>> >> >>>> code >>>> >> >>> >> >>>> 200 back with a soap response. I then used this version of >>>> >> >>> >> >>>> http >>>> >> >>> >> >>>> client >>>> >> >>> >> >>>> (with some class filesfrom the afc provided jar that were >>>> >> >>> >> >>>> missing >>>> >> >>> >> >>>> is >>>> >> >>> >> >>>> the plain jar file) and the connector worked as expected >>>> >> >>> >> >>>> as I >>>> >> >>> >> >>>> was >>>> >> >>> >> >>>> able >>>> >> >>> >> >>>> to index documents. Did someone else have this particular >>>> >> >>> >> >>>> issue? >>>> >> >>> >> >>>> I >>>> >> >>> >> >>>> noticed that acf is using httpclient 3.1 (from the >>>> >> >>> >> >>>> manifest >>>> >> >>> >> >>>> file), >>>> >> >>> >> >>>> but >>>> >> >>> >> >>>> I'm curious to know why http client was modified. >>>> >> >>> >> >>>> >>>> >> >>> >> >>>> BTW I've been using the latest trunk version (I did a >>>> >> >>> >> >>>> checkout >>>> >> >>> >> >>>> last >>>> >> >>> >> >>>> tuesday). I'm also new to Sharepoint >>>> >> >>> >> >>>> >>>> >> >>> >> >>>> Cheers, >>>> >> >>> >> >>>> >>>> >> >>> >> >>>> Martijn >>>> >> >>> >> >>> >>>> >> >>> >> >>> >>>> >> >>> >> >> >>>> >> >>> >> > >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >> -- >>>> >> >>> >> Met vriendelijke groet, >>>> >> >>> >> >>>> >> >>> >> Martijn van Groningen >>>> >> >>> > >>>> >> >>> > >>>> >> >>> >>>> >> >>> >>>> >> >>> >>>> >> >>> -- >>>> >> >>> Met vriendelijke groet, >>>> >> >>> >>>> >> >>> Martijn van Groningen >>>> >> >> >>>> >> > >>>> >> > >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Met vriendelijke groet, >>>> >> >>>> >> Martijn van Groningen >>>> > >>>> > >>>> >>>> >>>> >>>> -- >>>> Met vriendelijke groet, >>>> >>>> Martijn van Groningen >>> >> >> > > > > -- > Met vriendelijke groet, > > Martijn van Groningen >
