[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575837#comment-16575837 ] Steph van Schalkwyk commented on CONNECTORS-1523: - Olivier Thank you for the information. I'll look at the code later today, but it seems I can have both body and head in the englobing list.When i added head, it started parsing jsoup_title, which is what i needed. with best regards Steph > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575830#comment-16575830 ] Olivier Tavard commented on CONNECTORS-1523: Hello, OK thanks for the tests. For your question Steph, be aware that you can have only one englobing tag. On the UI you can choose multiple tags but for the 'englobing tag menu' but only the first one is important (body by default). The code will need a fix to adapt the UI consequently. For the sections "tags to remove" on the other hand you can have multiple tags that will be all taken into account. The goal of the code is to choose one englobing tag ie the part that you want to index and after you can have multiple filters to filter only the part that interests you in this englobing section. > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575726#comment-16575726 ] Steph van Schalkwyk commented on CONNECTORS-1523: - Hi Karl Just cloned and rebuilt and everything working now. Thanks! > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
Hi Karl I just cloned and built. Works now. Last build was Aug 8. Thanks for the help! Steph On Thu, Aug 9, 2018 at 8:20 PM, Steph van Schalkwyk (JIRA) wrote: > > [ https://issues.apache.org/jira/browse/CONNECTORS-1523? > page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel& > focusedCommentId=16575639#comment-16575639 ] > > Steph van Schalkwyk commented on CONNECTORS-1523: > - > > Hi Karl > I built this from trunk on 2018-08-06. > Just tried in in Chrome, FF and IE. > I'll clone right now and test. > Thanks! > Steph > > > > > > > > > HTML Extractor transformation connector - "No englobing tag specified" > > -- > > > > Key: CONNECTORS-1523 > > URL: https://issues.apache.org/ > jira/browse/CONNECTORS-1523 > > Project: ManifoldCF > > Issue Type: Bug > >Affects Versions: ManifoldCF 2.10 > >Reporter: Steph van Schalkwyk > >Priority: Major > > > > When adding Englobing tag to HTML Extractor transformation, Englobing > tag is not persisted. > > Can add on config screen in job edit, but value is not persisted. > > Results in "No englobing tag specified". > > > > -- > This message was sent by Atlassian JIRA > (v7.6.3#76005) >
[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575639#comment-16575639 ] Steph van Schalkwyk commented on CONNECTORS-1523: - Hi Karl I built this from trunk on 2018-08-06. Just tried in in Chrome, FF and IE. I'll clone right now and test. Thanks! Steph > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575552#comment-16575552 ] Karl Wright commented on CONNECTORS-1523: - [~svanschalkwyk], I am playing around with the UI in trunk and I cannot see the problem. I added several englobing tags one at a time, swapped around between tabs, saved, viewed, and edited, and they all behaved exactly as expected. Can you build trunk and try this out? On the offchance it may be browser related, what browser are you using? > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
I'm adding head here for the time-being:
if (includeFilters.isEmpty()) {
includeFilters.add(HtmlExtractorConfig.WHITELIST_DEFAULT);
On Thu, Aug 9, 2018 at 5:07 PM, Steph van Schalkwyk (JIRA)
wrote:
>
> [ https://issues.apache.org/jira/browse/CONNECTORS-1523?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&
> focusedCommentId=16575466#comment-16575466 ]
>
> Steph van Schalkwyk commented on CONNECTORS-1523:
> -
>
> Olivier
> I need the as well as the tags as englobing tags as the
> tag is contained in the tag.
> Thanks,
> Steph
> PS. The code seems to be in 2.10, but the UI will not add additional
> englobal tags.
>
>
>
>
>
>
> > HTML Extractor transformation connector - "No englobing tag specified"
> > --
> >
> > Key: CONNECTORS-1523
> > URL: https://issues.apache.org/
> jira/browse/CONNECTORS-1523
> > Project: ManifoldCF
> > Issue Type: Bug
> >Affects Versions: ManifoldCF 2.10
> >Reporter: Steph van Schalkwyk
> >Priority: Major
> >
> > When adding Englobing tag to HTML Extractor transformation, Englobing
> tag is not persisted.
> > Can add on config screen in job edit, but value is not persisted.
> > Results in "No englobing tag specified".
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)
>
[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575466#comment-16575466 ] Steph van Schalkwyk commented on CONNECTORS-1523: - Olivier I need the as well as the tags as englobing tags as the tag is contained in the tag. Thanks, Steph PS. The code seems to be in 2.10, but the UI will not add additional englobal tags. > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575385#comment-16575385 ] Irindu Nugawela commented on CONNECTORS-1490: - Hi Karl and Piergiorgio, The build is passing on my machine as well !image-2018-08-10-02-30-37-152.png! I have not yet executed a mvn install from the project root but I guess the required jars are locally available in my machine from previous build attemps during develpment > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: image-2018-08-10-02-30-37-152.png, > mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, mongodb-ant-test-ok.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575385#comment-16575385 ] Irindu Nugawela edited comment on CONNECTORS-1490 at 8/9/18 9:05 PM: - Hi Karl and Piergiorgio, The build is passing on my machine as well !image-2018-08-10-02-30-37-152.png! I have not yet executed a mvn install from the project root but I guess the required jars are locally available in my machine from previous build attempts during development was (Author: irindupera): Hi Karl and Piergiorgio, The build is passing on my machine as well !image-2018-08-10-02-30-37-152.png! I have not yet executed a mvn install from the project root but I guess the required jars are locally available in my machine from previous build attemps during develpment > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: image-2018-08-10-02-30-37-152.png, > mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, mongodb-ant-test-ok.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575381#comment-16575381 ] Karl Wright commented on CONNECTORS-1490: - [~irinduPera], now clean out your local Maven repository under .m2, and try that again. :-) > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: image-2018-08-10-02-30-37-152.png, > mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, mongodb-ant-test-ok.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Irindu Nugawela updated CONNECTORS-1490: Attachment: image-2018-08-10-02-30-37-152.png > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: image-2018-08-10-02-30-37-152.png, > mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, mongodb-ant-test-ok.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575290#comment-16575290 ] Karl Wright commented on CONNECTORS-1523: - The code was supposedly included in 2.10, according to the ticket. [~olivierfl], can you verify that it was applied correctly? > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575287#comment-16575287
]
Karl Wright commented on CONNECTORS-1490:
-
>From ant it's still invoking Maven to run the test, so it always fails because
>it can't find the ManifoldCF jars:
{code}
[exec] [INFO] BUILD FAILURE
[exec] [INFO]
[exec] [INFO] Total time: 1.853 s
[exec] [INFO] Finished at: 2018-08-09T15:17:30-04:00
[exec] [INFO]
[exec] [ERROR] Failed to execute goal on project mcf-mongodb-connector:
Could not resolve dependencies for project
org.apache.manifoldcf:mcf-mongodb-connector:jar:2.11-SNAPSHOT: The following
artifacts could not be resolved:
org.apache.manifoldcf:mcf-core:jar:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-connector-common:jar:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-agents:jar:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-pull-agent:jar:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-ui-core:jar:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-core:jar:tests:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-agents:jar:tests:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-pull-agent:jar:tests:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-api-service:war:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-authority-service:war:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-crawler-ui:war:2.11-SNAPSHOT: Failure to find
org.apache.manifoldcf:mcf-core:jar:2.11-SNAPSHOT in
http://oss.sonatype.org/content/repositories/snapshots was cached in the local
repository, resolution will not be reattempted until the update interval of
sonatype-repo has elapsed or updates are forced -> [Help 1]
[exec] [ERROR]
[exec] [ERROR] To see the full stack trace of the errors, re-run Maven
with the -e switch.
[exec] [ERROR] Re-run Maven using the -X switch to enable full debug
logging.
[exec] [ERROR]
[exec] [ERROR] For more information about the errors and possible
solutions, please read the following articles:
[exec] [ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[exec] Result: 1
BUILD FAILED
C:\wip\mcf\CONNECTORS-1490\connectors\mongodb\build.xml:75: condition satisfied
{code}
> GSOC: MongoDB Output Connector
> --
>
> Key: CONNECTORS-1490
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1490
> Project: ManifoldCF
> Issue Type: New Feature
> Components: MongoDB Output Connector
>Reporter: Piergiorgio Lucidi
>Assignee: Piergiorgio Lucidi
>Priority: Major
> Labels: MongoDB, gsoc2018, java, junit
> Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch,
> mcf-mongodb-connector(CONNECTORS-1490)1.patch,
> mongoDB-connectors-IT-OK-from-Ant.txt, mongodb-ant-test-ok.txt,
> mongodb-output-connection-configuration.PNG
>
> Original Estimate: 480h
> Remaining Estimate: 480h
>
> This is a project idea for [Google Summer of
> Code|https://summerofcode.withgoogle.com/] (GSOC).
> To discuss this or other ideas with your potential mentor from the Apache
> ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list,
> including "[GSOC]" in the subject. You may also comment on this Jira issue if
> you have created an account.
> We would like to extend the Content Migration capabilities adding MongoDB /
> GridFS as a new output connector for importing contents from one or more
> repositories supported by ManifoldCF. In this way we will help developers on
> migrating contents from different data sources on MongoDB.
> You will be involved in the development of the following tasks, you will
> learn how to:
> * Write the connector implementation
> * Implement unit tests
> * Build all the integration tests for testing the connector inside the
> framework
> * Write the documentation for this connector
> We have a complete documentation on how to implement an Output Connector:
> [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html]
> Take a look also at our book to understand better the framework and how to
> implement connectors:
> [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs]
>
> Prospective GSOC mentor:
> [piergior...@apache.org|mailto:piergior...@apache.org]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575098#comment-16575098 ] Olivier Tavard commented on CONNECTORS-1523: No it was not available for 2.10, it was just the first version of the code. If you want more documentation about it you can go there : [https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/237240321/HTML+Extractor+Transformation+connector] Could you give me more details about your question ? You mean if you want to select more than one englobing tab ? > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575089#comment-16575089 ] Steph van Schalkwyk commented on CONNECTORS-1523: - Thank you Olivier. Do you know if it is available in 2.10? Also, does it only filter element_type#id as in div#my_id or does it also filter element_type#css_class ? I'm crawling pages where the html has very few ids. Regards, Steph > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575082#comment-16575082 ] Olivier Tavard commented on CONNECTORS-1523: Hello, I checked it out and Karl already included the patch (r1831269) so I hope that the code will be included for next release of MCF. > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1521) Documentum Connector users ManifoldCF's local time in queries constraints against the Documentum server without reference to time zones
[
https://issues.apache.org/jira/browse/CONNECTORS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575081#comment-16575081
]
James Thomas commented on CONNECTORS-1521:
--
I didn't have the necessary credentials to open support tickets with OpenText,
but I've organised that now, so I'll go ahead and do it.
I don't think we should do a horrible hack :)
And, yes, the consistent timezone approach is the one I'd taken internally
after discovering this issue.
> Documentum Connector users ManifoldCF's local time in queries constraints
> against the Documentum server without reference to time zones
> ---
>
> Key: CONNECTORS-1521
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1521
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
>
> I find that the time/date constraints in queries to the Documentum server are
> based on the "raw" local time of the ManifoldCF server but appear to take no
> account of the time zones of the two servers.
> This can lead to recently modified files not being transferred to the output
> repository when you would naturally expect them to be. I'd like the times to
> be aligned, perhaps by including time zone in the query. In particular, is
> there a way to use UTC perhaps?
> Here's an example ...
> * create a folder in Documentum
> * set up a job to point at the folder and output to the file system
> * put two documents into a folder in Documentum
> * Select them, right click and export as CSV (to show the timestamps):
> {noformat}
> 1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
> 2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
> Check the local time on the ManifoldCF server machine. Observe that it's
> reporting consistent time with the DM server:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 09:07:25 BST 2018{noformat}
> Start the job and look for the query to Documentum in the manifoldcf.log file
> (line break added for readability):
> {noformat}
> DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 00:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('08/07/2018 08:07:34','mm/dd/ hh:mi:ss')
> AND (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0)) AND ( Folder('/Administrator/james', DESCEND) ))
> ^C{noformat}
> Notice that the latest date asked for is *before* the modification date of
> the files added to DM. (And is an hour out, see footnote.)
>
> See whether anything has been output by the File System connector. It hasn't:
> {noformat}
> [james@manifold]$ ls /bigdisc/source/PDFs/timezones/
> [james@manifold]$
> {noformat}
> Now:
> * change the timezone on the ManifoldCF server machine
> * restart the ManifoldCF server and the Documentum processes
> * reseed the job
> Check the local time on the ManifoldCF server machine; it has changed:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 10:10:29 CEST 2018{noformat}
> Start the job again and notice that the query has changed by an hour, plus
> the few minutes it took to change the date etc (and is still an hour out, see
> footnote):
> {noformat}
> r_modify_date<=date('08/07/2018 09:11:02','mm/dd/ hh:mi:ss')
> {noformat}
> Observe that the range of dates now covers the timestamps on the DM data, and
> also that some data has now been transferred by the File System connector:
> {noformat}
> [james@manifold]$ ls
> /bigdisc/source/PDFs/timezones/http/mfserver\:8080/da/component/
> drl?versionLabel=CURRENT&objectId=09018000e515
> drl?versionLabel=CURRENT&objectId=09018000e516
> {noformat}
>
>
> [Footnote] It appears that something is trying to take account of Daylight
> Saving Time too.
> If I set the server date to a time outside of DST, the query is aligned with
> the current time:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Mon Oct 29 00:01:13 CET 2018
> r_modify_date<=date('10/29/2018 00:01:39','mm/dd/ hh:mi:ss')
> {noformat}
> But if I set the time inside DST, the time is an hour before:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Sat Oct 27 00:00:06 CEST 2018
> r_modify_date<=date('10/26/2018 23:00:26','mm/dd/ hh:mi:ss')
> {noformat}
> This is perhaps a Java issue rather than a logic issue in the connector? See
> e.g. [https://stackoverflow.com/questions/6392/java-time-zone-is-messed-up]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
[ https://issues.apache.org/jira/browse/CONNECTORS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575050#comment-16575050 ] Olivier Tavard commented on CONNECTORS-1523: Hello, I submitted a patch about that. See https://issues.apache.org/jira/browse/CONNECTORS-1500. I do not know if the patch was added on the MCF code. > HTML Extractor transformation connector - "No englobing tag specified" > -- > > Key: CONNECTORS-1523 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Priority: Major > > When adding Englobing tag to HTML Extractor transformation, Englobing tag is > not persisted. > Can add on config screen in job edit, but value is not persisted. > Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CONNECTORS-1523) HTML Extractor transformation connector - "No englobing tag specified"
Steph van Schalkwyk created CONNECTORS-1523: --- Summary: HTML Extractor transformation connector - "No englobing tag specified" Key: CONNECTORS-1523 URL: https://issues.apache.org/jira/browse/CONNECTORS-1523 Project: ManifoldCF Issue Type: Bug Affects Versions: ManifoldCF 2.10 Reporter: Steph van Schalkwyk When adding Englobing tag to HTML Extractor transformation, Englobing tag is not persisted. Can add on config screen in job edit, but value is not persisted. Results in "No englobing tag specified". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574948#comment-16574948 ] Piergiorgio Lucidi commented on CONNECTORS-1490: Added the second dependency of Embedded MongoDB in the test-material folder (r1837738). > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, mongodb-ant-test-ok.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piergiorgio Lucidi updated CONNECTORS-1490: --- Attachment: mongodb-ant-test-ok.txt > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, mongodb-ant-test-ok.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574868#comment-16574868 ] Piergiorgio Lucidi commented on CONNECTORS-1490: Fixed in the r1837730. I have added the download of the new test library inside the Ant build script of this connector. Could you please update your code and try on your environments? > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574784#comment-16574784 ] Piergiorgio Lucidi commented on CONNECTORS-1490: Ok I'm fixing also this... > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1522) Add SSL trust certificates list to ElasticSearch output connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574712#comment-16574712 ] Karl Wright commented on CONNECTORS-1522: - [~svanschalkwyk] This is a significant amount of work, and I'm unfortunately severely overcommitted until October-ish. I hope this isn't urgent. If so, you may need to work on this yourself. > Add SSL trust certificates list to ElasticSearch output connector > - > > Key: CONNECTORS-1522 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1522 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Assignee: Karl Wright >Priority: Minor > Fix For: ManifoldCF 2.12 > > > Add "SSL trust certificate list" to Elasticsearch output connector. > Add User Id, Password functionality to ES output connector. > Above as per SOLR output connector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CONNECTORS-1522) Add SSL trust certificates list to ElasticSearch output connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright updated CONNECTORS-1522: Fix Version/s: ManifoldCF 2.12 > Add SSL trust certificates list to ElasticSearch output connector > - > > Key: CONNECTORS-1522 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1522 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Assignee: Karl Wright >Priority: Minor > Fix For: ManifoldCF 2.12 > > > Add "SSL trust certificate list" to Elasticsearch output connector. > Add User Id, Password functionality to ES output connector. > Above as per SOLR output connector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CONNECTORS-1522) Add SSL trust certificates list to ElasticSearch output connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1522: --- Assignee: Karl Wright > Add SSL trust certificates list to ElasticSearch output connector > - > > Key: CONNECTORS-1522 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1522 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Affects Versions: ManifoldCF 2.10 >Reporter: Steph van Schalkwyk >Assignee: Karl Wright >Priority: Minor > > Add "SSL trust certificate list" to Elasticsearch output connector. > Add User Id, Password functionality to ES output connector. > Above as per SOLR output connector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574676#comment-16574676 ] Karl Wright commented on CONNECTORS-1490: - [~piergiorgioluc...@gmail.com], it ran correctly because you'd previously done a "mvn install" for ManifoldCF. > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574635#comment-16574635 ] Piergiorgio Lucidi edited comment on CONNECTORS-1490 at 8/9/18 10:04 AM: - I have added a new repository inside the Maven pom.xml to add a new dependency used to start and stop the MongoDB test instance. The dependency is Embedded MongoDB that is available under Apache License. [https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo] Maybe we only need to add this dependency in the build, but I'm wondering why the ant test ran correctly for me. was (Author: piergiorgioluc...@gmail.com): I have added a new repository inside the Maven pom.xml to add a new dependency used to start and stop the MongoDB test instance. The dependency is Embedded MongoDB that is available under Apache License. [https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo] Maybe we only add this dependency in the build, but I'm wondering why the ant test ran correctly for me. > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574635#comment-16574635 ] Piergiorgio Lucidi commented on CONNECTORS-1490: I have added a new repository inside the Maven pom.xml to add a new dependency used to start and stop the MongoDB test instance. The dependency is Embedded MongoDB that is available under Apache License. [https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo] Maybe we only add this dependency in the build, but I'm wondering why the ant test ran correctly for me. > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574570#comment-16574570
]
Karl Wright commented on CONNECTORS-1490:
-
Hi [~piergiorgioluc...@gmail.com], we have to rethink this.
I executed the following steps:
{code}
ant make-core-deps make-deps
ant test
{code}
This fails because of the following:
{code}
[exec] [ERROR] Failed to execute goal on project mcf-mongodb-connector:
Could not resolve dependencies for project
org.apache.manifoldcf:mcf-mongodb-connector:jar:2.11-SNAPSHOT: The following
artifacts could not be resolved:
org.apache.manifoldcf:mcf-core:jar:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-connector-common:jar:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-agents:jar:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-pull-agent:jar:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-ui-core:jar:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-core:jar:tests:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-agents:jar:tests:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-pull-agent:jar:tests:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-api-service:war:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-authority-service:war:2.11-SNAPSHOT,
org.apache.manifoldcf:mcf-crawler-ui:war:2.11-SNAPSHOT: Could not find artifact
org.apache.manifoldcf:mcf-core:jar:2.11-SNAPSHOT in sonatype-repo
(http://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]
{code}
This is obviously because it's still shelling out to Maven, and it's expecting
the maven build to have been run first. We cannot insure that, and committing
a native ant build seems unreasonable because there are literally hundreds of
dependencies mongodb brings in for testing that we'd have to all download via
ant.
So it seems to me there are two choices. First choice is to simply not run any
Mongodb integration tests under Ant, and only run them under Maven. The second
choice is to revamp the ManifoldCF ant build to use ivy instead of manual
dependency resolution. The second approach is problematic too, though, because
we'd still be distribution a much much larger lib distribution. I don't know
how much larger. We'd also need to figure out how to build a lib distribution
since we'd effectively be replacing the "lib" directory with ivy support.
For now I therefore think the only possibility is disabling the Mongodb
integration tests under Ant. Can you do that?
> GSOC: MongoDB Output Connector
> --
>
> Key: CONNECTORS-1490
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1490
> Project: ManifoldCF
> Issue Type: New Feature
> Components: MongoDB Output Connector
>Reporter: Piergiorgio Lucidi
>Assignee: Piergiorgio Lucidi
>Priority: Major
> Labels: MongoDB, gsoc2018, java, junit
> Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch,
> mcf-mongodb-connector(CONNECTORS-1490)1.patch,
> mongoDB-connectors-IT-OK-from-Ant.txt,
> mongodb-output-connection-configuration.PNG
>
> Original Estimate: 480h
> Remaining Estimate: 480h
>
> This is a project idea for [Google Summer of
> Code|https://summerofcode.withgoogle.com/] (GSOC).
> To discuss this or other ideas with your potential mentor from the Apache
> ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list,
> including "[GSOC]" in the subject. You may also comment on this Jira issue if
> you have created an account.
> We would like to extend the Content Migration capabilities adding MongoDB /
> GridFS as a new output connector for importing contents from one or more
> repositories supported by ManifoldCF. In this way we will help developers on
> migrating contents from different data sources on MongoDB.
> You will be involved in the development of the following tasks, you will
> learn how to:
> * Write the connector implementation
> * Implement unit tests
> * Build all the integration tests for testing the connector inside the
> framework
> * Write the documentation for this connector
> We have a complete documentation on how to implement an Output Connector:
> [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html]
> Take a look also at our book to understand better the framework and how to
> implement connectors:
> [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs]
>
> Prospective GSOC mentor:
> [piergior...@apache.org|mailto:piergior...@apache.org]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574552#comment-16574552 ] Karl Wright commented on CONNECTORS-1490: - Ok, thanks. I'm going to try running the IT from ant here and see if they run natively. > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piergiorgio Lucidi updated CONNECTORS-1490: --- Attachment: mongoDB-connectors-IT-OK-from-Ant.txt > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongoDB-connectors-IT-OK-from-Ant.txt, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574542#comment-16574542 ] Piergiorgio Lucidi commented on CONNECTORS-1490: In the latest r1837702 you will find the fix for integration tests. I have also attached the execution of integration tests started using the ant build. > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1490) GSOC: MongoDB Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574424#comment-16574424 ] Piergiorgio Lucidi commented on CONNECTORS-1490: I'm looking at this and I should commit a fix for this very soon :P > GSOC: MongoDB Output Connector > -- > > Key: CONNECTORS-1490 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1490 > Project: ManifoldCF > Issue Type: New Feature > Components: MongoDB Output Connector >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Labels: MongoDB, gsoc2018, java, junit > Attachments: mcf-mongodb-connector(CONNECTORS-1490).patch, > mcf-mongodb-connector(CONNECTORS-1490)1.patch, > mongodb-output-connection-configuration.PNG > > Original Estimate: 480h > Remaining Estimate: 480h > > This is a project idea for [Google Summer of > Code|https://summerofcode.withgoogle.com/] (GSOC). > To discuss this or other ideas with your potential mentor from the Apache > ManifoldCF project, sign up and post to the dev@manifoldcf.apache.org list, > including "[GSOC]" in the subject. You may also comment on this Jira issue if > you have created an account. > We would like to extend the Content Migration capabilities adding MongoDB / > GridFS as a new output connector for importing contents from one or more > repositories supported by ManifoldCF. In this way we will help developers on > migrating contents from different data sources on MongoDB. > You will be involved in the development of the following tasks, you will > learn how to: > * Write the connector implementation > * Implement unit tests > * Build all the integration tests for testing the connector inside the > framework > * Write the documentation for this connector > We have a complete documentation on how to implement an Output Connector: > [https://manifoldcf.apache.org/release/release-2.9.1/en_US/writing-output-connectors.html] > Take a look also at our book to understand better the framework and how to > implement connectors: > [https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs] > > Prospective GSOC mentor: > [piergior...@apache.org|mailto:piergior...@apache.org] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
