[jira] [Commented] (NUTCH-2292) Mavenize the build for nutch-core and nutch-plugins

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782482#comment-16782482
 ] 

Lewis John McGibbney commented on NUTCH-2292:
-

Hi [~wastl-nagel] long story short... we need to rebase this against master and 
then talk through the issue. As it was a while ago, I've all but forgotten what 
decisions we made at the time and what the implications are/were.

What about we propose this as a GSoC project?

> Mavenize the build for nutch-core and nutch-plugins
> ---
>
> Key: NUTCH-2292
> URL: https://issues.apache.org/jira/browse/NUTCH-2292
> Project: Nutch
>  Issue Type: Improvement
>  Components: build
>Reporter: Thamme Gowda
>Assignee: Thamme Gowda
>Priority: Major
> Fix For: 1.16
>
>
> Convert the build system of  nutch-core as well as plugins to Apache Maven.
> *Plan :*
> Create multi-module maven project with the following structure
> {code}
> nutch-parent
>   |-- pom.xml (POM)
>   |-- nutch-core
>   |   |-- pom.xml (JAR)
>   |   |--src: sources
>   |-- nutch-plugins
>   |-- pom.xml (POM)
>   |-- plugin1
>   ||-- pom.xml (JAR)
>   | .
>   |-- pluginN
>|-- pom.xml (JAR)
> {code}
> NOTE: watch out for cyclic dependencies bwteen nutch-core and plugins, 
> introduce another POM to break the cycle if required.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NUTCH-2698) Remove sonar build task from build.xml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782479#comment-16782479
 ] 

ASF GitHub Bot commented on NUTCH-2698:
---

lewismc commented on pull request #443: NUTCH-2698 Remove sonar build task from 
build.xml
URL: https://github.com/apache/nutch/pull/443
 
 
   This issue addresses https://issues.apache.org/jira/browse/NUTCH-2698
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Remove sonar build task from build.xml
> --
>
> Key: NUTCH-2698
> URL: https://issues.apache.org/jira/browse/NUTCH-2698
> Project: Nutch
>  Issue Type: Task
>  Components: build
>Affects Versions: 1.15
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Major
> Fix For: 1.16
>
>
> build.xml currently has the following content
> {code}
>   
>   
>   
>   
>   
> 
> 
>   
>   
>   
> 
> 
> 
>  />
> 
> 
> 
>   version="1.4-SNAPSHOT" xmlns:sonar="antlib:org.sonar.ant"/>
>   
> {code}
> We should simply remove as it is defunct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NUTCH-2698) Remove sonar build task from build.xml

[Lewis John McGibbney (JIRA)]

Lewis John McGibbney created NUTCH-2698:
---

 Summary: Remove sonar build task from build.xml
 Key: NUTCH-2698
 URL: https://issues.apache.org/jira/browse/NUTCH-2698
 Project: Nutch
  Issue Type: Task
  Components: build
Affects Versions: 1.15
Reporter: Lewis John McGibbney
Assignee: Lewis John McGibbney
 Fix For: 1.16


build.xml currently has the following content
{code}
  
  
  

  
  


  

  
  











  
{code}

We should simply remove as it is defunct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NUTCH-2697) Upgrade Ivy to fix the issue of an unset packaging.type property.

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782477#comment-16782477
 ] 

Lewis John McGibbney commented on NUTCH-2697:
-

Apologies folks, thank you [~wastl-nagel] for reverting. This has prompted me 
to look at another part of build.xml for a fix so I will go ahead and submit 
that.

> Upgrade Ivy to fix the issue of an unset packaging.type property.
> -
>
> Key: NUTCH-2697
> URL: https://issues.apache.org/jira/browse/NUTCH-2697
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.16
>Reporter: Chris Gavin
>Priority: Major
> Fix For: 1.16
>
>
> Currently Nutch fails to build from a clean checkout due to 
> {{packaging.type}} not being set (even with the current workaround in 
> {{ivysettings.xml}}).
> {code:java}
> [ivy:resolve] :: problems summary ::
> [ivy:resolve]  WARNINGS
> [ivy:resolve] [FAILED ] 
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}: (0ms)
> [ivy:resolve]  local: tried
> [ivy:resolve] 
> /opt/work/.ivy2/local/javax.ws.rs/javax.ws.rs-api/2.1.1/${packaging.type}s/javax.ws.rs-api.${packaging.type}
> [ivy:resolve]  maven2: tried
> [ivy:resolve] 
> http://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve]  apache-snapshot: tried
> [ivy:resolve] 
> https://repository.apache.org/content/repositories/snapshots/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve]  sonatype: tried
> [ivy:resolve] 
> http://oss.sonatype.org/content/repositories/releases/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve] ::
> [ivy:resolve] :: FAILED DOWNLOADS ::
> [ivy:resolve] :: ^ see resolution messages for details ^ ::
> [ivy:resolve] ::
> [ivy:resolve] :: 
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [ivy:resolve] ::
> [ivy:resolve] 
> BUILD FAILED{code}
> This issue has been fixed in the latest version of Ivy so upgrading will 
> cause the build to work correctly again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NUTCH-2697) Upgrade Ivy to fix the issue of an unset packaging.type property.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782425#comment-16782425
 ] 

Hudson commented on NUTCH-2697:
---

SUCCESS: Integrated in Jenkins build Nutch-trunk #3615 (See 
[https://builds.apache.org/job/Nutch-trunk/3615/])
Revert "NUTCH-2697: Upgrade Ivy to fix the issue of an unset (github: 
[https://github.com/apache/nutch/commit/5fc56b62d866af00befcae022718c9e9f879eec3])
* (edit) ivy/ivysettings.xml
* (edit) default.properties


> Upgrade Ivy to fix the issue of an unset packaging.type property.
> -
>
> Key: NUTCH-2697
> URL: https://issues.apache.org/jira/browse/NUTCH-2697
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.16
>Reporter: Chris Gavin
>Priority: Major
> Fix For: 1.16
>
>
> Currently Nutch fails to build from a clean checkout due to 
> {{packaging.type}} not being set (even with the current workaround in 
> {{ivysettings.xml}}).
> {code:java}
> [ivy:resolve] :: problems summary ::
> [ivy:resolve]  WARNINGS
> [ivy:resolve] [FAILED ] 
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}: (0ms)
> [ivy:resolve]  local: tried
> [ivy:resolve] 
> /opt/work/.ivy2/local/javax.ws.rs/javax.ws.rs-api/2.1.1/${packaging.type}s/javax.ws.rs-api.${packaging.type}
> [ivy:resolve]  maven2: tried
> [ivy:resolve] 
> http://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve]  apache-snapshot: tried
> [ivy:resolve] 
> https://repository.apache.org/content/repositories/snapshots/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve]  sonatype: tried
> [ivy:resolve] 
> http://oss.sonatype.org/content/repositories/releases/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve] ::
> [ivy:resolve] :: FAILED DOWNLOADS ::
> [ivy:resolve] :: ^ see resolution messages for details ^ ::
> [ivy:resolve] ::
> [ivy:resolve] :: 
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [ivy:resolve] ::
> [ivy:resolve] 
> BUILD FAILED{code}
> This issue has been fixed in the latest version of Ivy so upgrading will 
> cause the build to work correctly again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : Nutch-trunk #3615

[Apache Jenkins Server]

See 

[jira] [Commented] (NUTCH-2292) Mavenize the build for nutch-core and nutch-plugins

[Sebastian Nagel (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782415#comment-16782415
 ] 

Sebastian Nagel commented on NUTCH-2292:


Hi [~lewismc], hi [~thammegowda],
I've tried to rebase the branch to the recent master, it wasn't successful: 
there are more than 40 conflicts already for the first commit on NUTCH-2292. I 
gave up after one hour trying to resolve them, I expect more conflicts coming 
up with the next commits applied on top of the current master. Even if the 
conflicts are resolved we would need to port manually all changes to ant and 
ivy files (all dependency upgrades and additions). Given that the last 
merge/rebase of master and NUTCH-2292 dates 2 years back and there have been 
significant changes to master since then (mainly NUTCH-2375, NUTCH-2583 and 
many new plugins), it might be less painful to pick the pom.xml and other new 
files and do the remaining changes again, mainly move source files into the 
directories required by Maven.

I have a couple of questions:
- [10b23cd0](https://github.com/commoncrawl/nutch/commit/10b23cd0) removes all 
calls of {{System.exit(jobResult)}}. How are the results of the Nutch jobs 
(inject, generate, fetch, parse, index, etc.) signalized to the calling script 
(bin/crawl) to control the crawl flow? If a step fails the workflow needs to be 
stopped. Some tools even signalize by the exit value that there is nothing to 
do, e.g. if the fetch list is empty.
- are there any notes how to proceed? Are the following points are still open, 
resp. is there a draft/plan/implementation how to address them?
-- NUTCH-2293 (integration test), e.g.,  cannot seriously test Fetcher without 
a protocol plugin already built
-- selection of plugins for installation
-- build the binary package
-- assemble the job file (required if run on Hadoop)

Thanks, great work! Would be good to have it finally, esp. given that the ant 
builds on Jenkins are failing most of the time (cf. NUTCH-2669, NUTCH-2672, 
NUTCH-2697)

> Mavenize the build for nutch-core and nutch-plugins
> ---
>
> Key: NUTCH-2292
> URL: https://issues.apache.org/jira/browse/NUTCH-2292
> Project: Nutch
>  Issue Type: Improvement
>  Components: build
>Reporter: Thamme Gowda
>Assignee: Thamme Gowda
>Priority: Major
> Fix For: 1.16
>
>
> Convert the build system of  nutch-core as well as plugins to Apache Maven.
> *Plan :*
> Create multi-module maven project with the following structure
> {code}
> nutch-parent
>   |-- pom.xml (POM)
>   |-- nutch-core
>   |   |-- pom.xml (JAR)
>   |   |--src: sources
>   |-- nutch-plugins
>   |-- pom.xml (POM)
>   |-- plugin1
>   ||-- pom.xml (JAR)
>   | .
>   |-- pluginN
>|-- pom.xml (JAR)
> {code}
> NOTE: watch out for cyclic dependencies bwteen nutch-core and plugins, 
> introduce another POM to break the cycle if required.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NUTCH-2697) Upgrade Ivy to fix the issue of an unset packaging.type property.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782403#comment-16782403
 ] 

ASF GitHub Bot commented on NUTCH-2697:
---

sebastian-nagel commented on issue #441: NUTCH-2697: Upgrade Ivy to fix the 
issue of an unset packaging.type p…
URL: https://github.com/apache/nutch/pull/441#issuecomment-468926977
 
 
   Hi @lewismc, I've reverted the commit because the update to ivy-2.5.0-rc1 
causes [NUTCH-2672](https://issues.apache.org/jira/browse/NUTCH-2672) which is 
worse because there is no work-around and the builds will fail everywhere. I've 
just verified on my laptop that NUTCH-2672 is still an issue when using 
ivy-2.5.0-rc1.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Upgrade Ivy to fix the issue of an unset packaging.type property.
> -
>
> Key: NUTCH-2697
> URL: https://issues.apache.org/jira/browse/NUTCH-2697
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.16
>Reporter: Chris Gavin
>Priority: Major
> Fix For: 1.16
>
>
> Currently Nutch fails to build from a clean checkout due to 
> {{packaging.type}} not being set (even with the current workaround in 
> {{ivysettings.xml}}).
> {code:java}
> [ivy:resolve] :: problems summary ::
> [ivy:resolve]  WARNINGS
> [ivy:resolve] [FAILED ] 
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}: (0ms)
> [ivy:resolve]  local: tried
> [ivy:resolve] 
> /opt/work/.ivy2/local/javax.ws.rs/javax.ws.rs-api/2.1.1/${packaging.type}s/javax.ws.rs-api.${packaging.type}
> [ivy:resolve]  maven2: tried
> [ivy:resolve] 
> http://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve]  apache-snapshot: tried
> [ivy:resolve] 
> https://repository.apache.org/content/repositories/snapshots/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve]  sonatype: tried
> [ivy:resolve] 
> http://oss.sonatype.org/content/repositories/releases/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve] ::
> [ivy:resolve] :: FAILED DOWNLOADS ::
> [ivy:resolve] :: ^ see resolution messages for details ^ ::
> [ivy:resolve] ::
> [ivy:resolve] :: 
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [ivy:resolve] ::
> [ivy:resolve] 
> BUILD FAILED{code}
> This issue has been fixed in the latest version of Ivy so upgrading will 
> cause the build to work correctly again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NUTCH-2697) Upgrade Ivy to fix the issue of an unset packaging.type property.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782402#comment-16782402
 ] 

ASF GitHub Bot commented on NUTCH-2697:
---

sebastian-nagel commented on pull request #442: Revert "NUTCH-2697: Upgrade Ivy 
to fix the issue of an unset packaging.type p…"
URL: https://github.com/apache/nutch/pull/442
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Upgrade Ivy to fix the issue of an unset packaging.type property.
> -
>
> Key: NUTCH-2697
> URL: https://issues.apache.org/jira/browse/NUTCH-2697
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.16
>Reporter: Chris Gavin
>Priority: Major
> Fix For: 1.16
>
>
> Currently Nutch fails to build from a clean checkout due to 
> {{packaging.type}} not being set (even with the current workaround in 
> {{ivysettings.xml}}).
> {code:java}
> [ivy:resolve] :: problems summary ::
> [ivy:resolve]  WARNINGS
> [ivy:resolve] [FAILED ] 
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}: (0ms)
> [ivy:resolve]  local: tried
> [ivy:resolve] 
> /opt/work/.ivy2/local/javax.ws.rs/javax.ws.rs-api/2.1.1/${packaging.type}s/javax.ws.rs-api.${packaging.type}
> [ivy:resolve]  maven2: tried
> [ivy:resolve] 
> http://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve]  apache-snapshot: tried
> [ivy:resolve] 
> https://repository.apache.org/content/repositories/snapshots/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve]  sonatype: tried
> [ivy:resolve] 
> http://oss.sonatype.org/content/repositories/releases/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve] ::
> [ivy:resolve] :: FAILED DOWNLOADS ::
> [ivy:resolve] :: ^ see resolution messages for details ^ ::
> [ivy:resolve] ::
> [ivy:resolve] :: 
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [ivy:resolve] ::
> [ivy:resolve] 
> BUILD FAILED{code}
> This issue has been fixed in the latest version of Ivy so upgrading will 
> cause the build to work correctly again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NUTCH-2697) Upgrade Ivy to fix the issue of an unset packaging.type property.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782400#comment-16782400
 ] 

ASF GitHub Bot commented on NUTCH-2697:
---

sebastian-nagel commented on pull request #442: Revert "NUTCH-2697: Upgrade Ivy 
to fix the issue of an unset packaging.type p…"
URL: https://github.com/apache/nutch/pull/442
 
 
   Reverts apache/nutch#441 because using ivy-2.5.0-rc1 causes 
[NUTCH-2672](https://issues.apache.org/jira/browse/NUTCH-2672).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Upgrade Ivy to fix the issue of an unset packaging.type property.
> -
>
> Key: NUTCH-2697
> URL: https://issues.apache.org/jira/browse/NUTCH-2697
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.16
>Reporter: Chris Gavin
>Priority: Major
> Fix For: 1.16
>
>
> Currently Nutch fails to build from a clean checkout due to 
> {{packaging.type}} not being set (even with the current workaround in 
> {{ivysettings.xml}}).
> {code:java}
> [ivy:resolve] :: problems summary ::
> [ivy:resolve]  WARNINGS
> [ivy:resolve] [FAILED ] 
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}: (0ms)
> [ivy:resolve]  local: tried
> [ivy:resolve] 
> /opt/work/.ivy2/local/javax.ws.rs/javax.ws.rs-api/2.1.1/${packaging.type}s/javax.ws.rs-api.${packaging.type}
> [ivy:resolve]  maven2: tried
> [ivy:resolve] 
> http://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve]  apache-snapshot: tried
> [ivy:resolve] 
> https://repository.apache.org/content/repositories/snapshots/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve]  sonatype: tried
> [ivy:resolve] 
> http://oss.sonatype.org/content/repositories/releases/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> [ivy:resolve] ::
> [ivy:resolve] :: FAILED DOWNLOADS ::
> [ivy:resolve] :: ^ see resolution messages for details ^ ::
> [ivy:resolve] ::
> [ivy:resolve] :: 
> javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}
> [ivy:resolve] ::
> [ivy:resolve] 
> BUILD FAILED{code}
> This issue has been fixed in the latest version of Ivy so upgrading will 
> cause the build to work correctly again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)