[
https://issues.apache.org/jira/browse/NUTCH-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114128#comment-18114128
]
ASF GitHub Bot commented on NUTCH-3197:
---------------------------------------
sebastian-nagel commented on code in PR #958:
URL: https://github.com/apache/nutch/pull/958#discussion_r3986597317
##########
src/bin/nutch:
##########
@@ -147,17 +147,18 @@ fi
local=true
-# NUTCH_JOB
-if [ -f "${NUTCH_HOME}"/*nutch*.job ]; then
- local=false
- for f in "$NUTCH_HOME"/*nutch*.job; do
+# NUTCH_JOB
+for f in "${NUTCH_HOME}"/*nutch*.job; do
Review Comment:
If there are two or more job files in NUTCH_HOME, that's already an issue
very likely. But ok for this fix. If we are lucky the last job file is the
right one.
##########
src/bin/nutch:
##########
@@ -336,7 +337,7 @@ else
# check that hadoop can be found on the path
if [ $(which hadoop | wc -l ) -eq 0 ]; then
echo "Can't find Hadoop executable. Add HADOOP_COMMON_HOME/bin to the path
or run in local mode."
- exit -1;
Review Comment:
Maybe we should change the behavior of Generator and exit with a more unique
code, in case no segment was created (nothing to fetch). But would be a
breaking change.
##########
src/bin/nutch:
##########
@@ -336,7 +337,7 @@ else
# check that hadoop can be found on the path
if [ $(which hadoop | wc -l ) -eq 0 ]; then
echo "Can't find Hadoop executable. Add HADOOP_COMMON_HOME/bin to the path
or run in local mode."
- exit -1;
Review Comment:
This is somewhat sensitive because the script bin/crawl expects that
`bin/nutch generate ...` exits with code 1 when no segment was created. All
errors should ideally use other error codes.
Of course, this is not 100% working, because the Generator uses Hadoop's
ToolRunner which maps exceptions (don't whether all) to 1.
> Yetus: fix the real problems the precommit gate correctly reported
> ------------------------------------------------------------------
>
> Key: NUTCH-3197
> URL: https://issues.apache.org/jira/browse/NUTCH-3197
> Project: Nutch
> Issue Type: Sub-task
> Reporter: Lewis John McGibbney
> Priority: Major
>
> After baselines, the remaining interesting failures are things we would want
> fixed even without Yetus: misspellings in Java/docs that are not deliberate
> test inputs; eleven shellcheck *errors* in src/bin/crawl and src/bin/nutch
> (bad exit usage, glob/-f mistakes, string-vs-numeric compare, "$@"
> mishandling); Dockerfile structure issues; missing @audience/@stability
> shelldocs on crawl helpers; and a single jshint style nit in the parse-js
> sample.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
