I've been having problems with the merge portion of the script too.
My solution was to check the success status of the merge ( $? ), and
if it failed, try again, or wait until next time.
nutch_bin/nutch mergesegs $merged_segment -dir $segments
if [ $? -ne 0 ]
then
echo "merging segments failed, lets abort now in case of large failure"
echo "this was the main sticking point for some reason"
echo "removing the merged segment in case of fire"
rm -r $merged_segment
exit
else
rm -r $segments/*
mv $merged_segment/* $segments/
rm -r $merged_segment
fi
