No, they are unnecessary. It is quite possible to set #retries to 50000 in fproxy...
On Fri, Aug 12, 2005 at 06:22:11AM -0500, Conrad J. Sabatier wrote:
>
> On 12-Aug-2005 Anonymous via Panta Rhei wrote:
> >
> > I've had a helluva time trying to d/l these and i'd be gratefull for
> > a re-insert by anyone who has them.
> >
> > File: xpcorp.ISO
> > Key: [EMAIL PROTECTED],t0N0QCiz0-a~LL08UL4XtA
> > Bytes: 512360448
> >
> > File: WinXPPlus.ISO
> > Key: [EMAIL PROTECTED],gNGGF1n0TvCRjdzgtA-NRA
> > Bytes:
> >
> > Thank you.
>
> You may be interested in a pair of bash scripts I wrote to help with
> large splitfile downloads.
>
> The first, get_splitfile, will try indefinitely to fetch the minimum
> number of blocks needed to decode a file, looping forever over the list
> of data/check blocks that make up each segment until either the
> required number has been successfully retrieved or a user interrupt.
> It will automatically spawn a number of background fetches equal to the
> number of segments in the splitfile times two (one thread for data
> blocks, one for check blocks).
>
> Note that no actual decoding is done. The idea is to get the raw
> blocks into the local datastore, so that a later request via fproxy
> will succeed more quickly and easily.
>
> Note also that each successfully retrieved block is stored in a
> directory *outside* of your actual freenet datastore, so a little
> additional disk space will be required.
>
> The second script, watch_splitfile, allows you to monitor the progress
> of the first script's operation.
>
> Read the header comments in each file for usage instructions and other
> information.
>
> Enjoy!
>
> P.S. Toad, should we maybe add these to our scripts collection in CVS?
> :-)
>
> --
> Conrad J. Sabatier <[EMAIL PROTECTED]> -- "In Unix veritas"
Content-Description: get_splitfile
> #!/usr/local/bin/bash
> #
> # get_splitfile -- download an FEC splitfile
> #
> # usage: get_splitfile key output_filename
> #
> # Uses a separate thread for each of the splitfile's segments to improve
> # throughput
> #
> # Requires: fcpget (freenet tools) for fetching metadata,
> # fnget (zzed's freenet tools) for fetching the actual data blocks
> #
> # Note: doesn't actually decode the file, but just downloads the blocks so
> they'
> ll
> # (hopefully) be ready and available in the local datastore when a real
> attempt
> to
> # download/decode the file is done via fproxy
> #
>
> HTL=$(awk -F'=' '/^%?maxHopsToLive=/ {print $2}' ../freenet/freenet.conf)
>
> ##############################################
> #
> # Subroutine definitions precede main program
> #
> ##############################################
>
> #
> # usage - display correct command line usage on stderr
> #
>
> usage()
> {
> {
> echo
> echo "usage: get_splitfile key output_filename"
> echo
> } >&2
> }
>
> ####################################
> #
> # number format conversion routines
> #
> ####################################
>
> #
> # d2h - convert unsigned decimal to hex
> #
>
> d2h()
> {
> local d=$1
> local h=""
> local nybble
>
> while [ ${d} -ne 0 ]
> do
> nybble=$((d & 15))
> case ${nybble} in
> 10) nybble="a";;
> 11) nybble="b";;
> 12) nybble="c";;
> 13) nybble="d";;
> 14) nybble="e";;
> 15) nybble="f";;
> *) nybble="${nybble}";;
> esac
> h=${nybble}${h}
> ((d >>= 4))
> done
>
> echo ${h}
> }
>
> #
> # h2d - convert hex string (without any leading "0x") to decimal
> #
>
> h2d()
> {
> local h=$1
> local d=0
> local nybble
>
> while [ -n "${h}" ]
> do
> nybble=${h:0:1}
> case ${nybble} in
> a) nybble=10;;
> b) nybble=11;;
> c) nybble=12;;
> d) nybble=13;;
> e) nybble=14;;
> f) nybble=15;;
> *) ;;
> esac
> d=$(((d << 4) + nybble))
> h=${h:1}
> done
>
> echo ${d}
> }
>
> #####################
> #
> # Begin main program
> #
> #####################
>
> # Check command line, exit if incorrect
>
> if [ ${#} -ne 2 ]
> then
> usage
> exit 1
> fi
>
> key="${1}"
>
> output_filename="${2}"
>
> # Import variables from spider
> #
> # Note: if not using spider, simply define the variables FCPGET (full path to
> # freenet tools' fcpget program) and FCP_HOST (the address of the node to
> connec
> t to)
> #
>
> . ${HOME}/freenet/spider/spider.vars
>
> # Prepend "freenet:" to key if necessary
>
> if [[ "${key#freenet:*}" == "${key}" ]]
> then
> key="freenet:${key}"
> fi
>
> # Replace slashes in key with colons to create filename (without leading
> # "freenet:") prefix for metadata file and segment files
>
> filename_prefix="$(echo "${key:8}" | sed -e 's#/#:#g')"
>
> # Create a temporary directory for downloaded data
>
> tempdir="${output_filename}.tmp"
> mkdir -p "${tempdir}"
>
> cd "${tempdir}"
>
> # Get the raw metadata for the file
>
> metadata_file="${output_filename}.metadata"
>
> while [[ ! -s "${metadata_file}" ]]
> do
> ${FCPGET} -n ${FCP_HOST} -R -l ${HTL} -v 3 "${key}" "${metadata_file}"
> || exit
> 1
> done
>
> # Get the splitfile segment map info
>
> FECSegmentSplitFile "${metadata_file}" "${output_filename}" || exit 1
>
> numsegs=$(find . -type f -name "${output_filename}.segment.*.header" | wc -l)
>
> # Start two background processes for each segment,
> # one to download the data blocks, and
> # one to download the check blocks
>
> for ((i=0; i<${numsegs}; ++i))
> do
> mkdir -p "segment.${i}"
>
> # read info from segment header
>
> BlocksRequired[$i]=$(h2d $(awk -F'=' '/^BlocksRequired=/ {print $2}'
> ${output_f
> ilename}.segment.${i}.header))
> BlockSize[$i]=$(h2d $(awk -F'=' '/^BlockSize=/ {print $2}'
> ${output_filename}.s
> egment.${i}.header))
> BlockCount[$i]=$(h2d $(awk -F'=' '/^BlockCount=/ {print $2}'
> ${output_filename}
> .segment.${i}.header))
> CheckBlockCount[$i]=$(h2d $(awk -F'=' '/^CheckBlockCount=/ {print $2}'
> ${output
> _filename}.segment.${i}.header))
> CheckBlockSize[$i]=$(h2d $(awk -F'=' '/^CheckBlockSize=/ {print $2}'
> ${output_f
> ilename}.segment.${i}.header))
>
> # get number of blocks already saved
>
> Blocks[$i]=$(find segment.${i} -type f -name "Block.*" ! -empty | wc -l)
> Checks[$i]=$(find segment.${i} -type f -name "Check.*" ! -empty | wc -l)
>
> # loop for reading data blocks
> #
> # do repeated passes through the list of blocks
> # until the required number of blocks have been read
>
> while ((${Blocks[$i]} < ${BlockCount[$i]})) &&
> ((${Blocks[$i]} + ${Checks[$i]} <
> ${BlocksRequired[$i]}))
> do
> while read -r
> do
> if [ ${Blocks[$i]} -ge ${BlockCount[$i]} ] ||
> [ $((${Blocks[$i]} + ${Checks[$i]})) -ge
> ${BlocksRequired[$i]} ]
> then
> break
> fi
>
> if [[ "${REPLY%%.*}" == "Block" ]]
> then
> block_name=${REPLY%=*}
> block_key=${REPLY#*=}
>
> # don't try fetching a block we've already got
>
> if [[ $(stat -f "%z"
> "segment.${i}/${block_name}" 2>/dev/null) -ne ${BlockSi
> ze[$i]} ]]
> then
> ${FNGET} -s ${FCP_HOST} -H ${HTL} -O
> "segment.${i}/${block_name}" "${block_
> key}"
>
> # old ft version
> #${FCPGET} -n ${FCP_HOST} -l ${HTL} -v
> 3 ${block_key} "segment.${i}/${block
> _name}"
> fi
> fi
>
> # update block count for next loop conditional test
>
> Blocks[$i]=$(find segment.${i} -type f -name "Block.*"
> ! -empty | wc -l)
> Checks[$i]=$(find segment.${i} -type f -name "Check.*"
> ! -empty | wc -l)
> done < "${output_filename}.segment.${i}.blockmap"
>
> # update block count for next loop conditional test
>
> Blocks[$i]=$(find segment.${i} -type f -name "Block.*" ! -empty
> | wc -l)
> Checks[$i]=$(find segment.${i} -type f -name "Check.*" ! -empty
> | wc -l)
> done &
>
> # loop for reading check blocks
>
> while ((${Checks[$i]} < ${CheckBlockCount[$i]})) &&
> ((${Blocks[$i]} + ${Checks[$i]} <
> ${BlocksRequired[$i]}))
> do
> while read -r
> do
> if [ ${Checks[$i]} -ge ${CheckBlockCount[$i]} ] ||
> [ $((${Blocks[$i]} + ${Checks[$i]})) -ge
> ${BlocksRequired[$i]} ]
> then
> break
> fi
>
>
> if [[ "${REPLY%%.*}" == "Check" ]]
> then
> block_name=${REPLY%=*}
> block_key=${REPLY#*=}
>
> # don't try fetching a block we've already got
>
> if [[ $(stat -f "%z"
> "segment.${i}/${block_name}" 2>/dev/null) -ne ${CheckBl
> ockSize[$i]} ]]
> then
> ${FNGET} -s ${FCP_HOST} -H ${HTL} -O
> "segment.${i}/${block_name}" "${block_
> key}"
>
> # old ft version
> #${FCPGET} -n ${FCP_HOST} -l ${HTL} -v
> 3 ${block_key} "segment.${i}/${block
> _name}"
> fi
> fi
>
> # update block count for next loop conditional test
>
> Blocks[$i]=$(find segment.${i} -type f -name "Block.*"
> ! -empty | wc -l)
> Checks[$i]=$(find segment.${i} -type f -name "Check.*"
> ! -empty | wc -l)
> done < "${output_filename}.segment.${i}.blockmap"
>
> # update block counts for next loop conditional test
>
> Blocks[$i]=$(find segment.${i} -type f -name "Block.*" ! -empty
> | wc -l)
> Checks[$i]=$(find segment.${i} -type f -name "Check.*" ! -empty
> | wc -l)
> done &
> done
Content-Description: watch_splitfile
> #!/usr/local/bin/bash
> #
> # watch_splitfile -- monitor the activity of a get_splitfile in progress
> #
> # must first do a "cd" to the directory where the splitfile is being
> downloaded
> # before running this script
> #
>
> numsegs=$(ls *.segment.*.header | wc -l)
>
> totalBlocksRequired=0
>
> for ((i=0; i<$numsegs; ++i))
> do
> BlocksRequired[$i]=$(h2d $(awk -F'=' '/BlocksRequired=/ {print $2}'
> *.segment.$
> {i}.header))
> totalBlocksRequired=$((totalBlocksRequired + ${BlocksRequired[$i]}))
> done
>
> while :
> do
> clear
> for ((d=0; d<$numsegs; ++d))
> do
> blocks=$(find segment.${d} -type f -name "Block.*" ! -empty |
> wc -l)
> checks=$(find segment.${d} -type f -name "Check.*" ! -empty |
> wc -l)
> percent=$(echo "((${blocks} + ${checks}) /
> ${BlocksRequired[$d]}) * 100" | bc
> -l)
>
> printf "segment.%d: %3d data + %3d check = %3d blocks found\t
> (%5.2f%% of %3d)
> \n" \
> ${d} ${blocks} ${checks} $((${blocks} + ${checks}))
> ${percent} ${BlocksRequir
> ed[$d]}
> done
>
> total_data_found=$(find segment.* -type f -name "Block.*" ! -empty | wc
> -l)
> total_check_found=$(find segment.* -type f -name "Check.*" ! -empty |
> wc -l)
>
> total_found=$((total_data_found + total_check_found))
> total_remaining=$((totalBlocksRequired - total_found))
>
> total_found_percent=$(echo "(${total_found} / ${totalBlocksRequired}) *
> 100" |
> bc -l)
> total_remaining_percent=$(echo "(${total_remaining} /
> ${totalBlocksRequired}) *
> 100" | bc -l)
>
> printf "Total: %7d data + %3d check = %3d blocks found\t (%5.2f%% of
> %3d)\n\n"
> \
> ${total_data_found} ${total_check_found} ${total_found}
> ${total_found_percent
> } ${totalBlocksRequired}
>
> printf "Total blocks required: %6d\n" ${totalBlocksRequired}
> printf "Total blocks found: %9d (%5.2f%%)\n" ${total_found}
> ${total_found_perce
> nt}
> printf "Total blocks remaining: %5d (%5.2f%%)\n\n" ${total_remaining}
> ${total_r
> emaining_percent}
>
> echo -e "Currently downloading:\n"
> ps axww | grep -o 'segment[^ ]* freenet:[EMAIL PROTECTED]' | sort
> sleep 30
> done
> _______________________________________________
> Support mailing list
> Support@freenetproject.org
> http://news.gmane.org/gmane.network.freenet.support
> Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support
> Or mailto:[EMAIL PROTECTED]
--
Matthew J Toseland - [EMAIL PROTECTED]
Freenet Project Official Codemonkey - http://freenetproject.org/
ICTHUS - Nothing is impossible. Our Boss says so.
signature.asc
Description: Digital signature
_______________________________________________ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
