On 12-Aug-2005 Anonymous via Panta Rhei wrote:
>
> I've had a helluva time trying to d/l these and i'd be gratefull for
> a re-insert by anyone who has them.
>
> File: xpcorp.ISO
> Key: [EMAIL PROTECTED],t0N0QCiz0-a~LL08UL4XtA
> Bytes: 512360448
>
> File: WinXPPlus.ISO
> Key: [EMAIL PROTECTED],gNGGF1n0TvCRjdzgtA-NRA
> Bytes:
>
> Thank you.
You may be interested in a pair of bash scripts I wrote to help with
large splitfile downloads.
The first, get_splitfile, will try indefinitely to fetch the minimum
number of blocks needed to decode a file, looping forever over the list
of data/check blocks that make up each segment until either the
required number has been successfully retrieved or a user interrupt.
It will automatically spawn a number of background fetches equal to the
number of segments in the splitfile times two (one thread for data
blocks, one for check blocks).
Note that no actual decoding is done. The idea is to get the raw
blocks into the local datastore, so that a later request via fproxy
will succeed more quickly and easily.
Note also that each successfully retrieved block is stored in a
directory *outside* of your actual freenet datastore, so a little
additional disk space will be required.
The second script, watch_splitfile, allows you to monitor the progress
of the first script's operation.
Read the header comments in each file for usage instructions and other
information.
Enjoy!
P.S. Toad, should we maybe add these to our scripts collection in CVS?
:-)
--
Conrad J. Sabatier <[EMAIL PROTECTED]> -- "In Unix veritas"
#!/usr/local/bin/bash
#
# get_splitfile -- download an FEC splitfile
#
# usage: get_splitfile key output_filename
#
# Uses a separate thread for each of the splitfile's segments to improve
# throughput
#
# Requires: fcpget (freenet tools) for fetching metadata,
# fnget (zzed's freenet tools) for fetching the actual data blocks
#
# Note: doesn't actually decode the file, but just downloads the blocks so they'
ll
# (hopefully) be ready and available in the local datastore when a real attempt
to
# download/decode the file is done via fproxy
#
HTL=$(awk -F'=' '/^%?maxHopsToLive=/ {print $2}' ../freenet/freenet.conf)
##############################################
#
# Subroutine definitions precede main program
#
##############################################
#
# usage - display correct command line usage on stderr
#
usage()
{
{
echo
echo "usage: get_splitfile key output_filename"
echo
} >&2
}
####################################
#
# number format conversion routines
#
####################################
#
# d2h - convert unsigned decimal to hex
#
d2h()
{
local d=$1
local h=""
local nybble
while [ ${d} -ne 0 ]
do
nybble=$((d & 15))
case ${nybble} in
10) nybble="a";;
11) nybble="b";;
12) nybble="c";;
13) nybble="d";;
14) nybble="e";;
15) nybble="f";;
*) nybble="${nybble}";;
esac
h=${nybble}${h}
((d >>= 4))
done
echo ${h}
}
#
# h2d - convert hex string (without any leading "0x") to decimal
#
h2d()
{
local h=$1
local d=0
local nybble
while [ -n "${h}" ]
do
nybble=${h:0:1}
case ${nybble} in
a) nybble=10;;
b) nybble=11;;
c) nybble=12;;
d) nybble=13;;
e) nybble=14;;
f) nybble=15;;
*) ;;
esac
d=$(((d << 4) + nybble))
h=${h:1}
done
echo ${d}
}
#####################
#
# Begin main program
#
#####################
# Check command line, exit if incorrect
if [ ${#} -ne 2 ]
then
usage
exit 1
fi
key="${1}"
output_filename="${2}"
# Import variables from spider
#
# Note: if not using spider, simply define the variables FCPGET (full path to
# freenet tools' fcpget program) and FCP_HOST (the address of the node to connec
t to)
#
. ${HOME}/freenet/spider/spider.vars
# Prepend "freenet:" to key if necessary
if [[ "${key#freenet:*}" == "${key}" ]]
then
key="freenet:${key}"
fi
# Replace slashes in key with colons to create filename (without leading
# "freenet:") prefix for metadata file and segment files
filename_prefix="$(echo "${key:8}" | sed -e 's#/#:#g')"
# Create a temporary directory for downloaded data
tempdir="${output_filename}.tmp"
mkdir -p "${tempdir}"
cd "${tempdir}"
# Get the raw metadata for the file
metadata_file="${output_filename}.metadata"
while [[ ! -s "${metadata_file}" ]]
do
${FCPGET} -n ${FCP_HOST} -R -l ${HTL} -v 3 "${key}" "${metadata_file}"
|| exit
1
done
# Get the splitfile segment map info
FECSegmentSplitFile "${metadata_file}" "${output_filename}" || exit 1
numsegs=$(find . -type f -name "${output_filename}.segment.*.header" | wc -l)
# Start two background processes for each segment,
# one to download the data blocks, and
# one to download the check blocks
for ((i=0; i<${numsegs}; ++i))
do
mkdir -p "segment.${i}"
# read info from segment header
BlocksRequired[$i]=$(h2d $(awk -F'=' '/^BlocksRequired=/ {print $2}'
${output_f
ilename}.segment.${i}.header))
BlockSize[$i]=$(h2d $(awk -F'=' '/^BlockSize=/ {print $2}'
${output_filename}.s
egment.${i}.header))
BlockCount[$i]=$(h2d $(awk -F'=' '/^BlockCount=/ {print $2}'
${output_filename}
.segment.${i}.header))
CheckBlockCount[$i]=$(h2d $(awk -F'=' '/^CheckBlockCount=/ {print $2}'
${output
_filename}.segment.${i}.header))
CheckBlockSize[$i]=$(h2d $(awk -F'=' '/^CheckBlockSize=/ {print $2}'
${output_f
ilename}.segment.${i}.header))
# get number of blocks already saved
Blocks[$i]=$(find segment.${i} -type f -name "Block.*" ! -empty | wc -l)
Checks[$i]=$(find segment.${i} -type f -name "Check.*" ! -empty | wc -l)
# loop for reading data blocks
#
# do repeated passes through the list of blocks
# until the required number of blocks have been read
while ((${Blocks[$i]} < ${BlockCount[$i]})) &&
((${Blocks[$i]} + ${Checks[$i]} <
${BlocksRequired[$i]}))
do
while read -r
do
if [ ${Blocks[$i]} -ge ${BlockCount[$i]} ] ||
[ $((${Blocks[$i]} + ${Checks[$i]})) -ge
${BlocksRequired[$i]} ]
then
break
fi
if [[ "${REPLY%%.*}" == "Block" ]]
then
block_name=${REPLY%=*}
block_key=${REPLY#*=}
# don't try fetching a block we've already got
if [[ $(stat -f "%z"
"segment.${i}/${block_name}" 2>/dev/null) -ne ${BlockSi
ze[$i]} ]]
then
${FNGET} -s ${FCP_HOST} -H ${HTL} -O
"segment.${i}/${block_name}" "${block_
key}"
# old ft version
#${FCPGET} -n ${FCP_HOST} -l ${HTL} -v
3 ${block_key} "segment.${i}/${block
_name}"
fi
fi
# update block count for next loop conditional test
Blocks[$i]=$(find segment.${i} -type f -name "Block.*"
! -empty | wc -l)
Checks[$i]=$(find segment.${i} -type f -name "Check.*"
! -empty | wc -l)
done < "${output_filename}.segment.${i}.blockmap"
# update block count for next loop conditional test
Blocks[$i]=$(find segment.${i} -type f -name "Block.*" ! -empty
| wc -l)
Checks[$i]=$(find segment.${i} -type f -name "Check.*" ! -empty
| wc -l)
done &
# loop for reading check blocks
while ((${Checks[$i]} < ${CheckBlockCount[$i]})) &&
((${Blocks[$i]} + ${Checks[$i]} <
${BlocksRequired[$i]}))
do
while read -r
do
if [ ${Checks[$i]} -ge ${CheckBlockCount[$i]} ] ||
[ $((${Blocks[$i]} + ${Checks[$i]})) -ge
${BlocksRequired[$i]} ]
then
break
fi
if [[ "${REPLY%%.*}" == "Check" ]]
then
block_name=${REPLY%=*}
block_key=${REPLY#*=}
# don't try fetching a block we've already got
if [[ $(stat -f "%z"
"segment.${i}/${block_name}" 2>/dev/null) -ne ${CheckBl
ockSize[$i]} ]]
then
${FNGET} -s ${FCP_HOST} -H ${HTL} -O
"segment.${i}/${block_name}" "${block_
key}"
# old ft version
#${FCPGET} -n ${FCP_HOST} -l ${HTL} -v
3 ${block_key} "segment.${i}/${block
_name}"
fi
fi
# update block count for next loop conditional test
Blocks[$i]=$(find segment.${i} -type f -name "Block.*"
! -empty | wc -l)
Checks[$i]=$(find segment.${i} -type f -name "Check.*"
! -empty | wc -l)
done < "${output_filename}.segment.${i}.blockmap"
# update block counts for next loop conditional test
Blocks[$i]=$(find segment.${i} -type f -name "Block.*" ! -empty
| wc -l)
Checks[$i]=$(find segment.${i} -type f -name "Check.*" ! -empty
| wc -l)
done &
done
#!/usr/local/bin/bash
#
# watch_splitfile -- monitor the activity of a get_splitfile in progress
#
# must first do a "cd" to the directory where the splitfile is being downloaded
# before running this script
#
numsegs=$(ls *.segment.*.header | wc -l)
totalBlocksRequired=0
for ((i=0; i<$numsegs; ++i))
do
BlocksRequired[$i]=$(h2d $(awk -F'=' '/BlocksRequired=/ {print $2}'
*.segment.$
{i}.header))
totalBlocksRequired=$((totalBlocksRequired + ${BlocksRequired[$i]}))
done
while :
do
clear
for ((d=0; d<$numsegs; ++d))
do
blocks=$(find segment.${d} -type f -name "Block.*" ! -empty |
wc -l)
checks=$(find segment.${d} -type f -name "Check.*" ! -empty |
wc -l)
percent=$(echo "((${blocks} + ${checks}) /
${BlocksRequired[$d]}) * 100" | bc
-l)
printf "segment.%d: %3d data + %3d check = %3d blocks found\t
(%5.2f%% of %3d)
\n" \
${d} ${blocks} ${checks} $((${blocks} + ${checks}))
${percent} ${BlocksRequir
ed[$d]}
done
total_data_found=$(find segment.* -type f -name "Block.*" ! -empty | wc
-l)
total_check_found=$(find segment.* -type f -name "Check.*" ! -empty |
wc -l)
total_found=$((total_data_found + total_check_found))
total_remaining=$((totalBlocksRequired - total_found))
total_found_percent=$(echo "(${total_found} / ${totalBlocksRequired}) *
100" |
bc -l)
total_remaining_percent=$(echo "(${total_remaining} /
${totalBlocksRequired}) *
100" | bc -l)
printf "Total: %7d data + %3d check = %3d blocks found\t (%5.2f%% of
%3d)\n\n"
\
${total_data_found} ${total_check_found} ${total_found}
${total_found_percent
} ${totalBlocksRequired}
printf "Total blocks required: %6d\n" ${totalBlocksRequired}
printf "Total blocks found: %9d (%5.2f%%)\n" ${total_found}
${total_found_perce
nt}
printf "Total blocks remaining: %5d (%5.2f%%)\n\n" ${total_remaining}
${total_r
emaining_percent}
echo -e "Currently downloading:\n"
ps axww | grep -o 'segment[^ ]* freenet:[EMAIL PROTECTED]' | sort
sleep 30
done
_______________________________________________
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:[EMAIL PROTECTED]
