Bug#733750: support patching hexen2 game data

[Alexandre Detiste]

2018-01-26 0:26 GMT+01:00 Simon McVittie :
>
> Control: tags -1 = pending
>
> On Thu, 25 Jan 2018 at 14:56:19 +0100, Alexandre Detiste wrote:
> > This also confuses GDP which ends up thinking data1/pak*.pak?v1.11 are 
> > downloadable.
> > It then dowloads gamedata-all-1.29a.tgz for nothing.
>
> Thanks, fixed in git. The code was correctly taking into account the
> other_parts of a multi-part archive that is present, but not doing the
> same for one that is merely downloadable (and deltas are treated like
> the first part of a multi-part archive).

Thank you very much, I had myself figured the first line to change,
but not the second one.

Can you please also adapt the help_text to reflect the new improved status ?

> help_text: |
>  For hexen2-data, you must provide the version 1.11 patched versions
>  of data1/pak0.pak and data1/pak1.pak, or a directory containing them.

I guess I could write some broken/weird English by myself.

Greetings,

Bug#733750: support patching hexen2 game data

[Simon McVittie]

Control: tags -1 = pending

On Thu, 25 Jan 2018 at 14:56:19 +0100, Alexandre Detiste wrote:
> This also confuses GDP which ends up thinking data1/pak*.pak?v1.11 are 
> downloadable.
> It then dowloads gamedata-all-1.29a.tgz for nothing.

Thanks, fixed in git. The code was correctly taking into account the
other_parts of a multi-part archive that is present, but not doing the
same for one that is merely downloadable (and deltas are treated like
the first part of a multi-part archive).

smcv

Bug#733750: support patching hexen2 game data

[Alexandre Detiste]

Le mercredi 24 janvier 2018, 19 h 59 min 31 s CET Simon McVittie a écrit :
> It's encoded in the YAML as though the patch was the first part of
> a multi-part archive, the unpatched file was the second part, and the
> patched file is what we unpack from the archive - which is a little weird,
> but seems better than new syntax for deltas. So the YAML syntax should
> end up something like this:
> 
> data1pk0.xd3:
>   unpack:
> format: xdelta3
> other_parts:
>   - data1/pak0.pak?v1.03
> provides:
>   - data1/pak0.pak

Hi,

This also confuses GDP which ends up thinking data1/pak*.pak?v1.11 are 
downloadable.
It then dowloads gamedata-all-1.29a.tgz for nothing.

I guess UT suffer from the same thing.

 ./run hexen2 --no-search
INFO:game_data_packager.build:will not produce "hexen2-demo-data" because we 
have the full version "hexen2-data"
INFO:game_data_packager.download:downloading 
http://localhost/gamedata-all-1.29a.tgz
extracting patchdat/data1/data1pk1.xd3 from 
/tmp/gdptmp.7rlo7ffw/gamedata-all-1.29a.tgz
extracting patchdat/data1/data1pk0.xd3 from 
/tmp/gdptmp.7rlo7ffw/gamedata-all-1.29a.tgz
ERROR:game_data_packager.build:could not find data1/pak1.pak?v1.03:
  expected:
size:   76958474 bytes
md5:9a2010aafb9c0fe71c37d01292030270
sha1:   bd7d83349f8718ae5f16614ae99857b06a22fdd0
sha256: None
ERROR:game_data_packager.build:could not find data1/pak1.pak?v1.11:
  expected:
size:   75601170 bytes
md5:c2ac5b0640773eed9ebe1cda2eca2ad0
sha1:   34c17f508bb6cd9ae4401307d0930a6f00761826
sha256: 6940cc03dc731f9c19f4e8a21259d7f191bd60a7487b50ceed3f8268e0d32a56
ERROR:game_data_packager.build:could not find data1/pak0.pak?v1.03:
  expected:
size:   21714275 bytes
md5:b53c9391d16134cb3baddc1085f18683
sha1:   15358c105f88f8099fe889150a83022992c7357b
sha256: None
ERROR:game_data_packager.build:could not find data1/pak0.pak?v1.11:
  expected:
size:   22704056 bytes
md5:c9675191e75dd25a3b9ed81ee7e05eff
sha1:   7c6b19d76ce85771026ea8dbdc397cd1109355b7
sha256: 57936acc4cd9abed7acb480dd9c6d59c7c98e96e4eca4e2d04f390d2dfa5ffde
ERROR:game_data_packager.build:Failed to download necessary files for 
hexen2-data
ERROR:game_data_packager.build:Unable to complete any packages because 
downloads failed.


DEBUG:game_data_packager.build:trying to fill any gaps for hexen2-data
DEBUG:game_data_packager.build:gap needs to be filled for hexen2-data: 
data1/pak1.pak?v1.11
DEBUG:game_data_packager.build:gap needs to be filled for hexen2-data: 
data1/pak0.pak?v1.11
DEBUG:game_data_packager.build:could not find data1/pak1.pak?v1.11, trying to 
derive it...
DEBUG:game_data_packager.build:could not find patchdat/data1/data1pk1.xd3, 
trying to derive it...
DEBUG:game_data_packager.build:could not find gamedata-all-1.29a.tgz, trying to 
derive it...
DEBUG:game_data_packager.build:data1/pak1.pak?v1.11: FillResult.DOWNLOAD_NEEDED
DEBUG:game_data_packager.build:could not find data1/pak0.pak?v1.11, trying to 
derive it...
DEBUG:game_data_packager.build:could not find patchdat/data1/data1pk0.xd3, 
trying to derive it...
DEBUG:game_data_packager.build:data1/pak0.pak?v1.11: FillResult.DOWNLOAD_NEEDED
DEBUG:game_data_packager.build:unable to fill gap for hexen2-data: 
data1/pak1.pak?v1.11
DEBUG:game_data_packager.build:unable to fill gap for hexen2-data: 
data1/pak0.pak?v1.11
DEBUG:game_data_packager.build:hexen2-data: FillResult.DOWNLOAD_NEEDED
DEBUG:game_data_packager.build:hexen2-data is possible

Bug#733750: support patching hexen2 game data

[Simon McVittie]

On Wed, 24 Jan 2018 at 17:34:52 +, Jonathan Dowland wrote:
> However (and I'm not sure why I didn't file it this way in the first
> place) I think it would be nice if game-data-packager could handle the
> patching of hexen2 data where necessary.

Yes, it would be nice. We basically need something resembling
/usr/share/doc/uhexen2/examples/upgrade-111.sh in the declarative
g-d-p syntax in data/hexen2.yaml: that script already includes a
download URL and how to unpack the necessary files from what was
downloaded. /usr/share/doc/uhexen2/README.Debian has md5sums, although
I'd prefer to have the complete size/md5/sha1/sha256 set (someone who
owns v1.03 media will need to tell me those, because I only have the
Steam version, which comes pre-patched).

h2patch seems to use xdelta3 patches, so we could either teach g-d-p
to run h2patch with a "build-dependency" on uhexen2, or cut out the
middle-man by teaching g-d-p to run xdelta3. If you git grep for xdelta,
which I added for UT99, it should be reasonably obvious how to add
xdelta3 support to data/hexen2.yaml and game_data_packager/build.py.

It's encoded in the YAML as though the patch was the first part of
a multi-part archive, the unpatched file was the second part, and the
patched file is what we unpack from the archive - which is a little weird,
but seems better than new syntax for deltas. So the YAML syntax should
end up something like this:

data1pk0.xd3:
  unpack:
format: xdelta3
other_parts:
  - data1/pak0.pak?v1.03
provides:
  - data1/pak0.pak

game-data-packager already knows how to download files and extract
various common archives, so invoking xdelta3 should be the only part
that will need new Python code.

(In the general case, we can't do anything about obsolete versions of
data files apart from issue warnings; but in the case where patches exist
and can be applied without executing arbitrary game-publisher-supplied
binaries, we can certainly do that.)

smcv