Re: [Reprozip-users] Web Archiving

[Rémi Rampin]

2018-05-11 09:54 EDT, Rasa Bočytė :

> A couple of questions:
> - apart from tracing the server, do I need to trace any other processes to
> ensure that I can reproduce the experiment?
> - how do I ensure that my package includes all the website data? I tried
> running a php inbuilt server (reprozip trace php -S localhost:8000 -t
> path/to/the/website/files) and the finished package only captured data from
> the web pages that I visited while running the experiment.
> - can I somehow include the browser needed to run the website into the
> package?
>

Hi Rasa,

Apologies for the delay. To answer your question, when using LAMP you would
need to trace both Apache (which would capture the PHP interpreter, since
it is embedded as a module) and MySQL (you can see Capturing Connections to
Servers 
for more information).

Only the PHP files you visited will be traced by ReproZip, however you can
add the whole site folder (usually under /var/www) and MySQL data folder (
/var/lib/mysql) to the additional_files section of the configuration file

if you don't want to visit every single page (make sure to visit some of
them to get all the libraries and database connections involved).

Let us know if you have more questions!
-- 
Rémi
___
Reprozip-users mailing list
Reprozip-users@vgc.poly.edu
https://vgc.poly.edu/mailman/listinfo/reprozip-users

Re: [Reprozip-users] Web Archiving

[Rémi Rampin]

2018-04-06 09:11 EDT, Rasa Bočytė <rboc...@beeldengeluid.nl>:

> If I understand correctly, ReproZip can describe environments that are
> necessary to run a particular software or a web application that is used to
> create a dynamic website (Nikola in the case of your website or Django with
> StackedUp).
>

ReproZip can capture any software environment. That can be a web server,
that is serving a website (possibly dynamic), for example in the Django
case. Nikola is *not* a web server, it is a program that generates static
files initially, and those files will later be served to user through the
use of a web server.

Would it work if there is no such software? [...] could it capture the
> environment if the content is placed on a server just as files and folders
> with no software?
>

Every website necessarily has a web server. If all the content is in static
files without any kind of database or dynamic aspect to it (like files
generated by Nikola), preservation is a lot easier, since you can just zip
up those files. Is this your use case?

-- 
Rémi Rampin
ReproZip Developer
Center for Data Science, New York University
___
Reprozip-users mailing list
Reprozip-users@vgc.poly.edu
https://vgc.poly.edu/mailman/listinfo/reprozip-users

Re: [Reprozip-users] Web Archiving

[Rémi Rampin]

Rasa,

2018-04-04 08:03 EDT, Rasa Bočytė <rboc...@beeldengeluid.nl>:

> In our case, we are getting all the source files directly from content
> creators and we are looking for a way to record and store all the
> technical, administrative and descriptive metadata, and visualise
> dependencies on software/hardware/file formats/ etc. (similar to what
> Binder does).
>

I didn't think Binder did that (this binder?
<https://github.com/jupyterhub/binderhub>). It is certainly a good resource
for reproducing environments already described as a Docker image or Conda
YaML, but I am not aware of ways to use it to track or visualize
dependencies or any metadata.

We have been mostly considering migration as it is a more scalable approach
> and less technically demanding. Do you find that virtualisation is a better
> strategy for website preservation? At least from the archival community, we
> have heard some reservations about using Docker since it is not considered
> a stable platform.
>

When you talk of migration, do you mean to new hardware? What would you be
migrating to? Or do you mean upgrading underlying software/frameworks?
The way I see it, virtualization (sometimes referred to as "preserving the
mess") is definitely less technically demanding than migration. Could you
share a bit more about what you mean by this?

Thanks

PS: Please make sure you keep us...@reprozip.org in the recipients list.
-- 
Rémi Rampin
ReproZip Developer
Center for Data Science, New York University
___
Reprozip-users mailing list
Reprozip-users@vgc.poly.edu
https://vgc.poly.edu/mailman/listinfo/reprozip-users

Re: [Reprozip-users] Web Archiving

[Rémi Rampin]

-- Forwarded message --
From: Rasa Bočytė <rboc...@beeldengeluid.nl>
Date: Wed, Apr 4, 2018 at 8:03 AM
Subject: Re: [Reprozip-users] Web Archiving
To: Rémi Rampin <remi.ram...@nyu.edu>


Dear Remi,

thank you for your response! It is good to hear that other people are
working on similar issues as well!

Could you tell me a bit more about your work on trying to packaging dynamic
websites? Are you working on specific cases or just exploring the
possibilities? I would be very interested to hear how you approach this.

In our case, we are getting all the source files directly from content
creators and we are looking for a way to record and store all the
technical, administrative and descriptive metadata, and visualise
dependencies on software/hardware/file formats/ etc. (similar to what
Binder does). We would try to get as much information from the creators
(probably via a questionnaire) about all the technical details as well as
creative processes, and preferably record it in a machine and human
readable format (XML) or README file.

At the end of the day, we want to see whether we can come up with some kind
of scalable strategy for websites preservation or whether this would have
to be done very much on a case-by-case basis. We have been mostly
considering migration as it is a more scalable approach and less
technically demanding. Do you find that virtualisation is a better strategy
for website preservation? At least from the archival community, we have
heard some reservations about using Docker since it is not considered a
stable platform.


Kind regards,
Rasa

On 3 April 2018 at 19:11, Rémi Rampin <remi.ram...@nyu.edu> wrote:

> 2018-03-30 08:59 EDT, Rasa Bočytė <rboc...@beeldengeluid.nl>:
>
>> What I am most interested in is to know whether ReproZip could be used to:
>> - document the original environment from which the files were acquired
>> (web server, hardware, software)
>> - record extra technical details and instructions that could be added
>> manually
>> - maintain dependencies between files and folders
>> - capture metadata
>>
>
> Hello Rasa,
>
> Thanks for your note! We have been looking specifically at packing dynamic
> websites, so your email is timely. In short, ReproZip can do the bullet
> points you’ve outlined in your email.
>
> ReproZip stores the kernel version and distribution information (provided
> by Python’s platform module
> <https://docs.python.org/3/library/platform.html>). That is usually
> enough to get a virtual machine (or base Docker image) to run the software.
> It also stores the version numbers of all packages from which files are
> used (on deb-based and rpm-based distributions for now).
>
> For instructions, ReproZip stores the command-lines used to run the
> software that is getting packaged. It can also record which files are
> “input” or “output” of the process, which is useful when reproducing, to
> allow the user to replace inputs and look at outputs easily. We try to
> infer this information, but we rely on the user to check this, and provide
> descriptive names for them (however this means editing the YAML
> configuration file, so our experience is that not everyone takes the time).
> They can also edit the command-lines/environment variables.
>
> I am not entirely sure what you mean by “dependencies between files and
> folders”. The path to files in the original environment is stored, so the
> reproduction can happen with the exact same setup.
>
> As for metadata, ReproZip can automatically capture technical and
> administrative metadata in the config.yml file included in the ReproZip
> package. The user (archivist, in your case!) will need to add descriptive
> metadata to the package in something like a README, a finding aid, or by
> archiving the ReproZip package in a repository and filling in the required
> metadata form.
>
> We have a few examples of ReproZip packing and preserving websites. We
> packed a general blog made with Django (a small web application using a
> SQLite3 database) -- you can view more information about that website and
> the instructions for unpacking it here
> <https://github.com/ViDA-NYU/reprozip-examples/tree/master/django-blog>.
> We’ve also packed a data journalism app called Stacked Up, a dynamic web
> application to explore the textbook inventory of Philadelphia public
> schools (data is stored in a PostgreSQL database)-- you can view more
> information about that website and the instructions for unpacking it here
> <https://github.com/ViDA-NYU/reprozip-examples/tree/master/stacked-up>.
> You can view all of our examples in our examples site
> <https://examples.reprozip.org/>.
>
> Best regards,
> --
> Rémi Rampin
> ReproZip Developer
> Center for Data Science

Re: [Reprozip-users] Web Archiving

[Rémi Rampin]

2018-03-30 08:59 EDT, Rasa Bočytė <rboc...@beeldengeluid.nl>:

> What I am most interested in is to know whether ReproZip could be used to:
> - document the original environment from which the files were acquired
> (web server, hardware, software)
> - record extra technical details and instructions that could be added
> manually
> - maintain dependencies between files and folders
> - capture metadata
>

Hello Rasa,

Thanks for your note! We have been looking specifically at packing dynamic
websites, so your email is timely. In short, ReproZip can do the bullet
points you’ve outlined in your email.

ReproZip stores the kernel version and distribution information (provided
by Python’s platform module
<https://docs.python.org/3/library/platform.html>). That is usually enough
to get a virtual machine (or base Docker image) to run the software. It
also stores the version numbers of all packages from which files are used
(on deb-based and rpm-based distributions for now).

For instructions, ReproZip stores the command-lines used to run the
software that is getting packaged. It can also record which files are
“input” or “output” of the process, which is useful when reproducing, to
allow the user to replace inputs and look at outputs easily. We try to
infer this information, but we rely on the user to check this, and provide
descriptive names for them (however this means editing the YAML
configuration file, so our experience is that not everyone takes the time).
They can also edit the command-lines/environment variables.

I am not entirely sure what you mean by “dependencies between files and
folders”. The path to files in the original environment is stored, so the
reproduction can happen with the exact same setup.

As for metadata, ReproZip can automatically capture technical and
administrative metadata in the config.yml file included in the ReproZip
package. The user (archivist, in your case!) will need to add descriptive
metadata to the package in something like a README, a finding aid, or by
archiving the ReproZip package in a repository and filling in the required
metadata form.

We have a few examples of ReproZip packing and preserving websites. We
packed a general blog made with Django (a small web application using a
SQLite3 database) -- you can view more information about that website and
the instructions for unpacking it here
<https://github.com/ViDA-NYU/reprozip-examples/tree/master/django-blog>.
We’ve also packed a data journalism app called Stacked Up, a dynamic web
application to explore the textbook inventory of Philadelphia public
schools (data is stored in a PostgreSQL database)-- you can view more
information about that website and the instructions for unpacking it here
<https://github.com/ViDA-NYU/reprozip-examples/tree/master/stacked-up>. You
can view all of our examples in our examples site
<https://examples.reprozip.org/>.

Best regards,
-- 
Rémi Rampin
ReproZip Developer
Center for Data Science, New York University
___
Reprozip-users mailing list
Reprozip-users@vgc.poly.edu
https://vgc.poly.edu/mailman/listinfo/reprozip-users

Re: [Reprozip-users] Fwd: Reprozip Project

[Rémi Rampin]

2017-10-30 16:11 EDT, Rémi Rampin <remi.ram...@nyu.edu>:

> I will add this option to ReproZip, so it will be in the next version that
> we'll release, but it might take some time before that is available in
> Ubuntu.
>

I have pushed the new option to GitHub: here
<https://github.com/ViDA-NYU/reprozip/commit/59b3af6f9ae35852e0ad1252d7d00b314d40ca02?diff=split>

Let me know if you manage to apply this change, or if there are issues with
it.

Thanks for the suggestion!

-- 
Rémi
___
Reprozip-users mailing list
Reprozip-users@vgc.poly.edu
https://vgc.poly.edu/mailman/listinfo/reprozip-users

Re: [Reprozip-users] Problems with installing reprozip

[Rémi Rampin]

2016-12-21 20:50 EST, Nadège :
>
> Sorry, I am not making much sense. So I deleted
> '/home/nadege/.local/lib/python*` a first time and tried to type pip
> install reprozip, but I got the same result as the first time. So I deleted
> python again and installed virtualenv but I cannot tell whether it is
> working or not because I have never used it before, so I will study this
> tomorrow.
>
> Thank you a lot for your help.
___
Reprozip-users mailing list
Reprozip-users@vgc.poly.edu
https://vgc.poly.edu/mailman/listinfo/reprozip-users

Re: [Reprozip-users] Problems with installing reprozip

[Rémi Rampin]

2016-12-21 20:13 EST, Nadège :
>
> Sadly now everytime I try to use pip, I get the error message that you can
> see on the pictures that I sent in my first email. Nothing with pip works,
> whether it is to install a package or just do a pip list. I even tried to
> uninstall pip and it did not work either. So I can't even use virtualenv...


You deleted `/home/nadege/.local/lib/python*` and pip isn't fixed?

Running from your user, there should be no way for pip to mess things up
system-wide...

-- 
Rémi
___
Reprozip-users mailing list
Reprozip-users@vgc.poly.edu
https://vgc.poly.edu/mailman/listinfo/reprozip-users