On 07/11/2017 02:55 AM, Marek Marczykowski-Górecki wrote: > On Mon, Jul 10, 2017 at 09:38:22PM -0700, Andrew Morgan wrote: >> Hello everyone, it's that time of the week again. > >> On this episode: tests, file managing, something else?!?! > >> Grab it with format here: >> https://blog.amorgan.xyz/gsoc-weekly-progress-report-6.html > >> Otherwise text-only is below: > >> --- > >> Hello again! Few different things going on this week so let's just jump >> right in. > >> ## Tests > >> First off a basic set of tests have been written for the >> `qvm-file-trust` tool. So far these are all unit tests, and mainly test >> sending input to each method in `qvm-file-trust`, ensuring that certain >> methods are called with the correct parameters, and anything that is >> returned contains the correct data. > >> In the process of writing these tests I uncovered a few bugs and quirks >> here and there in the code, which is awesome because that's exactly what >> tests are designed to do. Currently I only test each method directly, >> however in the long run there will be tests that send varied input >> externally to the program. These will check the output as well as the >> trust status of any files that may be involved. > >> As was evident from my last post, I had a bit of trouble getting used to >> some 'pythonic' quirks, as well as the whole concept of mocking. >> Nevertheless, after getting a few fundamentals down on how to feed in >> fake inputs and data, writing the majority of the tests was mostly just >> time-consuming. > > Speaking of "pythonic" way, you should really consider using > console_scripts entry point and proper python package, instead of using > hacks for importing file directly. See here: > https://setuptools.readthedocs.io/en/latest/setuptools.html#basic-use > https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation > https://github.com/QubesOS/qubes-core-agent-linux/blob/master/setup.py > > At this stage the change is still simple: > 1. Rename the actual script to qvm_file_trust.py and put in to some > subdirectory (python package name) - like qubesfiletrust, add empty > __init__.py file there too. > 2. Add setup.py with appropriate console_scripts entry point (pointing > at `qubesfiletrust.qvm_file_trust:main`). > 3. Move tests to that package too > 4. Change hacky import into simply `import .qvm_file_trust` or `import > qubesfiletrust.qvm_file_trust` > > In Makefile you can install it with: > > python setup.py install --root $(DESTDIR) > > And to run tests directly from git without installation, add "." (or > full path to the source dir) to PYTHONPATH.
Very cool, I'll set this up ASAP. Not running the tests manually before every commit is very welcome :) > >> I also ran a few passes of [pylint](https://pypi.python.org/pypi/pylint) >> over `qvm-file-trust` to really make sure it was clean and following >> python's syntactical standards. > > A hint: take a look at pylint + unit tests integration with (Travis) CI > we have here: > https://github.com/qubesos/qubes-core-admin/ > > I'm not saying to do it now, but you may be interested in some examples Yeah I wasn't sure if it would be necessary to set up travis on my repo since the end goal is to integrate it into the main repos at some point. I'm pretty sure unittest has a pylint plugin I could utilize locally (and thus before every commit). > >> I still plan to add a few more tests for the cli tool to the suite, as >> well as those for the upcoming untrusted folder watching daemon. > >> Speaking of which... > >> ## Look out! It's a daemon! > >> I took a little time this week to start work on the C daemon that will >> watch all untrusted folders and mark any files created inside them as >> untrusted as well. For starters, I've switched to making it a C++ deamon >> instead. As far as I can tell there isn't any noticable performance >> difference between the two languages, and C++ comes with a few >> abstractions for various low-level things like file handling, meaning >> less room for errors and bugs. I'm also much more experienced in C++, so >> it seemed to the correct choice. > > +1 > >> While following the [Qubes Code Style >> Guidelines](https://www.qubes-os.org/doc/coding-style/), I was able to >> implement parsing of the global and local rule lists (including >> functionality such as comment ignoring and override rules) as well as >> being able to call on `qvm-file-trust` and set any file as untrusted >> through it. I'm currently working on iterating through the rule list and >> setting up an `inotify` watch on each folder (and its subfolders) for >> any new files placed within them. > >> In a very scientific test of setting up a basic C++ program to watch >> `/tmp` and holding down tab after `ls /tmp/`, generating many >> simulataneous inotify events, the CPU usage of the program was 0%. >> Whether it will spike up when creating thousands of files within >> untrusted folders remains to be seen, but I'm trying to keep the code as >> efficient as possible. > > Slightly more scientific way would be something like this with output > redirected to /dev/null, otherwise most of the time you're waiting for > terminal output... For example: > > for x in `seq 100000`; do ls /tmp >/dev/null; done Unfortunately only scanning the directory with `ls /tmp<tab>` actually produces any events, just ls'ing it doesn't. So instead I ran the following, which should more accurately cover our use case (notified of files being created in a watched directory): for x in `seq 100000`; do touch /tmp/asd; rm /tmp/asd; done Our C++ program seemed to max out at 0.7% CPU, which I think is pretty good :) One thing that may take more CPU is when we call our python program to mark each of these new files as untrusted though. It might not end well if we have to start up and shut down the python interpreter for every single new file... Possible solutions might be: 1. Mark the file as untrusted manually (chmod 0, xattr) through the C++ program. (Con: if we change the definition of an untrusted file down the line then we have to edit the C++ program as well) 2. Every second or so, grab all newly created files and just do a batch run of `qvm-file-trust` with all of them. (Con: still need to start the python interpreter but at least we're not hammering it now) > >> One thing I wasn't entirely sure about was where to send logs. I'm not >> sure if I should just log to syslog (and thus journald), or >> /var/log/qubes/... (or both). It's simple enough to do either one. > > Either stderr (being redirected to syslog or file by the caller), or > syslog. Leave log file management (including rotation, adding timestamps > etc) to appropriate tools. Those places we still use /var/log/qubes > directly are artifacts of the past mistakes... Thanks, I'll just log to stderr for flexibility then. > >> ## Water-based File Managers > >> I've started with modifying some of Dolphin's source code. Unfortunately >> I've been having a little bit of trouble with their build tool, >> [kdesrc-build](https://kdesrc-build.kde.org/). The amount of source code >> necessary to build Dolphin is apparently quite large. Every time I try >> to download the full dependencies of Dolphin it seems to stall halfway >> through. > >> This may not be the only way to work on Dolphin, but I haven't seen an >> alternative so far... > >> The other matter is that Whonix's current Dolphin version is 4.14.2, >> while an updated Arch system's Dolphin version is 17.04.2 (and source is >> 17.04.70). > >> According to [this bug report](https://phabricator.whonix.org/T633), >> Whonix should be using KDE Frameworks 5 in v14 (next release), which >> should be included in Qubes soon following release. Due to this, I'm >> wondering whether I should base my patches on top of Dolphin's latest >> code or the legacy v4.14. If the latest, then it may be best to develop >> on Arch instead of Qubes, or maybe I could find a pre-release build of >> Whonix 14... > > I'd focus on the new version. Patrick, can you confirm the above? What > is the base Debian version for Whonix 14? > >> For Nautilus I've just been using the latest code and it's going well. >> Update on that in a bit! > >> In any case, I've outlined the required changes that need to be made to >> each file manager below: > >> Nautilus: > >> 1. Expose file open events to extensions > >> 2. Respect not opening file if extension requests not to (upstream may >> not like this part :) > > I'd not assume this - instead ask on appropriate mailing list. IMO this > should also be acceptable if using clean mechanism for this. > >> Dolphin: > >> 1. Work around the complete prevention of attempting to open files for >> which we have no read permissions for (instead check with our cli tool >> and decide what to do based on its output) > >> 2. If the file is deemed an untrusted file, attempt to open it with our >> dvm-desktop handler > >> 3. Failing that, display an error > >> That should about cover the functionality. Hopefully it'll be at least >> partially upstream-able. I'm open to suggestions if anyone has any :) > >> That about wraps it up for this week! As always the code is available >> [here](https://github.com/anoadragon453/qubes-mime-types). > > And as usual, see comments there. > > -- You received this message because you are subscribed to the Google Groups "qubes-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/ok3k3r%24olp%241%40blaine.gmane.org. For more options, visit https://groups.google.com/d/optout.
signature.asc
Description: OpenPGP digital signature
