Hello community, here is the log from the commit of package python-pyct for openSUSE:Factory checked in at 2019-01-08 12:26:13 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/python-pyct (Old) and /work/SRC/openSUSE:Factory/.python-pyct.new.28833 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-pyct" Tue Jan 8 12:26:13 2019 rev:2 rq:660797 version:0.4.6 Changes: -------- --- /work/SRC/openSUSE:Factory/python-pyct/python-pyct.changes 2018-10-31 13:15:35.391388205 +0100 +++ /work/SRC/openSUSE:Factory/.python-pyct.new.28833/python-pyct.changes 2019-01-08 12:28:25.460275719 +0100 @@ -1,0 +2,8 @@ +Sat Dec 22 03:45:56 UTC 2018 - Todd R <[email protected]> + +- Update to 0.4.6 + * Tests were added to make sure the the commands in pyct.cmd are well documented + * The --use-test-data flag was added to fetch-data and examples to allow the use of smaller datasets in regular examples. To use this feature place a smaller dataset with the same name in your project's examples/data/.data_stubs/. See pyviz/pyviz#128 for first use of this feature. + * clean-data was added to delete small test data files masquerading as real ones. + +------------------------------------------------------------------- Old: ---- pyct-0.4.5.tar.gz New: ---- pyct-0.4.6.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ python-pyct.spec ++++++ --- /var/tmp/diff_new_pack.MUgswL/_old 2019-01-08 12:28:25.832275313 +0100 +++ /var/tmp/diff_new_pack.MUgswL/_new 2019-01-08 12:28:25.832275313 +0100 @@ -18,7 +18,7 @@ %{?!python_module:%define python_module() python-%{**} python3-%{**}} Name: python-pyct -Version: 0.4.5 +Version: 0.4.6 Release: 0 Summary: Python package for common tasks for users License: BSD-3-Clause @@ -34,6 +34,7 @@ BuildRequires: %{python_module PyYAML} BuildRequires: %{python_module flake8} BuildRequires: %{python_module param >= 1.7.0} +BuildRequires: %{python_module pytest} BuildRequires: %{python_module requests} # /SECTION Requires: python-PyYAML @@ -68,6 +69,12 @@ %python_install %python_expand %fdupes %{buildroot}%{$python_sitelib} +%check +rm -rf build _build* +%{python_expand rm -rf build _build* +pytest-%{$python_bin_suffix} +} + %files %{python_files} %doc README.md %license LICENSE.txt ++++++ pyct-0.4.5.tar.gz -> pyct-0.4.6.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pyct-0.4.5/PKG-INFO new/pyct-0.4.6/PKG-INFO --- old/pyct-0.4.5/PKG-INFO 2018-07-12 10:23:10.000000000 +0200 +++ new/pyct-0.4.6/PKG-INFO 2018-11-01 21:36:09.000000000 +0100 @@ -1,6 +1,6 @@ Metadata-Version: 2.1 Name: pyct -Version: 0.4.5 +Version: 0.4.6 Summary: python package common tasks for users (e.g. copy examples, fetch data, ...) Home-page: https://ioam.github.io/parambokeh Author: PyViz @@ -40,13 +40,18 @@ ``` $ datashader examples --help - usage: datashader examples [-h] [--path PATH] [-v] [--force] + usage: datashader examples [-h] [--path PATH] [-v] [--force] [--use-test-data] optional arguments: - -h, --help show this help message and exit - --path PATH location to place examples and data + -h, --help show this help message and exit + --path PATH location to place examples and data -v, --verbose - --force if PATH already exists, force overwrite existing examples if older than source examples + --force if PATH already exists, force overwrite existing examples + if older than source examples. ALSO force any existing data + files to be replaced + --use-test-data Use data's test files, if any, instead of fetching full + data. If test file not in '.data_stubs', fall back to + fetching full data. ``` To copy the examples of e.g. datashader but not download the data, @@ -59,19 +64,26 @@ -h, --help show this help message and exit --path PATH where to copy examples -v, --verbose - --force if PATH already exists, force overwrite existing examples if older than source examples + --force if PATH already exists, force overwrite existing files if + older than source files ``` And to download the data only, the `fetch-data` command: ``` usage: datashader fetch-data [-h] [--path PATH] [--datasets DATASETS] [-v] + [--force] [--use-test-data] optional arguments: -h, --help show this help message and exit --path PATH where to put data - --datasets DATASETS *name* of datasets file; must exist either in path specified by --path or in package/examples/ + --datasets DATASETS *name* of datasets file; must exist either in path + specified by --path or in package/examples/ -v, --verbose + --force Force any existing data files to be replaced + --use-test-data Use data's test files, if any, instead of fetching full + data. If test file not in '.data_stubs', fall back to + fetching full data. ``` Can specify different 'datasets' file: @@ -92,6 +104,49 @@ Skipping Depth data for the Chesapeake and Delaware Bay region of the USA ``` + Can use smaller files instead of large ones by using the `--use-test-data` flag + and placing a small file with the same name in `examples/data/.data_stubs`: + + ``` + $ tree examples/data -a + examples/data + ├── .data_stubs + │ └── nyc_taxi_wide.parq + └── diamonds.csv + + $ cat examples/dataset.yml + data: + + - url: http://s3.amazonaws.com/datashader-data/nyc_taxi_wide.parq + title: 'NYC Taxi Data' + files: + - nyc_taxi_wide.parq + + - url: http://s3.amazonaws.com/datashader-data/maccdc2012_graph.zip + title: 'National CyberWatch Mid-Atlantic Collegiate Cyber Defense Competition' + files: + - maccdc2012_nodes.parq + - maccdc2012_edges.parq + - maccdc2012_full_nodes.parq + - maccdc2012_full_edges.parq + + $ pyviz fetch-data --path=examples --use-test-data + Fetching data defined in /tmp/pyviz/examples/datasets.yml and placing in /tmp/pyviz/examples/data + Copying test data file '/tmp/pyviz/examples/data/.data_stubs/nyc_taxi_wide.parq' to '/tmp/pyviz/examples/data/nyc_taxi_wide.parq' + No test file found for: /tmp/pyviz/examples/data/.data_stubs/maccdc2012_nodes.parq. Using regular file instead + Downloading National CyberWatch Mid-Atlantic Collegiate Cyber Defense Competition 1 of 1 + [################################] 59/59 - 00:00:00 + ``` + + To clean up any potential test files masquerading as real data use `clean-data`: + + ``` + usage: pyviz clean-data [-h] [--path PATH] + + optional arguments: + -h, --help show this help message and exit + --path PATH where to clean data + ``` ## pyct.build @@ -111,6 +166,6 @@ Classifier: Development Status :: 4 - Beta Requires-Python: >=2.7 Description-Content-Type: text/markdown -Provides-Extra: doc Provides-Extra: cmd +Provides-Extra: doc Provides-Extra: tests diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pyct-0.4.5/README.md new/pyct-0.4.6/README.md --- old/pyct-0.4.5/README.md 2018-07-12 10:22:07.000000000 +0200 +++ new/pyct-0.4.6/README.md 2018-11-01 21:34:30.000000000 +0100 @@ -27,13 +27,18 @@ ``` $ datashader examples --help -usage: datashader examples [-h] [--path PATH] [-v] [--force] +usage: datashader examples [-h] [--path PATH] [-v] [--force] [--use-test-data] optional arguments: - -h, --help show this help message and exit - --path PATH location to place examples and data + -h, --help show this help message and exit + --path PATH location to place examples and data -v, --verbose - --force if PATH already exists, force overwrite existing examples if older than source examples + --force if PATH already exists, force overwrite existing examples + if older than source examples. ALSO force any existing data + files to be replaced + --use-test-data Use data's test files, if any, instead of fetching full + data. If test file not in '.data_stubs', fall back to + fetching full data. ``` To copy the examples of e.g. datashader but not download the data, @@ -46,19 +51,26 @@ -h, --help show this help message and exit --path PATH where to copy examples -v, --verbose - --force if PATH already exists, force overwrite existing examples if older than source examples + --force if PATH already exists, force overwrite existing files if + older than source files ``` And to download the data only, the `fetch-data` command: ``` usage: datashader fetch-data [-h] [--path PATH] [--datasets DATASETS] [-v] + [--force] [--use-test-data] optional arguments: -h, --help show this help message and exit --path PATH where to put data - --datasets DATASETS *name* of datasets file; must exist either in path specified by --path or in package/examples/ + --datasets DATASETS *name* of datasets file; must exist either in path + specified by --path or in package/examples/ -v, --verbose + --force Force any existing data files to be replaced + --use-test-data Use data's test files, if any, instead of fetching full + data. If test file not in '.data_stubs', fall back to + fetching full data. ``` Can specify different 'datasets' file: @@ -79,6 +91,49 @@ Skipping Depth data for the Chesapeake and Delaware Bay region of the USA ``` +Can use smaller files instead of large ones by using the `--use-test-data` flag +and placing a small file with the same name in `examples/data/.data_stubs`: + +``` +$ tree examples/data -a +examples/data +├── .data_stubs +│ └── nyc_taxi_wide.parq +└── diamonds.csv + +$ cat examples/dataset.yml +data: + + - url: http://s3.amazonaws.com/datashader-data/nyc_taxi_wide.parq + title: 'NYC Taxi Data' + files: + - nyc_taxi_wide.parq + + - url: http://s3.amazonaws.com/datashader-data/maccdc2012_graph.zip + title: 'National CyberWatch Mid-Atlantic Collegiate Cyber Defense Competition' + files: + - maccdc2012_nodes.parq + - maccdc2012_edges.parq + - maccdc2012_full_nodes.parq + - maccdc2012_full_edges.parq + +$ pyviz fetch-data --path=examples --use-test-data +Fetching data defined in /tmp/pyviz/examples/datasets.yml and placing in /tmp/pyviz/examples/data +Copying test data file '/tmp/pyviz/examples/data/.data_stubs/nyc_taxi_wide.parq' to '/tmp/pyviz/examples/data/nyc_taxi_wide.parq' +No test file found for: /tmp/pyviz/examples/data/.data_stubs/maccdc2012_nodes.parq. Using regular file instead +Downloading National CyberWatch Mid-Atlantic Collegiate Cyber Defense Competition 1 of 1 +[################################] 59/59 - 00:00:00 +``` + +To clean up any potential test files masquerading as real data use `clean-data`: + +``` +usage: pyviz clean-data [-h] [--path PATH] + +optional arguments: + -h, --help show this help message and exit + --path PATH where to clean data +``` ## pyct.build diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pyct-0.4.5/pyct/.version new/pyct-0.4.6/pyct/.version --- old/pyct-0.4.5/pyct/.version 2018-07-12 10:23:10.000000000 +0200 +++ new/pyct-0.4.6/pyct/.version 2018-11-01 21:36:09.000000000 +0100 @@ -1 +1 @@ -{"git_describe": "v0.4.5-0-g125ce27", "version_string": "0.4.5"} \ No newline at end of file +{"git_describe": "v0.4.6-0-ga58d62d", "version_string": "0.4.6"} \ No newline at end of file diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pyct-0.4.5/pyct/cmd.py new/pyct-0.4.6/pyct/cmd.py --- old/pyct-0.4.5/pyct/cmd.py 2018-07-12 10:22:07.000000000 +0200 +++ new/pyct-0.4.6/pyct/cmd.py 2018-11-01 21:34:30.000000000 +0100 @@ -9,6 +9,7 @@ import inspect import argparse import distutils.dir_util +import shutil def _find_examples(name): module_path = os.path.dirname(inspect.getfile(importlib.import_module(name))) @@ -24,10 +25,16 @@ raise ValueError("Could not find examples for %s at any of %s"%(name,candidates)) -def examples(name,path,verbose=False,force=False): - """Copy examples and fetch data (if any) to the supplied path. See copy-examples and fetch-data for more flexibility.""" +def examples(name,path,verbose=False,use_test_data=False,force=False): + """ + Copy examples and fetch data (if any) to the supplied path. + See copy-examples and fetch-data for more flexibility. + + NOTE: force operates both on example and data over-writing + pre-existing files. + """ copy_examples(name, path, verbose, force) - fetch_data(name,path,require_datasets=False) + fetch_data(name,path,require_datasets=False,use_test_data=use_test_data,force=force) def copy_examples(name,path,verbose=False,force=False): @@ -80,7 +87,7 @@ # print('this download script requires the requests module: conda install requests') # sys.exit(1) - + STREAM = sys.stderr BAR_TEMPLATE = '%s[%s%s] %i/%i - %s\r' @@ -96,6 +103,8 @@ # How many intervals (excluding the current one) to calculate the simple moving # average ETA_SMA_WINDOW = 9 +DATA_DIR = 'data' +DATA_STUBS_DIR = '.data_stubs' class Bar(object): @@ -254,7 +263,7 @@ os.remove(output_path) -def _process_dataset(dataset, output_dir, here): +def _process_dataset(dataset, output_dir, here, use_test_data=False, force=False): '''Process each download spec in datasets.yml Typically each dataset list entry in the yml has @@ -276,14 +285,14 @@ requires_download = True break - if not requires_download: + if force is False and not requires_download: print('Skipping {0}'.format(dataset['title'])) return url = dataset['url'] title_fmt = dataset['title'] + ' {} of {}' if url.endswith('/'): urls = [url + f for f in dataset['files']] - output_paths = [os.path.join(here, 'data', fname) + output_paths = [os.path.join(here, DATA_DIR, fname) for fname in dataset['files']] unpacked = ['.'.join(output_path.split('.')[:(-2 if output_path.endswith('gz') else -1)]) + '*' @@ -297,11 +306,19 @@ zipped = zip(urls, output_paths, unpacked) for idx, (url, output_path, unpack) in enumerate(zipped): running_title = title_fmt.format(idx + 1, len(urls)) - if glob.glob(unpack) or os.path.exists(unpack.replace('*','')): + if force is False and (glob.glob(unpack) or os.path.exists(unpack.replace('*',''))): # Skip a file if a similar one is downloaded: # i.e. one that has same name but dif't extension print('Skipping {0}'.format(running_title)) continue + test = os.path.join(output_dir, DATA_STUBS_DIR, unpack) + if use_test_data and os.path.exists(test): + target = os.path.join(output_dir, unpack) + print("Copying test data file '{0}' to '{1}'".format(test, target)) + shutil.copyfile(test, target) + continue + elif use_test_data and not os.path.exists(test): + print("No test file found for: {}. Using regular file instead".format(test)) _url_to_binary_write(url, output_path, running_title) _extract_downloaded_archive(output_path) @@ -310,7 +327,7 @@ print('this download script requires the requests module: conda install requests') sys.exit(1) -def fetch_data(name,path,datasets="datasets.yml",require_datasets=True): +def fetch_data(name,path,datasets="datasets.yml",require_datasets=True,use_test_data=False,force=False): '''Fetch sample datasets as defined by path/datasets if it exists or else module's own examples/datasets otherwise. Datasets are placed in path/data @@ -323,15 +340,45 @@ if not os.path.exists(info_file) and require_datasets is False: print("No datasets to download") return - - print("Fetching data defined in %s and placing in %s"%(info_file,os.path.join(path,"data"))) # data is added later... + + print("Fetching data defined in %s and placing in %s"%(info_file,os.path.join(path,DATA_DIR))) # data is added later... with open(info_file) as f: info = ordered_load(f.read()) for topic, downloads in info.items(): output_dir = os.path.join(path, topic) for d in downloads: - _process_dataset(d, output_dir, path) + _process_dataset(d, output_dir, path, use_test_data=use_test_data, force=force) + +def clean_data(name, path): + '''Remove up any data files that are copied from test files + ''' + path = os.path.abspath(path) + if not os.path.exists(path): + path = _find_examples(name) + + data_dir = os.path.join(path, DATA_DIR) + test_dir = os.path.join(data_dir, DATA_STUBS_DIR) + if not os.path.exists(test_dir) or len(os.listdir(test_dir)) == 0: + print("No test files found") + return + + for f in os.listdir(test_dir): + data_file = os.path.join(data_dir, f) + if not os.path.isfile(data_file): + print("Test file was not copied to data:", f) + continue + + test_file = os.path.join(test_dir, f) + if os.path.isfile(test_file): + data_s = os.path.getsize(data_file) + test_s = os.path.getsize(test_file) + if data_s == test_s: + print("Removing copied test file:", f) + os.remove(data_file) + else: + print("Size of test file {:.2e} did not match " + "size of data file {:.2e}".format(test_s, data_s)) # TODO: cmds=None defaults to 'all', basically, which is a bit confusing @@ -344,7 +391,7 @@ if cmds is None: # again a reg (duplicated in substitute_main) - cmds = ['examples','copy-examples','fetch-data'] + cmds = ['examples','copy-examples','fetch-data','clean-data'] # use dict/reg instead if 'copy-examples' in cmds: @@ -356,31 +403,44 @@ if 'fetch-data' in cmds: d_parser = parser.add_parser('fetch-data', help=inspect.getdoc(fetch_data)) - d_parser.set_defaults(func=lambda args: fetch_data(name,args.path,args.datasets)) + d_parser.set_defaults(func=lambda args: fetch_data(name,args.path,args.datasets,use_test_data=args.use_test_data,force=args.force)) d_parser.add_argument('--path',type=str,help='where to put data',default='%s-examples'%name) d_parser.add_argument('--datasets',type=str,help='*name* of datasets file; must exist either in path specified by --path or in package/examples/',default='datasets.yml') d_parser.add_argument('-v', '--verbose', action='count', default=0) + d_parser.add_argument('--force',action='store_true', help='Force any existing data files to be replaced') + d_parser.add_argument('--use-test-data',action='store_true', + help=("Use data's test files, if any, instead of fetching full data. " + "If test file not in '.data_stubs', fall back to fetching full data.")) if 'examples' in cmds: egd_parser = parser.add_parser('examples', help=inspect.getdoc(examples)) - egd_parser.set_defaults(func=lambda args: examples(name, args.path, args.verbose, args.force)) + egd_parser.set_defaults(func=lambda args: examples(name, args.path, args.verbose, args.use_test_data, args.force)) egd_parser.add_argument('--path',type=str,help='location to place examples and data',default='%s-examples'%name) egd_parser.add_argument('-v', '--verbose', action='count', default=0) - egd_parser.add_argument('--force', action='store_true', help='if PATH already exists, force overwrite existing examples if older than source examples') - + egd_parser.add_argument('--force', action='store_true', + help=('if PATH already exists, force overwrite existing examples if older ' + 'than source examples. ALSO force any existing data files to be replaced')) + egd_parser.add_argument('--use-test-data',action='store_true', + help=("Use data's test files, if any, instead of fetching full data. " + "If test file not in '.data_stubs', fall back to fetching full data.")) + + if 'clean-data' in cmds: + cd_parser = parser.add_parser('clean-data', help=inspect.getdoc(clean_data)) + cd_parser.set_defaults(func=lambda args: clean_data(name,args.path)) + cd_parser.add_argument('--path',type=str,help='where to clean data',default='%s-examples'%name) def substitute_main(name,cmds=None,args=None): # can use if your module has no other commands if cmds is None: # again a reg - cmds = ['examples','copy-examples','fetch-data'] - + cmds = ['examples','copy-examples','fetch-data', 'clean-data'] + mod = importlib.import_module(name) parser = argparse.ArgumentParser(description="%s commands"%name) parser.add_argument('--version', action='version', version='%(prog)s '+mod.__version__) subparsers = parser.add_subparsers(title='available commands') add_commands(subparsers,name,cmds,args) args = parser.parse_args() - args.func(args) if hasattr(args,'func') else parser.error("must supply command to run") - + args.func(args) if hasattr(args,'func') else parser.error("must supply command to run") + diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pyct-0.4.5/pyct/tests/test_cmd.py new/pyct-0.4.6/pyct/tests/test_cmd.py --- old/pyct-0.4.5/pyct/tests/test_cmd.py 1970-01-01 01:00:00.000000000 +0100 +++ new/pyct-0.4.6/pyct/tests/test_cmd.py 2018-11-01 21:34:30.000000000 +0100 @@ -0,0 +1,229 @@ +import os +import pytest +import pyct.cmd +from pyct.cmd import fetch_data, clean_data, copy_examples, examples + +# Same as in pyct/examples/datasets.yml +DATASETS_CONTENT = u""" +data: + - url: this_should_never_be_used + title: 'Test Data' + files: + - test_data.csv +""" + +# Same as in pyct/examples/data/.data_stubs/test_data.csv +TEST_FILE_CONTENT = u""" +name,score,rank +Alice,100.5,1 +Bob,50.3,2 +Charlie,25,3 +""" + +REAL_FILE_CONTENT = u""" +name,score,rank +Alice,100.5,1 +Bob,50.3,2 +Charlie,25,3 +Dave,28,4 +Eve,25,3 +Frank,75,9 +""" + +EXAMPLE_CONTENT = u"""{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**NOTE:** This is a temporary notebook that gets created for tests." + ] + }, + ], + "metadata": { + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} +""" + + [email protected](autouse=True) +def tmp_module(tmp_path): + """This sets up a temporary directory structure meant to mimic a module + """ + project = tmp_path / "static_module" + project.mkdir() + examples = project / "examples" + examples.mkdir() + (examples / "Test_Example_Notebook.ipynb").write_text(EXAMPLE_CONTENT) + (examples / "datasets.yml").write_text(DATASETS_CONTENT) + (examples / "data").mkdir() + (examples / "data" / ".data_stubs").mkdir() + (examples / "data" / ".data_stubs" / "test_data.csv").write_text(TEST_FILE_CONTENT) + return project + [email protected](autouse=True) +def monkeypatch_find_examples(monkeypatch, tmp_module): + """Monkeypatching find examples to use a tmp examples. + """ + def _find_examples(name): + return os.path.join(str(tmp_module), "examples") + monkeypatch.setattr(pyct.cmd, '_find_examples', _find_examples) + [email protected](scope='function') +def tmp_project(tmp_path): + project = tmp_path / "test_project" + project.mkdir() + return project + [email protected](scope='function') +def tmp_project_with_examples(tmp_path): + project = tmp_path + examples = project / "examples" + examples.mkdir() + datasets = examples / "datasets.yml" + datasets.write_text(DATASETS_CONTENT) + (examples / "data").mkdir() + example = examples / "Test_Example_Notebook.ipynb" + example.write_text(u"Fake notebook contents") + return project + [email protected](scope='function') +def tmp_project_with_stubs(tmp_project_with_examples): + project = tmp_project_with_examples + data_stubs = project / "examples" / "data" / ".data_stubs" + data_stubs.mkdir() + return project + [email protected](scope='function') +def tmp_project_with_test_file(tmp_project_with_stubs): + project = tmp_project_with_stubs + data_stub = project / "examples" / "data" / ".data_stubs" / "test_data.csv" + data_stub.write_text(TEST_FILE_CONTENT) + return project + + +def test_examples_with_use_test_data(tmp_project): + project = tmp_project + path = str(project / "examples") + examples(name="pyct", path=path, use_test_data=True) + assert (project / "examples" / "data" / "test_data.csv").is_file() + assert (project / "examples" / "Test_Example_Notebook.ipynb").is_file() + +def test_examples_with_prexisting_content_in_target_raises_error(tmp_project_with_examples): + project = tmp_project_with_examples + path = str(project / "examples") + data = project / "examples" / "data" / "test_data.csv" + data.write_text(REAL_FILE_CONTENT) + with pytest.raises(ValueError): + examples(name="pyct", path=path, use_test_data=True) + assert (project / "examples" / "Test_Example_Notebook.ipynb").is_file() + assert (project / "examples" / "Test_Example_Notebook.ipynb").read_text() != EXAMPLE_CONTENT + assert (project / "examples" / "data" / "test_data.csv").is_file() + assert (project / "examples" / "data" / "test_data.csv").read_text() == REAL_FILE_CONTENT + +def test_examples_using_test_data_and_force_with_prexisting_content_in_target(tmp_project_with_examples): + project = tmp_project_with_examples + path = str(project / "examples") + data = project / "examples" / "data" / "test_data.csv" + data.write_text(REAL_FILE_CONTENT) + examples(name="pyct", path=path, use_test_data=True, force=True) + assert (project / "examples" / "Test_Example_Notebook.ipynb").is_file() + assert (project / "examples" / "Test_Example_Notebook.ipynb").read_text() == EXAMPLE_CONTENT + assert (project / "examples" / "data" / "test_data.csv").is_file() + assert (project / "examples" / "data" / "test_data.csv").read_text() == TEST_FILE_CONTENT + +def test_copy_examples(tmp_project): + project = tmp_project + path = str(project / "examples") + copy_examples(name="pyct", path=path) + assert (project / "examples" / "Test_Example_Notebook.ipynb").is_file() + +def test_copy_examples_with_prexisting_content_in_target_raises_error(tmp_project_with_examples): + project = tmp_project_with_examples + path = str(project / "examples") + with pytest.raises(ValueError): + copy_examples(name="pyct", path=path) + assert (project / "examples" / "Test_Example_Notebook.ipynb").is_file() + assert (project / "examples" / "Test_Example_Notebook.ipynb").read_text() != EXAMPLE_CONTENT + +def test_copy_examples_using_force_with_prexisting_content_in_target(tmp_project_with_examples): + project = tmp_project_with_examples + path = str(project / "examples") + copy_examples(name="pyct", path=path, force=True) + assert (project / "examples" / "Test_Example_Notebook.ipynb").is_file() + assert (project / "examples" / "Test_Example_Notebook.ipynb").read_text() == EXAMPLE_CONTENT + +def test_fetch_data_using_test_data_with_no_file_in_data_copies_from_stubs(tmp_project_with_test_file): + project = tmp_project_with_test_file + path = str(project / "examples") + fetch_data(name="pyct", path=path, use_test_data=True) + assert (project / "examples" / "data" / "test_data.csv").is_file() + assert (project / "examples" / "data" / "test_data.csv").read_text() == TEST_FILE_CONTENT + +def test_fetch_data_using_test_data_with_file_in_data_skips(tmp_project_with_test_file): + project = tmp_project_with_test_file + path = str(project / "examples") + data = project / "examples" / "data" / "test_data.csv" + data.write_text(REAL_FILE_CONTENT) + fetch_data(name="pyct", path=path, use_test_data=True) + assert (project / "examples" / "data" / "test_data.csv").is_file() + assert (project / "examples" / "data" / "test_data.csv").read_text() == REAL_FILE_CONTENT + +def test_fetch_data_using_test_data_and_force_with_file_in_data_over_writes(tmp_project_with_test_file): + project = tmp_project_with_test_file + path = str(project / "examples") + data = project / "examples" / "data" / "test_data.csv" + data.write_text(REAL_FILE_CONTENT) + fetch_data(name="pyct", path=path, use_test_data=True, force=True) + assert (project / "examples" / "data" / "test_data.csv").is_file() + assert (project / "examples" / "data" / "test_data.csv").read_text() == TEST_FILE_CONTENT + +def test_clean_data_when_data_file_is_real_does_nothing(tmp_project_with_test_file): + project = tmp_project_with_test_file + path = str(project / "examples") + data = project / "examples" / "data" / "test_data.csv" + data.write_text(REAL_FILE_CONTENT) + clean_data(name="pyct", path=path) + assert (project / "examples" / "data" / "test_data.csv").is_file() + assert (project / "examples" / "data" / "test_data.csv").read_text() == REAL_FILE_CONTENT + +def test_clean_data_when_data_file_is_from_stubs_removes_file_from_data(tmp_project_with_test_file): + project = tmp_project_with_test_file + path = str(project / "examples") + data = project / "examples" / "data" / "test_data.csv" + data.write_text(TEST_FILE_CONTENT) + clean_data(name="pyct", path=path) + assert not (project / "examples" / "data" / "test_data.csv").is_file() + assert (project / "examples" / "data" / ".data_stubs" / "test_data.csv").is_file() + assert (project / "examples" / "data" / ".data_stubs" / "test_data.csv").read_text() == TEST_FILE_CONTENT + +def test_clean_data_when_file_not_in_data_does_nothing(tmp_project_with_test_file): + project = tmp_project_with_test_file + path = str(project / "examples") + clean_data(name="pyct", path=path) + assert not (project / "examples" / "data" / "test_data.csv").is_file() + assert (project / "examples" / "data" / ".data_stubs" / "test_data.csv").is_file() + assert (project / "examples" / "data" / ".data_stubs" / "test_data.csv").read_text() == TEST_FILE_CONTENT + +def test_clean_data_when_stubs_is_empty_does_nothing(tmp_project_with_stubs): + project = tmp_project_with_stubs + path = str(project / "examples") + data = project / "examples" / "data" / "test_data.csv" + data.write_text(REAL_FILE_CONTENT) + clean_data(name="pyct", path=path) + assert (project / "examples" / "data" / "test_data.csv").is_file() + assert not (project / "examples" / "data" / ".data_stubs" / "test_data.csv").is_file() + +def test_clean_data_when_no_stubs_dir_does_nothing(tmp_project_with_examples): + project = tmp_project_with_examples + path = str(project / "examples") + data = project / "examples" / "data" / "test_data.csv" + data.write_text(REAL_FILE_CONTENT) + clean_data(name="pyct", path=path) + assert (project / "examples" / "data" / "test_data.csv").is_file() diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pyct-0.4.5/pyct.egg-info/PKG-INFO new/pyct-0.4.6/pyct.egg-info/PKG-INFO --- old/pyct-0.4.5/pyct.egg-info/PKG-INFO 2018-07-12 10:23:10.000000000 +0200 +++ new/pyct-0.4.6/pyct.egg-info/PKG-INFO 2018-11-01 21:36:09.000000000 +0100 @@ -1,6 +1,6 @@ Metadata-Version: 2.1 Name: pyct -Version: 0.4.5 +Version: 0.4.6 Summary: python package common tasks for users (e.g. copy examples, fetch data, ...) Home-page: https://ioam.github.io/parambokeh Author: PyViz @@ -40,13 +40,18 @@ ``` $ datashader examples --help - usage: datashader examples [-h] [--path PATH] [-v] [--force] + usage: datashader examples [-h] [--path PATH] [-v] [--force] [--use-test-data] optional arguments: - -h, --help show this help message and exit - --path PATH location to place examples and data + -h, --help show this help message and exit + --path PATH location to place examples and data -v, --verbose - --force if PATH already exists, force overwrite existing examples if older than source examples + --force if PATH already exists, force overwrite existing examples + if older than source examples. ALSO force any existing data + files to be replaced + --use-test-data Use data's test files, if any, instead of fetching full + data. If test file not in '.data_stubs', fall back to + fetching full data. ``` To copy the examples of e.g. datashader but not download the data, @@ -59,19 +64,26 @@ -h, --help show this help message and exit --path PATH where to copy examples -v, --verbose - --force if PATH already exists, force overwrite existing examples if older than source examples + --force if PATH already exists, force overwrite existing files if + older than source files ``` And to download the data only, the `fetch-data` command: ``` usage: datashader fetch-data [-h] [--path PATH] [--datasets DATASETS] [-v] + [--force] [--use-test-data] optional arguments: -h, --help show this help message and exit --path PATH where to put data - --datasets DATASETS *name* of datasets file; must exist either in path specified by --path or in package/examples/ + --datasets DATASETS *name* of datasets file; must exist either in path + specified by --path or in package/examples/ -v, --verbose + --force Force any existing data files to be replaced + --use-test-data Use data's test files, if any, instead of fetching full + data. If test file not in '.data_stubs', fall back to + fetching full data. ``` Can specify different 'datasets' file: @@ -92,6 +104,49 @@ Skipping Depth data for the Chesapeake and Delaware Bay region of the USA ``` + Can use smaller files instead of large ones by using the `--use-test-data` flag + and placing a small file with the same name in `examples/data/.data_stubs`: + + ``` + $ tree examples/data -a + examples/data + ├── .data_stubs + │ └── nyc_taxi_wide.parq + └── diamonds.csv + + $ cat examples/dataset.yml + data: + + - url: http://s3.amazonaws.com/datashader-data/nyc_taxi_wide.parq + title: 'NYC Taxi Data' + files: + - nyc_taxi_wide.parq + + - url: http://s3.amazonaws.com/datashader-data/maccdc2012_graph.zip + title: 'National CyberWatch Mid-Atlantic Collegiate Cyber Defense Competition' + files: + - maccdc2012_nodes.parq + - maccdc2012_edges.parq + - maccdc2012_full_nodes.parq + - maccdc2012_full_edges.parq + + $ pyviz fetch-data --path=examples --use-test-data + Fetching data defined in /tmp/pyviz/examples/datasets.yml and placing in /tmp/pyviz/examples/data + Copying test data file '/tmp/pyviz/examples/data/.data_stubs/nyc_taxi_wide.parq' to '/tmp/pyviz/examples/data/nyc_taxi_wide.parq' + No test file found for: /tmp/pyviz/examples/data/.data_stubs/maccdc2012_nodes.parq. Using regular file instead + Downloading National CyberWatch Mid-Atlantic Collegiate Cyber Defense Competition 1 of 1 + [################################] 59/59 - 00:00:00 + ``` + + To clean up any potential test files masquerading as real data use `clean-data`: + + ``` + usage: pyviz clean-data [-h] [--path PATH] + + optional arguments: + -h, --help show this help message and exit + --path PATH where to clean data + ``` ## pyct.build @@ -111,6 +166,6 @@ Classifier: Development Status :: 4 - Beta Requires-Python: >=2.7 Description-Content-Type: text/markdown -Provides-Extra: doc Provides-Extra: cmd +Provides-Extra: doc Provides-Extra: tests diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pyct-0.4.5/pyct.egg-info/SOURCES.txt new/pyct-0.4.6/pyct.egg-info/SOURCES.txt --- old/pyct-0.4.5/pyct.egg-info/SOURCES.txt 2018-07-12 10:23:10.000000000 +0200 +++ new/pyct-0.4.6/pyct.egg-info/SOURCES.txt 2018-11-01 21:36:09.000000000 +0100 @@ -11,4 +11,6 @@ pyct.egg-info/SOURCES.txt pyct.egg-info/dependency_links.txt pyct.egg-info/requires.txt -pyct.egg-info/top_level.txt \ No newline at end of file +pyct.egg-info/top_level.txt +pyct/tests/__init__.py +pyct/tests/test_cmd.py \ No newline at end of file diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pyct-0.4.5/pyct.egg-info/requires.txt new/pyct-0.4.6/pyct.egg-info/requires.txt --- old/pyct-0.4.5/pyct.egg-info/requires.txt 2018-07-12 10:23:10.000000000 +0200 +++ new/pyct-0.4.6/pyct.egg-info/requires.txt 2018-11-01 21:36:09.000000000 +0100 @@ -10,3 +10,4 @@ [tests] flake8 +pytest diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pyct-0.4.5/setup.cfg new/pyct-0.4.6/setup.cfg --- old/pyct-0.4.5/setup.cfg 2018-07-12 10:23:10.000000000 +0200 +++ new/pyct-0.4.6/setup.cfg 2018-11-01 21:36:09.000000000 +0100 @@ -37,6 +37,7 @@ requests tests = flake8 + pytest doc = nbsite sphinx_ioam_theme
