Update Python documentation
Project: http://git-wip-us.apache.org/repos/asf/arrow-site/repo Commit: http://git-wip-us.apache.org/repos/asf/arrow-site/commit/6360599f Tree: http://git-wip-us.apache.org/repos/asf/arrow-site/tree/6360599f Diff: http://git-wip-us.apache.org/repos/asf/arrow-site/diff/6360599f Branch: refs/heads/asf-site Commit: 6360599fd8c8a821eea069ff69658a853bcb0122 Parents: 475bbde Author: Wes McKinney <wes.mckin...@twosigma.com> Authored: Mon May 8 00:52:38 2017 -0400 Committer: Wes McKinney <wes.mckin...@twosigma.com> Committed: Mon May 8 00:52:38 2017 -0400 ---------------------------------------------------------------------- docs/python/.buildinfo | 2 +- docs/python/_modules/index.html | 277 +++------ docs/python/_modules/pyarrow.html | 284 +++------- docs/python/_modules/pyarrow/ipc.html | 286 +++------- docs/python/_modules/pyarrow/parquet.html | 291 +++------- docs/python/_sources/api.rst.txt | 19 +- docs/python/_sources/data.rst.txt | 316 +++++++++++ docs/python/_sources/development.rst.txt | 218 +++++++ docs/python/_sources/filesystems.rst.txt | 8 +- .../_sources/generated/pyarrow.Field.rst.txt | 5 +- .../_sources/generated/pyarrow.Schema.rst.txt | 4 +- docs/python/_sources/index.rst.txt | 6 +- docs/python/_sources/install.rst.txt | 117 +--- docs/python/_sources/ipc.rst.txt | 136 +++++ docs/python/_sources/jemalloc.rst.txt | 9 +- docs/python/_sources/memory.rst.txt | 235 ++++++++ docs/python/_sources/pandas.rst.txt | 36 +- docs/python/_sources/parquet.rst.txt | 243 ++++++-- docs/python/_static/contents.png | Bin 0 -> 107 bytes docs/python/_static/navigation.png | Bin 0 -> 120 bytes docs/python/_static/pygments.css | 4 + docs/python/_static/sphinxdoc.css | 345 ++++++++++++ docs/python/api.html | 504 ++++------------- docs/python/data.html | 524 +++++++++++++++++ docs/python/development.html | 291 ++++++++++ docs/python/filesystems.html | 332 ++++------- docs/python/generated/pyarrow.Array.html | 364 ++++-------- docs/python/generated/pyarrow.ArrayValue.html | 361 ++++-------- docs/python/generated/pyarrow.BinaryArray.html | 364 ++++-------- docs/python/generated/pyarrow.BinaryValue.html | 361 ++++-------- docs/python/generated/pyarrow.BooleanArray.html | 364 ++++-------- docs/python/generated/pyarrow.BooleanValue.html | 361 ++++-------- docs/python/generated/pyarrow.Buffer.html | 346 ++++-------- docs/python/generated/pyarrow.BufferReader.html | 346 ++++-------- docs/python/generated/pyarrow.ChunkedArray.html | 343 ++++------- docs/python/generated/pyarrow.Column.html | 343 ++++------- docs/python/generated/pyarrow.DataType.html | 347 ++++-------- docs/python/generated/pyarrow.Date32Array.html | 364 ++++-------- docs/python/generated/pyarrow.Date32Value.html | 361 ++++-------- docs/python/generated/pyarrow.Date64Array.html | 364 ++++-------- docs/python/generated/pyarrow.Date64Value.html | 361 ++++-------- docs/python/generated/pyarrow.DecimalArray.html | 364 ++++-------- docs/python/generated/pyarrow.DecimalType.html | 347 ++++-------- docs/python/generated/pyarrow.DecimalValue.html | 361 ++++-------- .../generated/pyarrow.DictionaryArray.html | 364 ++++-------- .../generated/pyarrow.DictionaryType.html | 347 ++++-------- docs/python/generated/pyarrow.DoubleValue.html | 361 ++++-------- docs/python/generated/pyarrow.Field.html | 372 ++++-------- docs/python/generated/pyarrow.FileReader.html | 340 ++++------- docs/python/generated/pyarrow.FileWriter.html | 340 ++++------- .../generated/pyarrow.FixedSizeBinaryArray.html | 364 ++++-------- .../generated/pyarrow.FixedSizeBinaryType.html | 347 ++++-------- .../generated/pyarrow.FixedSizeBinaryValue.html | 361 ++++-------- docs/python/generated/pyarrow.FloatValue.html | 361 ++++-------- .../generated/pyarrow.FloatingPointArray.html | 364 ++++-------- .../generated/pyarrow.InMemoryOutputStream.html | 346 ++++-------- docs/python/generated/pyarrow.Int16Array.html | 364 ++++-------- docs/python/generated/pyarrow.Int16Value.html | 361 ++++-------- docs/python/generated/pyarrow.Int32Array.html | 364 ++++-------- docs/python/generated/pyarrow.Int32Value.html | 361 ++++-------- docs/python/generated/pyarrow.Int64Array.html | 364 ++++-------- docs/python/generated/pyarrow.Int64Value.html | 361 ++++-------- docs/python/generated/pyarrow.Int8Array.html | 364 ++++-------- docs/python/generated/pyarrow.Int8Value.html | 361 ++++-------- docs/python/generated/pyarrow.IntegerArray.html | 364 ++++-------- docs/python/generated/pyarrow.ListArray.html | 364 ++++-------- docs/python/generated/pyarrow.ListValue.html | 361 ++++-------- .../generated/pyarrow.MemoryMappedFile.html | 346 ++++-------- docs/python/generated/pyarrow.MemoryPool.html | 343 ++++------- docs/python/generated/pyarrow.NA.html | 359 ++++-------- docs/python/generated/pyarrow.NAType.html | 361 ++++-------- docs/python/generated/pyarrow.NativeFile.html | 346 ++++-------- docs/python/generated/pyarrow.NullArray.html | 364 ++++-------- docs/python/generated/pyarrow.NumericArray.html | 364 ++++-------- docs/python/generated/pyarrow.PythonFile.html | 346 ++++-------- docs/python/generated/pyarrow.RecordBatch.html | 345 ++++-------- docs/python/generated/pyarrow.Scalar.html | 361 ++++-------- docs/python/generated/pyarrow.Schema.html | 363 ++++-------- docs/python/generated/pyarrow.StreamReader.html | 340 ++++------- docs/python/generated/pyarrow.StreamWriter.html | 340 ++++------- docs/python/generated/pyarrow.StringArray.html | 364 ++++-------- docs/python/generated/pyarrow.StringValue.html | 361 ++++-------- docs/python/generated/pyarrow.Table.html | 343 ++++------- docs/python/generated/pyarrow.Tensor.html | 342 ++++------- docs/python/generated/pyarrow.Time32Array.html | 364 ++++-------- docs/python/generated/pyarrow.Time32Type.html | 347 ++++-------- docs/python/generated/pyarrow.Time64Array.html | 364 ++++-------- docs/python/generated/pyarrow.Time64Type.html | 347 ++++-------- .../generated/pyarrow.TimestampArray.html | 364 ++++-------- .../python/generated/pyarrow.TimestampType.html | 347 ++++-------- .../generated/pyarrow.TimestampValue.html | 361 ++++-------- docs/python/generated/pyarrow.UInt16Array.html | 364 ++++-------- docs/python/generated/pyarrow.UInt16Value.html | 361 ++++-------- docs/python/generated/pyarrow.UInt32Array.html | 364 ++++-------- docs/python/generated/pyarrow.UInt32Value.html | 361 ++++-------- docs/python/generated/pyarrow.UInt64Array.html | 364 ++++-------- docs/python/generated/pyarrow.UInt64Value.html | 361 ++++-------- docs/python/generated/pyarrow.UInt8Array.html | 364 ++++-------- docs/python/generated/pyarrow.UInt8Value.html | 361 ++++-------- docs/python/generated/pyarrow.array.html | 362 ++++-------- docs/python/generated/pyarrow.binary.html | 363 ++++-------- docs/python/generated/pyarrow.bool_.html | 363 ++++-------- .../generated/pyarrow.create_memory_map.html | 344 ++++------- docs/python/generated/pyarrow.date32.html | 363 ++++-------- docs/python/generated/pyarrow.date64.html | 363 ++++-------- docs/python/generated/pyarrow.decimal.html | 363 ++++-------- .../generated/pyarrow.default_memory_pool.html | 341 ++++------- docs/python/generated/pyarrow.dictionary.html | 363 ++++-------- docs/python/generated/pyarrow.field.html | 385 ++++--------- docs/python/generated/pyarrow.float16.html | 363 ++++-------- docs/python/generated/pyarrow.float32.html | 363 ++++-------- docs/python/generated/pyarrow.float64.html | 363 ++++-------- .../generated/pyarrow.from_numpy_dtype.html | 363 ++++-------- .../pyarrow.get_record_batch_size.html | 341 ++++------- .../generated/pyarrow.get_tensor_size.html | 340 ++++------- docs/python/generated/pyarrow.int16.html | 363 ++++-------- docs/python/generated/pyarrow.int32.html | 363 ++++-------- docs/python/generated/pyarrow.int64.html | 363 ++++-------- docs/python/generated/pyarrow.int8.html | 363 ++++-------- .../generated/pyarrow.jemalloc_memory_pool.html | 341 ++++------- docs/python/generated/pyarrow.list_.html | 363 ++++-------- docs/python/generated/pyarrow.memory_map.html | 344 ++++------- docs/python/generated/pyarrow.null.html | 363 ++++-------- .../pyarrow.parquet.ParquetDataset.html | 341 ++++------- .../generated/pyarrow.parquet.ParquetFile.html | 341 ++++------- .../generated/pyarrow.parquet.read_table.html | 341 ++++------- .../pyarrow.parquet.write_metadata.html | 341 ++++------- .../generated/pyarrow.parquet.write_table.html | 341 ++++------- docs/python/generated/pyarrow.read_tensor.html | 340 ++++------- docs/python/generated/pyarrow.schema.html | 376 ++++--------- .../generated/pyarrow.set_memory_pool.html | 341 ++++------- docs/python/generated/pyarrow.string.html | 363 ++++-------- docs/python/generated/pyarrow.struct.html | 363 ++++-------- docs/python/generated/pyarrow.time32.html | 363 ++++-------- docs/python/generated/pyarrow.time64.html | 363 ++++-------- docs/python/generated/pyarrow.timestamp.html | 363 ++++-------- .../pyarrow.total_allocated_bytes.html | 341 ++++------- docs/python/generated/pyarrow.uint16.html | 363 ++++-------- docs/python/generated/pyarrow.uint32.html | 363 ++++-------- docs/python/generated/pyarrow.uint64.html | 363 ++++-------- docs/python/generated/pyarrow.uint8.html | 363 ++++-------- docs/python/generated/pyarrow.write_tensor.html | 340 ++++------- docs/python/genindex.html | 276 +++------ docs/python/getting_involved.html | 317 ++++------- docs/python/index.html | 354 ++++-------- docs/python/install.html | 438 ++++----------- docs/python/ipc.html | Bin 0 -> 17201 bytes docs/python/jemalloc.html | 313 +++-------- docs/python/memory.html | Bin 0 -> 25705 bytes docs/python/objects.inv | Bin 2960 -> 3385 bytes docs/python/pandas.html | 366 ++++-------- docs/python/parquet.html | 563 ++++++++++--------- docs/python/search.html | 303 +++------- docs/python/searchindex.js | 2 +- 154 files changed, 15085 insertions(+), 33715 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/.buildinfo ---------------------------------------------------------------------- diff --git a/docs/python/.buildinfo b/docs/python/.buildinfo index cfff128..87ebcea 100644 --- a/docs/python/.buildinfo +++ b/docs/python/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 8a14f2870997e906f4c89eca99e5d190 +config: 0da6273003dcdb1b32e160e026bf7ef6 tags: 645f666f9bcd5a90fca523b33c5a78b7 http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_modules/index.html ---------------------------------------------------------------------- diff --git a/docs/python/_modules/index.html b/docs/python/_modules/index.html index e1d0045..88617d8 100644 --- a/docs/python/_modules/index.html +++ b/docs/python/_modules/index.html @@ -1,161 +1,62 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> -<!DOCTYPE html> -<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> -<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> -<head> - <meta charset="utf-8"> - - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - - <title>Overview: module code — pyarrow documentation</title> - - - - - - - - - - - +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - - - - - - <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../genindex.html"/> - <link rel="search" title="Search" href="../search.html"/> - <link rel="top" title="pyarrow documentation" href="../index.html"/> - - - <script src="../_static/js/modernizr.min.js"></script> - -</head> - -<body class="wy-body-for-nav" role="document"> - - - <div class="wy-grid-for-nav"> - + <title>Overview: module code — pyarrow documentation</title> - <nav data-toggle="wy-nav-shift" class="wy-nav-side"> - <div class="wy-side-scroll"> - <div class="wy-side-nav-search"> - - - - <a href="../index.html" class="icon icon-home"> pyarrow - - - - </a> - - - - - - - -<div role="search"> - <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> - <input type="text" name="q" placeholder="Search docs" /> - <input type="hidden" name="check_keywords" value="yes" /> - <input type="hidden" name="area" value="default" /> - </form> + <link rel="stylesheet" href="../_static/sphinxdoc.css" type="text/css" /> + <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: '../', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' + }; + </script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> + <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> + <link rel="index" title="Index" href="../genindex.html" /> + <link rel="search" title="Search" href="../search.html" /> + </head> + <body role="document"> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="../genindex.html" title="General Index" + accesskey="I">index</a></li> + <li class="nav-item nav-item-0"><a href="../index.html">pyarrow documentation</a> »</li> + </ul> + </div> + <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> + <div class="sphinxsidebarwrapper"> +<div id="searchbox" style="display: none" role="search"> + <h3>Quick search</h3> + <form class="search" action="../search.html" method="get"> + <div><input type="text" name="q" /></div> + <div><input type="submit" value="Go" /></div> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> </div> - - - </div> - - <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> - - - - - - - <p class="caption"><span class="caption-text">Getting Started</span></p> -<ul> -<li class="toctree-l1"><a class="reference internal" href="../install.html">Install PyArrow</a></li> -<li class="toctree-l1"><a class="reference internal" href="../pandas.html">Pandas Interface</a></li> -<li class="toctree-l1"><a class="reference internal" href="../filesystems.html">File interfaces and Memory Maps</a></li> -<li class="toctree-l1"><a class="reference internal" href="../parquet.html">Reading/Writing Parquet files</a></li> -<li class="toctree-l1"><a class="reference internal" href="../api.html">API Reference</a></li> -<li class="toctree-l1"><a class="reference internal" href="../getting_involved.html">Getting Involved</a></li> -</ul> -<p class="caption"><span class="caption-text">Additional Features</span></p> -<ul> -<li class="toctree-l1"><a class="reference internal" href="../jemalloc.html">jemalloc MemoryPool</a></li> -</ul> - - - +<script type="text/javascript">$('#searchbox').show(0);</script> </div> </div> - </nav> - - <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - - - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href="../index.html">pyarrow</a> - - </nav> - - - - <div class="wy-nav-content"> - <div class="rst-content"> - - - - - - - - - - - - - - - -<div role="navigation" aria-label="breadcrumbs navigation"> - - <ul class="wy-breadcrumbs"> - - <li><a href="../index.html">Docs</a> »</li> - - <li>Overview: module code</li> - - - <li class="wy-breadcrumbs-aside"> - - - - </li> - - </ul> - - - <hr/> -</div> - <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> - <div itemprop="articleBody"> + <div class="document"> + <div class="documentwrapper"> + <div class="bodywrapper"> + <div class="body" role="main"> <h1>All modules for which code is available</h1> <ul><li><a href="pyarrow.html">pyarrow</a></li> @@ -167,67 +68,23 @@ <li><a href="pyarrow/parquet.html">pyarrow.parquet</a></li> </ul></ul> - </div> - <div class="articleComments"> - - </div> </div> - <footer> - - - <hr/> - - <div role="contentinfo"> - <p> - © Copyright 2016 Apache Software Foundation. - - </p> - </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. - -</footer> - </div> </div> - - </section> - - </div> - - - - - - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'../', - VERSION:'', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="../_static/jquery.js"></script> - <script type="text/javascript" src="../_static/underscore.js"></script> - <script type="text/javascript" src="../_static/doctools.js"></script> - <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> - - - - - - <script type="text/javascript" src="../_static/js/theme.js"></script> - - - - - <script type="text/javascript"> - jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); - }); - </script> - - -</body> + <div class="clearer"></div> + </div> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="../genindex.html" title="General Index" + >index</a></li> + <li class="nav-item nav-item-0"><a href="../index.html">pyarrow documentation</a> »</li> + </ul> + </div> + <div class="footer" role="contentinfo"> + © Copyright 2016-2017 Apache Software Foundation. + Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 1.5.5. + </div> + </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_modules/pyarrow.html ---------------------------------------------------------------------- diff --git a/docs/python/_modules/pyarrow.html b/docs/python/_modules/pyarrow.html index b5ef93c..aed49c6 100644 --- a/docs/python/_modules/pyarrow.html +++ b/docs/python/_modules/pyarrow.html @@ -1,164 +1,63 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> -<!DOCTYPE html> -<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> -<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> -<head> - <meta charset="utf-8"> - - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - - <title>pyarrow — pyarrow documentation</title> - - - - - - - - - - - +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - - - - - - <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../genindex.html"/> - <link rel="search" title="Search" href="../search.html"/> - <link rel="top" title="pyarrow documentation" href="../index.html"/> - <link rel="up" title="Module code" href="index.html"/> - - - <script src="../_static/js/modernizr.min.js"></script> - -</head> - -<body class="wy-body-for-nav" role="document"> - - - <div class="wy-grid-for-nav"> - + <title>pyarrow — pyarrow documentation</title> - <nav data-toggle="wy-nav-shift" class="wy-nav-side"> - <div class="wy-side-scroll"> - <div class="wy-side-nav-search"> - - - - <a href="../index.html" class="icon icon-home"> pyarrow - - - - </a> - - - - - - - -<div role="search"> - <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> - <input type="text" name="q" placeholder="Search docs" /> - <input type="hidden" name="check_keywords" value="yes" /> - <input type="hidden" name="area" value="default" /> - </form> + <link rel="stylesheet" href="../_static/sphinxdoc.css" type="text/css" /> + <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: '../', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' + }; + </script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> + <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> + <link rel="index" title="Index" href="../genindex.html" /> + <link rel="search" title="Search" href="../search.html" /> + </head> + <body role="document"> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="../genindex.html" title="General Index" + accesskey="I">index</a></li> + <li class="nav-item nav-item-0"><a href="../index.html">pyarrow documentation</a> »</li> + <li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Module code</a> »</li> + </ul> + </div> + <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> + <div class="sphinxsidebarwrapper"> +<div id="searchbox" style="display: none" role="search"> + <h3>Quick search</h3> + <form class="search" action="../search.html" method="get"> + <div><input type="text" name="q" /></div> + <div><input type="submit" value="Go" /></div> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> </div> - - - </div> - - <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> - - - - - - - <p class="caption"><span class="caption-text">Getting Started</span></p> -<ul> -<li class="toctree-l1"><a class="reference internal" href="../install.html">Install PyArrow</a></li> -<li class="toctree-l1"><a class="reference internal" href="../pandas.html">Pandas Interface</a></li> -<li class="toctree-l1"><a class="reference internal" href="../filesystems.html">File interfaces and Memory Maps</a></li> -<li class="toctree-l1"><a class="reference internal" href="../parquet.html">Reading/Writing Parquet files</a></li> -<li class="toctree-l1"><a class="reference internal" href="../api.html">API Reference</a></li> -<li class="toctree-l1"><a class="reference internal" href="../getting_involved.html">Getting Involved</a></li> -</ul> -<p class="caption"><span class="caption-text">Additional Features</span></p> -<ul> -<li class="toctree-l1"><a class="reference internal" href="../jemalloc.html">jemalloc MemoryPool</a></li> -</ul> - - - +<script type="text/javascript">$('#searchbox').show(0);</script> </div> </div> - </nav> - - <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - - - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href="../index.html">pyarrow</a> - - </nav> - - - - <div class="wy-nav-content"> - <div class="rst-content"> - - - - - - - - - - - - - - - -<div role="navigation" aria-label="breadcrumbs navigation"> - - <ul class="wy-breadcrumbs"> - - <li><a href="../index.html">Docs</a> »</li> - - <li><a href="index.html">Module code</a> »</li> - - <li>pyarrow</li> - - - <li class="wy-breadcrumbs-aside"> - - - - </li> - - </ul> - - - <hr/> -</div> - <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> - <div itemprop="articleBody"> + <div class="document"> + <div class="documentwrapper"> + <div class="bodywrapper"> + <div class="body" role="main"> <h1>Source code for pyarrow</h1><div class="highlight"><pre> <span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span> @@ -182,7 +81,7 @@ <span class="kn">from</span> <span class="nn">pkg_resources</span> <span class="k">import</span> <span class="n">get_distribution</span><span class="p">,</span> <span class="n">DistributionNotFound</span> <span class="k">try</span><span class="p">:</span> - <span class="n">__version__</span> <span class="o">=</span> <span class="n">get_distribution</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span><span class="o">.</span><span class="n">version</span> + <span class="n">__version__</span> <span class="o">=</span> <span class="n">get_distribution</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span><span class="o">.</span><span class="n">version</span> <span class="k">except</span> <span class="n">DistributionNotFound</span><span class="p">:</span> <span class="c1"># package is not installed</span> <span class="k">pass</span> @@ -272,67 +171,24 @@ <span class="n">localfs</span> <span class="o">=</span> <span class="n">LocalFilesystem</span><span class="o">.</span><span class="n">get_instance</span><span class="p">()</span> </pre></div> - </div> - <div class="articleComments"> - - </div> </div> - <footer> - - - <hr/> - - <div role="contentinfo"> - <p> - © Copyright 2016 Apache Software Foundation. - - </p> - </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. - -</footer> - </div> </div> - - </section> - - </div> - - - - - - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'../', - VERSION:'', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="../_static/jquery.js"></script> - <script type="text/javascript" src="../_static/underscore.js"></script> - <script type="text/javascript" src="../_static/doctools.js"></script> - <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> - - - - - - <script type="text/javascript" src="../_static/js/theme.js"></script> - - - - - <script type="text/javascript"> - jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); - }); - </script> - - -</body> + <div class="clearer"></div> + </div> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="../genindex.html" title="General Index" + >index</a></li> + <li class="nav-item nav-item-0"><a href="../index.html">pyarrow documentation</a> »</li> + <li class="nav-item nav-item-1"><a href="index.html" >Module code</a> »</li> + </ul> + </div> + <div class="footer" role="contentinfo"> + © Copyright 2016-2017 Apache Software Foundation. + Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 1.5.5. + </div> + </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_modules/pyarrow/ipc.html ---------------------------------------------------------------------- diff --git a/docs/python/_modules/pyarrow/ipc.html b/docs/python/_modules/pyarrow/ipc.html index aaaa165..750e2a3 100644 --- a/docs/python/_modules/pyarrow/ipc.html +++ b/docs/python/_modules/pyarrow/ipc.html @@ -1,166 +1,64 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> -<!DOCTYPE html> -<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> -<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> -<head> - <meta charset="utf-8"> - - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - - <title>pyarrow.ipc — pyarrow documentation</title> - - - - - - - - - - - +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - - - - - - <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../../genindex.html"/> - <link rel="search" title="Search" href="../../search.html"/> - <link rel="top" title="pyarrow documentation" href="../../index.html"/> - <link rel="up" title="pyarrow" href="../pyarrow.html"/> - - - <script src="../../_static/js/modernizr.min.js"></script> - -</head> - -<body class="wy-body-for-nav" role="document"> - - - <div class="wy-grid-for-nav"> - + <title>pyarrow.ipc — pyarrow documentation</title> - <nav data-toggle="wy-nav-shift" class="wy-nav-side"> - <div class="wy-side-scroll"> - <div class="wy-side-nav-search"> - - - - <a href="../../index.html" class="icon icon-home"> pyarrow - - - - </a> - - - - - - - -<div role="search"> - <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> - <input type="text" name="q" placeholder="Search docs" /> - <input type="hidden" name="check_keywords" value="yes" /> - <input type="hidden" name="area" value="default" /> - </form> + <link rel="stylesheet" href="../../_static/sphinxdoc.css" type="text/css" /> + <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: '../../', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' + }; + </script> + <script type="text/javascript" src="../../_static/jquery.js"></script> + <script type="text/javascript" src="../../_static/underscore.js"></script> + <script type="text/javascript" src="../../_static/doctools.js"></script> + <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> + <link rel="index" title="Index" href="../../genindex.html" /> + <link rel="search" title="Search" href="../../search.html" /> + </head> + <body role="document"> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="../../genindex.html" title="General Index" + accesskey="I">index</a></li> + <li class="nav-item nav-item-0"><a href="../../index.html">pyarrow documentation</a> »</li> + <li class="nav-item nav-item-1"><a href="../index.html" >Module code</a> »</li> + <li class="nav-item nav-item-2"><a href="../pyarrow.html" accesskey="U">pyarrow</a> »</li> + </ul> + </div> + <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> + <div class="sphinxsidebarwrapper"> +<div id="searchbox" style="display: none" role="search"> + <h3>Quick search</h3> + <form class="search" action="../../search.html" method="get"> + <div><input type="text" name="q" /></div> + <div><input type="submit" value="Go" /></div> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> </div> - - - </div> - - <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> - - - - - - - <p class="caption"><span class="caption-text">Getting Started</span></p> -<ul> -<li class="toctree-l1"><a class="reference internal" href="../../install.html">Install PyArrow</a></li> -<li class="toctree-l1"><a class="reference internal" href="../../pandas.html">Pandas Interface</a></li> -<li class="toctree-l1"><a class="reference internal" href="../../filesystems.html">File interfaces and Memory Maps</a></li> -<li class="toctree-l1"><a class="reference internal" href="../../parquet.html">Reading/Writing Parquet files</a></li> -<li class="toctree-l1"><a class="reference internal" href="../../api.html">API Reference</a></li> -<li class="toctree-l1"><a class="reference internal" href="../../getting_involved.html">Getting Involved</a></li> -</ul> -<p class="caption"><span class="caption-text">Additional Features</span></p> -<ul> -<li class="toctree-l1"><a class="reference internal" href="../../jemalloc.html">jemalloc MemoryPool</a></li> -</ul> - - - +<script type="text/javascript">$('#searchbox').show(0);</script> </div> </div> - </nav> - - <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - - - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href="../../index.html">pyarrow</a> - - </nav> - - - - <div class="wy-nav-content"> - <div class="rst-content"> - - - - - - - - - - - - - - - -<div role="navigation" aria-label="breadcrumbs navigation"> - - <ul class="wy-breadcrumbs"> - - <li><a href="../../index.html">Docs</a> »</li> - - <li><a href="../index.html">Module code</a> »</li> - - <li><a href="../pyarrow.html">pyarrow</a> »</li> - - <li>pyarrow.ipc</li> - - - <li class="wy-breadcrumbs-aside"> - - - - </li> - - </ul> - - - <hr/> -</div> - <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> - <div itemprop="articleBody"> + <div class="document"> + <div class="documentwrapper"> + <div class="bodywrapper"> + <div class="body" role="main"> <h1>Source code for pyarrow.ipc</h1><div class="highlight"><pre> <span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span> @@ -248,67 +146,25 @@ <span class="bp">self</span><span class="o">.</span><span class="n">_open</span><span class="p">(</span><span class="n">sink</span><span class="p">,</span> <span class="n">schema</span><span class="p">)</span></div></div> </pre></div> - </div> - <div class="articleComments"> - - </div> </div> - <footer> - - - <hr/> - - <div role="contentinfo"> - <p> - © Copyright 2016 Apache Software Foundation. - - </p> - </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. - -</footer> - </div> </div> - - </section> - - </div> - - - - - - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'../../', - VERSION:'', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="../../_static/jquery.js"></script> - <script type="text/javascript" src="../../_static/underscore.js"></script> - <script type="text/javascript" src="../../_static/doctools.js"></script> - <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> - - - - - - <script type="text/javascript" src="../../_static/js/theme.js"></script> - - - - - <script type="text/javascript"> - jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); - }); - </script> - - -</body> + <div class="clearer"></div> + </div> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="../../genindex.html" title="General Index" + >index</a></li> + <li class="nav-item nav-item-0"><a href="../../index.html">pyarrow documentation</a> »</li> + <li class="nav-item nav-item-1"><a href="../index.html" >Module code</a> »</li> + <li class="nav-item nav-item-2"><a href="../pyarrow.html" >pyarrow</a> »</li> + </ul> + </div> + <div class="footer" role="contentinfo"> + © Copyright 2016-2017 Apache Software Foundation. + Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 1.5.5. + </div> + </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_modules/pyarrow/parquet.html ---------------------------------------------------------------------- diff --git a/docs/python/_modules/pyarrow/parquet.html b/docs/python/_modules/pyarrow/parquet.html index ab582d2..7a386b9 100644 --- a/docs/python/_modules/pyarrow/parquet.html +++ b/docs/python/_modules/pyarrow/parquet.html @@ -1,166 +1,64 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> -<!DOCTYPE html> -<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> -<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> -<head> - <meta charset="utf-8"> - - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - - <title>pyarrow.parquet — pyarrow documentation</title> - - - - - - - - - - - +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - - - - - - <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../../genindex.html"/> - <link rel="search" title="Search" href="../../search.html"/> - <link rel="top" title="pyarrow documentation" href="../../index.html"/> - <link rel="up" title="pyarrow" href="../pyarrow.html"/> - - - <script src="../../_static/js/modernizr.min.js"></script> - -</head> - -<body class="wy-body-for-nav" role="document"> - - - <div class="wy-grid-for-nav"> - + <title>pyarrow.parquet — pyarrow documentation</title> - <nav data-toggle="wy-nav-shift" class="wy-nav-side"> - <div class="wy-side-scroll"> - <div class="wy-side-nav-search"> - - - - <a href="../../index.html" class="icon icon-home"> pyarrow - - - - </a> - - - - - - - -<div role="search"> - <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> - <input type="text" name="q" placeholder="Search docs" /> - <input type="hidden" name="check_keywords" value="yes" /> - <input type="hidden" name="area" value="default" /> - </form> + <link rel="stylesheet" href="../../_static/sphinxdoc.css" type="text/css" /> + <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: '../../', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' + }; + </script> + <script type="text/javascript" src="../../_static/jquery.js"></script> + <script type="text/javascript" src="../../_static/underscore.js"></script> + <script type="text/javascript" src="../../_static/doctools.js"></script> + <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> + <link rel="index" title="Index" href="../../genindex.html" /> + <link rel="search" title="Search" href="../../search.html" /> + </head> + <body role="document"> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="../../genindex.html" title="General Index" + accesskey="I">index</a></li> + <li class="nav-item nav-item-0"><a href="../../index.html">pyarrow documentation</a> »</li> + <li class="nav-item nav-item-1"><a href="../index.html" >Module code</a> »</li> + <li class="nav-item nav-item-2"><a href="../pyarrow.html" accesskey="U">pyarrow</a> »</li> + </ul> + </div> + <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> + <div class="sphinxsidebarwrapper"> +<div id="searchbox" style="display: none" role="search"> + <h3>Quick search</h3> + <form class="search" action="../../search.html" method="get"> + <div><input type="text" name="q" /></div> + <div><input type="submit" value="Go" /></div> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> </div> - - - </div> - - <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> - - - - - - - <p class="caption"><span class="caption-text">Getting Started</span></p> -<ul> -<li class="toctree-l1"><a class="reference internal" href="../../install.html">Install PyArrow</a></li> -<li class="toctree-l1"><a class="reference internal" href="../../pandas.html">Pandas Interface</a></li> -<li class="toctree-l1"><a class="reference internal" href="../../filesystems.html">File interfaces and Memory Maps</a></li> -<li class="toctree-l1"><a class="reference internal" href="../../parquet.html">Reading/Writing Parquet files</a></li> -<li class="toctree-l1"><a class="reference internal" href="../../api.html">API Reference</a></li> -<li class="toctree-l1"><a class="reference internal" href="../../getting_involved.html">Getting Involved</a></li> -</ul> -<p class="caption"><span class="caption-text">Additional Features</span></p> -<ul> -<li class="toctree-l1"><a class="reference internal" href="../../jemalloc.html">jemalloc MemoryPool</a></li> -</ul> - - - +<script type="text/javascript">$('#searchbox').show(0);</script> </div> </div> - </nav> - - <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - - - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href="../../index.html">pyarrow</a> - - </nav> - - - - <div class="wy-nav-content"> - <div class="rst-content"> - - - - - - - - - - - - - - - -<div role="navigation" aria-label="breadcrumbs navigation"> - - <ul class="wy-breadcrumbs"> - - <li><a href="../../index.html">Docs</a> »</li> - - <li><a href="../index.html">Module code</a> »</li> - - <li><a href="../pyarrow.html">pyarrow</a> »</li> - - <li>pyarrow.parquet</li> - - - <li class="wy-breadcrumbs-aside"> - - - - </li> - - </ul> - - - <hr/> -</div> - <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> - <div itemprop="articleBody"> + <div class="document"> + <div class="documentwrapper"> + <div class="bodywrapper"> + <div class="body" role="main"> <h1>Source code for pyarrow.parquet</h1><div class="highlight"><pre> <span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span> @@ -315,7 +213,7 @@ <span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="p">(</span><span class="s1">'</span><span class="si">{0}</span><span class="s1">(</span><span class="si">{1!r}</span><span class="s1">, row_group=</span><span class="si">{2!r}</span><span class="s1">, partition_keys=</span><span class="si">{3!r}</span><span class="s1">)'</span> - <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">path</span><span class="p">,</span> + <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">path</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">row_group</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">partition_keys</span><span class="p">))</span> @@ -373,6 +271,9 @@ <span class="n">reader</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_open</span><span class="p">(</span><span class="n">open_file_func</span><span class="p">)</span> <span class="k">elif</span> <span class="n">file</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> <span class="n">reader</span> <span class="o">=</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="n">file</span><span class="p">)</span> + <span class="k">else</span><span class="p">:</span> + <span class="c1"># try to read the local path</span> + <span class="n">reader</span> <span class="o">=</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">path</span><span class="p">)</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">row_group</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> <span class="n">table</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">read_row_group</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">row_group</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span> @@ -825,67 +726,25 @@ <span class="n">writer</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></div> </pre></div> - </div> - <div class="articleComments"> - - </div> </div> - <footer> - - - <hr/> - - <div role="contentinfo"> - <p> - © Copyright 2016 Apache Software Foundation. - - </p> - </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. - -</footer> - </div> </div> - - </section> - - </div> - - - - - - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'../../', - VERSION:'', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="../../_static/jquery.js"></script> - <script type="text/javascript" src="../../_static/underscore.js"></script> - <script type="text/javascript" src="../../_static/doctools.js"></script> - <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> - - - - - - <script type="text/javascript" src="../../_static/js/theme.js"></script> - - - - - <script type="text/javascript"> - jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); - }); - </script> - - -</body> + <div class="clearer"></div> + </div> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="../../genindex.html" title="General Index" + >index</a></li> + <li class="nav-item nav-item-0"><a href="../../index.html">pyarrow documentation</a> »</li> + <li class="nav-item nav-item-1"><a href="../index.html" >Module code</a> »</li> + <li class="nav-item nav-item-2"><a href="../pyarrow.html" >pyarrow</a> »</li> + </ul> + </div> + <div class="footer" role="contentinfo"> + © Copyright 2016-2017 Apache Software Foundation. + Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 1.5.5. + </div> + </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/api.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/api.rst.txt b/docs/python/_sources/api.rst.txt index 08a0694..a8dd8c5 100644 --- a/docs/python/_sources/api.rst.txt +++ b/docs/python/_sources/api.rst.txt @@ -22,7 +22,7 @@ API Reference ************* -.. _api.functions: +.. _api.types: Type and Schema Factory Functions --------------------------------- @@ -58,6 +58,8 @@ Type and Schema Factory Functions schema from_numpy_dtype +.. _api.value: + Scalar Value Types ------------------ @@ -88,6 +90,7 @@ Scalar Value Types TimestampValue DecimalValue +.. _api.array: Array Types and Constructors ---------------------------- @@ -122,6 +125,8 @@ Array Types and Constructors DecimalArray ListArray +.. _api.table: + Tables and Record Batches ------------------------- @@ -134,6 +139,8 @@ Tables and Record Batches Table get_record_batch_size +.. _api.tensor: + Tensor type and Functions ------------------------- @@ -145,6 +152,8 @@ Tensor type and Functions get_tensor_size read_tensor +.. _api.io: + Input / Output and Shared Memory -------------------------------- @@ -160,6 +169,8 @@ Input / Output and Shared Memory create_memory_map PythonFile +.. _api.ipc: + Interprocess Communication and Messaging ---------------------------------------- @@ -171,6 +182,8 @@ Interprocess Communication and Messaging StreamReader StreamWriter +.. _api.memory_pool: + Memory Pools ------------ @@ -183,6 +196,8 @@ Memory Pools total_allocated_bytes set_memory_pool +.. _api.type_classes: + Type Classes ------------ @@ -201,6 +216,8 @@ Type Classes .. currentmodule:: pyarrow.parquet +.. _api.parquet: + Apache Parquet -------------- http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/data.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/data.rst.txt b/docs/python/_sources/data.rst.txt new file mode 100644 index 0000000..04e74ae --- /dev/null +++ b/docs/python/_sources/data.rst.txt @@ -0,0 +1,316 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _data: + +In-Memory Data Model +==================== + +Apache Arrow defines columnar array data structures by composing type metadata +with memory buffers, like the ones explained in the documentation on +:ref:`Memory and IO <io>`. These data structures are exposed in Python through +a series of interrelated classes: + +* **Type Metadata**: Instances of ``pyarrow.DataType``, which describe a logical + array type +* **Schemas**: Instances of ``pyarrow.Schema``, which describe a named + collection of types. These can be thought of as the column types in a + table-like object. +* **Arrays**: Instances of ``pyarrow.Array``, which are atomic, contiguous + columnar data structures composed from Arrow Buffer objects +* **Record Batches**: Instances of ``pyarrow.RecordBatch``, which are a + collection of Array objects with a particular Schema +* **Tables**: Instances of ``pyarrow.Table``, a logical table data structure in + which each column consists of one or more ``pyarrow.Array`` objects of the + same type. + +We will examine these in the sections below in a series of examples. + +.. _data.types: + +Type Metadata +------------- + +Apache Arrow defines language agnostic column-oriented data structures for +array data. These include: + +* **Fixed-length primitive types**: numbers, booleans, date and times, fixed + size binary, decimals, and other values that fit into a given number +* **Variable-length primitive types**: binary, string +* **Nested types**: list, struct, and union +* **Dictionary type**: An encoded categorical type (more on this later) + +Each logical data type in Arrow has a corresponding factory function for +creating an instance of that type object in Python: + +.. ipython:: python + + import pyarrow as pa + t1 = pa.int32() + t2 = pa.string() + t3 = pa.binary() + t4 = pa.binary(10) + t5 = pa.timestamp('ms') + + t1 + print(t1) + print(t4) + print(t5) + +We use the name **logical type** because the **physical** storage may be the +same for one or more types. For example, ``int64``, ``float64``, and +``timestamp[ms]`` all occupy 64 bits per value. + +These objects are `metadata`; they are used for describing the data in arrays, +schemas, and record batches. In Python, they can be used in functions where the +input data (e.g. Python objects) may be coerced to more than one Arrow type. + +The :class:`~pyarrow.Field` type is a type plus a name and optional +user-defined metadata: + +.. ipython:: python + + f0 = pa.field('int32_field', t1) + f0 + f0.name + f0.type + +Arrow supports **nested value types** like list, struct, and union. When +creating these, you must pass types or fields to indicate the data types of the +types' children. For example, we can define a list of int32 values with: + +.. ipython:: python + + t6 = pa.list_(t1) + t6 + +A `struct` is a collection of named fields: + +.. ipython:: python + + fields = [ + pa.field('s0', t1), + pa.field('s1', t2), + pa.field('s2', t4), + pa.field('s3', t6) + ] + + t7 = pa.struct(fields) + print(t7) + +See :ref:`Data Types API <api.types>` for a full listing of data type +functions. + +.. _data.schema: + +Schemas +------- + +The :class:`~pyarrow.Schema` type is similar to the ``struct`` array type; it +defines the column names and types in a record batch or table data +structure. The ``pyarrow.schema`` factory function makes new Schema objects in +Python: + +.. ipython:: python + + fields = [ + pa.field('s0', t1), + pa.field('s1', t2), + pa.field('s2', t4), + pa.field('s3', t6) + ] + + my_schema = pa.schema(fields) + my_schema + +In some applications, you may not create schemas directly, only using the ones +that are embedded in :ref:`IPC messages <ipc>`. + +.. _data.array: + +Arrays +------ + +For each data type, there is an accompanying array data structure for holding +memory buffers that define a single contiguous chunk of columnar array +data. When you are using PyArrow, this data may come from IPC tools, though it +can also be created from various types of Python sequences (lists, NumPy +arrays, pandas data). + +A simple way to create arrays is with ``pyarrow.array``, which is similar to +the ``numpy.array`` function: + +.. ipython:: python + + arr = pa.array([1, 2, None, 3]) + arr + +The array's ``type`` attribute is the corresponding piece of type metadata: + +.. ipython:: python + + arr.type + +Each in-memory array has a known length and null count (which will be 0 if +there are no null values): + +.. ipython:: python + + len(arr) + arr.null_count + +Scalar values can be selected with normal indexing. ``pyarrow.array`` converts +``None`` values to Arrow nulls; we return the special ``pyarrow.NA`` value for +nulls: + +.. ipython:: python + + arr[0] + arr[2] + +Arrow data is immutable, so values can be selected but not assigned. + +Arrays can be sliced without copying: + +.. ipython:: python + + arr[3] + +``pyarrow.array`` can create simple nested data structures like lists: + +.. ipython:: python + + nested_arr = pa.array([[], None, [1, 2], [None, 1]]) + print(nested_arr.type) + +Dictionary Arrays +~~~~~~~~~~~~~~~~~ + +The **Dictionary** type in PyArrow is a special array type that is similar to a +factor in R or a ``pandas.Categorical``. It enables one or more record batches +in a file or stream to transmit integer *indices* referencing a shared +**dictionary** containing the distinct values in the logical array. This is +particularly often used with strings to save memory and improve performance. + +The way that dictionaries are handled in the Apache Arrow format and the way +they appear in C++ and Python is slightly different. We define a special +:class:`~.DictionaryArray` type with a corresponding dictionary type. Let's +consider an example: + +.. ipython:: python + + indices = pa.array([0, 1, 0, 1, 2, 0, None, 2]) + dictionary = pa.array(['foo', 'bar', 'baz']) + + dict_array = pa.DictionaryArray.from_arrays(indices, dictionary) + dict_array + +Here we have: + +.. ipython:: python + + print(dict_array.type) + dict_array.indices + dict_array.dictionary + +When using :class:`~.DictionaryArray` with pandas, the analogue is +``pandas.Categorical`` (more on this later): + +.. ipython:: python + + dict_array.to_pandas() + +.. _data.record_batch: + +Record Batches +-------------- + +A **Record Batch** in Apache Arrow is a collection of equal-length array +instances. Let's consider a collection of arrays: + +.. ipython:: python + + data = [ + pa.array([1, 2, 3, 4]), + pa.array(['foo', 'bar', 'baz', None]), + pa.array([True, None, False, True]) + ] + +A record batch can be created from this list of arrays using +``RecordBatch.from_arrays``: + +.. ipython:: python + + batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2']) + batch.num_columns + batch.num_rows + batch.schema + + batch[1] + +A record batch can be sliced without copying memory like an array: + +.. ipython:: python + + batch2 = batch.slice(1, 3) + batch2[1] + +.. _data.table: + +Tables +------ + +The PyArrow :class:`~.Table` type is not part of the Apache Arrow +specification, but is rather a tool to help with wrangling multiple record +batches and array pieces as a single logical dataset. As a relevant example, we +may receive multiple small record batches in a socket stream, then need to +concatenate them into contiguous memory for use in NumPy or pandas. The Table +object makes this efficient without requiring additional memory copying. + +Considering the record batch we created above, we can create a Table containing +one or more copies of the batch using ``Table.from_batches``: + +.. ipython:: python + + batches = [batch] * 5 + table = pa.Table.from_batches(batches) + table + table.num_rows + +The table's columns are instances of :class:`~.Column`, which is a container +for one or more arrays of the same type. + +.. ipython:: python + + c = table[0] + c + c.data + c.data.num_chunks + c.data.chunk(0) + +As you'll see in the :ref:`pandas section <pandas>`, we can convert thee +objects to contiguous NumPy arrays for use in pandas: + +.. ipython:: python + + c.to_pandas() + +Custom Schema and Field Metadata +-------------------------------- + +TODO http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/development.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/development.rst.txt b/docs/python/_sources/development.rst.txt new file mode 100644 index 0000000..440c1c4 --- /dev/null +++ b/docs/python/_sources/development.rst.txt @@ -0,0 +1,218 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _development: + +*********** +Development +*********** + +Developing with conda +===================== + +Linux and macOS +--------------- + +System Requirements +~~~~~~~~~~~~~~~~~~~ + +On macOS, any modern XCode (6.4 or higher; the current version is 8.3.1) is +sufficient. + +On Linux, for this guide, we recommend using gcc 4.8 or 4.9, or clang 3.7 or +higher. You can check your version by running + +.. code-block:: shell + + $ gcc --version + +On Ubuntu 16.04 and higher, you can obtain gcc 4.9 with: + +.. code-block:: shell + + $ sudo apt-get install g++-4.9 + +Finally, set gcc 4.9 as the active compiler using: + +.. code-block:: shell + + export CC=gcc-4.9 + export CXX=g++-4.9 + +Environment Setup and Build +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First, let's create a conda environment with all the C++ build and Python +dependencies from conda-forge: + +.. code-block:: shell + + conda create -y -q -n pyarrow-dev \ + python=3.6 numpy six setuptools cython pandas pytest \ + cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \ + brotli jemalloc -c conda-forge + source activate pyarrow-dev + +Now, let's clone the Arrow and Parquet git repositories: + +.. code-block:: shell + + mkdir repos + cd repos + git clone https://github.com/apache/arrow.git + git clone https://github.com/apache/parquet-cpp.git + +You should now see + + +.. code-block:: shell + + $ ls -l + total 8 + drwxrwxr-x 12 wesm wesm 4096 Apr 15 19:19 arrow/ + drwxrwxr-x 12 wesm wesm 4096 Apr 15 19:19 parquet-cpp/ + +We need to set some environment variables to let Arrow's build system know +about our build toolchain: + +.. code-block:: shell + + export ARROW_BUILD_TYPE=release + + export ARROW_BUILD_TOOLCHAIN=$CONDA_PREFIX + export PARQUET_BUILD_TOOLCHAIN=$CONDA_PREFIX + export ARROW_HOME=$CONDA_PREFIX + export PARQUET_HOME=$CONDA_PREFIX + +Now build and install the Arrow C++ libraries: + +.. code-block:: shell + + mkdir arrow/cpp/build + pushd arrow/cpp/build + + cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \ + -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ + -DARROW_PYTHON=on \ + -DARROW_BUILD_TESTS=OFF \ + .. + make -j4 + make install + popd + +Now, optionally build and install the Apache Parquet libraries in your +toolchain: + +.. code-block:: shell + + mkdir parquet-cpp/build + pushd parquet-cpp/build + + cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \ + -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \ + -DPARQUET_BUILD_BENCHMARKS=off \ + -DPARQUET_BUILD_EXECUTABLES=off \ + -DPARQUET_ZLIB_VENDORED=off \ + -DPARQUET_BUILD_TESTS=off \ + .. + + make -j4 + make install + popd + +Now, build pyarrow: + +.. code-block:: shell + + cd arrow/python + python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \ + --with-parquet --with-jemalloc --inplace + +If you did not build parquet-cpp, you can omit ``--with-parquet``. + +You should be able to run the unit tests with: + +.. code-block:: shell + + $ py.test pyarrow + ================================ test session starts ==================== + platform linux -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 + rootdir: /home/wesm/arrow-clone/python, inifile: + collected 198 items + + pyarrow/tests/test_array.py ........... + pyarrow/tests/test_convert_builtin.py ..................... + pyarrow/tests/test_convert_pandas.py ............................. + pyarrow/tests/test_feather.py .......................... + pyarrow/tests/test_hdfs.py sssssssssssssss + pyarrow/tests/test_io.py .................. + pyarrow/tests/test_ipc.py ........ + pyarrow/tests/test_jemalloc.py ss + pyarrow/tests/test_parquet.py .................... + pyarrow/tests/test_scalars.py .......... + pyarrow/tests/test_schema.py ......... + pyarrow/tests/test_table.py ............. + pyarrow/tests/test_tensor.py ................ + + ====================== 181 passed, 17 skipped in 0.98 seconds =========== + +Windows +======= + +First, make sure you can `build the C++ library <https://github.com/apache/arrow/blob/master/cpp/doc/Windows.md>`_. + +Now, we need to build and install the C++ libraries someplace. + +.. code-block:: shell + + mkdir cpp\build + cd cpp\build + set ARROW_HOME=C:\thirdparty + cmake -G "Visual Studio 14 2015 Win64" ^ + -DCMAKE_INSTALL_PREFIX=%ARROW_HOME% ^ + -DCMAKE_BUILD_TYPE=Release ^ + -DARROW_BUILD_TESTS=off ^ + -DARROW_PYTHON=on .. + cmake --build . --target INSTALL --config Release + cd ..\.. + +After that, we must put the install directory's bin path in our ``%PATH%``: + +.. code-block:: shell + + set PATH=%ARROW_HOME%\bin;%PATH% + +Now, we can build pyarrow: + +.. code-block:: shell + + cd python + python setup.py build_ext --inplace + +Running C++ unit tests with Python +---------------------------------- + +Getting ``python-test.exe`` to run is a bit tricky because your +``%PYTHONPATH%`` must be configured given the active conda environment: + +.. code-block:: shell + + set CONDA_ENV=C:\Users\wesm\Miniconda\envs\arrow-test + set PYTHONPATH=%CONDA_ENV%\Lib;%CONDA_ENV%\Lib\site-packages;%CONDA_ENV%\python35.zip;%CONDA_ENV%\DLLs;%CONDA_ENV% + +Now ``python-test.exe`` or simply ``ctest`` (to run all tests) should work. http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/filesystems.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/filesystems.rst.txt b/docs/python/_sources/filesystems.rst.txt index 9e00ddd..61c03c5 100644 --- a/docs/python/_sources/filesystems.rst.txt +++ b/docs/python/_sources/filesystems.rst.txt @@ -15,10 +15,12 @@ .. specific language governing permissions and limitations .. under the License. -File interfaces and Memory Maps -=============================== +Filesystem Interfaces +===================== -PyArrow features a number of file-like interfaces +In this section, we discuss filesystem-like interfaces in PyArrow. + +.. _hdfs: Hadoop File System (HDFS) ------------------------- http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/generated/pyarrow.Field.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.Field.rst.txt b/docs/python/_sources/generated/pyarrow.Field.rst.txt index d94089e..b029dfe 100644 --- a/docs/python/_sources/generated/pyarrow.Field.rst.txt +++ b/docs/python/_sources/generated/pyarrow.Field.rst.txt @@ -13,7 +13,9 @@ pyarrow\.Field .. autosummary:: - ~Field.from_py + ~Field.add_metadata + ~Field.equals + ~Field.remove_metadata @@ -23,6 +25,7 @@ pyarrow\.Field .. autosummary:: + ~Field.metadata ~Field.name ~Field.nullable ~Field.type http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/generated/pyarrow.Schema.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.Schema.rst.txt b/docs/python/_sources/generated/pyarrow.Schema.rst.txt index 397c01a..280d4c1 100644 --- a/docs/python/_sources/generated/pyarrow.Schema.rst.txt +++ b/docs/python/_sources/generated/pyarrow.Schema.rst.txt @@ -13,9 +13,10 @@ pyarrow\.Schema .. autosummary:: + ~Schema.add_metadata ~Schema.equals ~Schema.field_by_name - ~Schema.from_fields + ~Schema.remove_metadata @@ -25,6 +26,7 @@ pyarrow\.Schema .. autosummary:: + ~Schema.metadata ~Schema.names \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/index.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/index.rst.txt b/docs/python/_sources/index.rst.txt index ecb8e8f..4bfbe44 100644 --- a/docs/python/_sources/index.rst.txt +++ b/docs/python/_sources/index.rst.txt @@ -35,8 +35,12 @@ structures. :caption: Getting Started install - pandas + development + memory + data + ipc filesystems + pandas parquet api getting_involved http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/install.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/install.rst.txt b/docs/python/_sources/install.rst.txt index 278b466..a2a6520 100644 --- a/docs/python/_sources/install.rst.txt +++ b/docs/python/_sources/install.rst.txt @@ -37,115 +37,14 @@ Install the latest version from PyPI: pip install pyarrow .. note:: - Currently there are only binary artifcats available for Linux and MacOS. - Otherwise this will only pull the python sources and assumes an existing - installation of the C++ part of Arrow. - To retrieve the binary artifacts, you'll need a recent ``pip`` version that - supports features like the ``manylinux1`` tag. - -Building from source --------------------- - -First, clone the master git repository: - -.. code-block:: bash - - git clone https://github.com/apache/arrow.git arrow - -System requirements -~~~~~~~~~~~~~~~~~~~ - -Building pyarrow requires: - -* A C++11 compiler - - * Linux: gcc >= 4.8 or clang >= 3.5 - * OS X: XCode 6.4 or higher preferred - -* `CMake <https://cmake.org/>`_ - -Python requirements -~~~~~~~~~~~~~~~~~~~ - -You will need Python (CPython) 2.7, 3.4, or 3.5 installed. Earlier releases and -are not being targeted. - -.. note:: - This library targets CPython only due to an emphasis on interoperability with - pandas and NumPy, which are only available for CPython. - -The build requires NumPy, Cython, and a few other Python dependencies: - -.. code-block:: bash - - pip install cython - cd arrow/python - pip install -r requirements.txt - -Installing Arrow C++ library -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -First, you should choose an installation location for Arrow C++. In the future -using the default system install location will work, but for now we are being -explicit: - -.. code-block:: bash - - export ARROW_HOME=$HOME/local - -Now, we build Arrow: - -.. code-block:: bash - - cd arrow/cpp - - mkdir dev-build - cd dev-build - - cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME .. - - make - - # Use sudo here if $ARROW_HOME requires it - make install - -To get the optional Parquet support, you should also build and install -`parquet-cpp <https://github.com/apache/parquet-cpp/blob/master/README.md>`_. -Install `pyarrow` -~~~~~~~~~~~~~~~~~ - - -.. code-block:: bash - - cd arrow/python - - # --with-parquet enables the Apache Parquet support in PyArrow - # --with-jemalloc enables the jemalloc allocator support in PyArrow - # --build-type=release disables debugging information and turns on - # compiler optimizations for native code - python setup.py build_ext --with-parquet --with-jemalloc --build-type=release install - python setup.py install - -.. warning:: - On XCode 6 and prior there are some known OS X `@rpath` issues. If you are - unable to import pyarrow, upgrading XCode may be the solution. - -.. note:: - In development installations, you will also need to set a correct - ``LD_LIBRARY_PATH``. This is most probably done with - ``export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH``. - - -.. code-block:: python + Currently there are only binary artifacts available for Linux and MacOS. + Otherwise this will only pull the python sources and assumes an existing + installation of the C++ part of Arrow. To retrieve the binary artifacts, + you'll need a recent ``pip`` version that supports features like the + ``manylinux1`` tag. - In [1]: import pyarrow +Installing from source +---------------------- - In [2]: pyarrow.array([1,2,3]) - Out[2]: - <pyarrow.array.Int64Array object at 0x7f899f3e60e8> - [ - 1, - 2, - 3 - ] +See :ref:`development`. http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/ipc.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/ipc.rst.txt b/docs/python/_sources/ipc.rst.txt new file mode 100644 index 0000000..e63e745 --- /dev/null +++ b/docs/python/_sources/ipc.rst.txt @@ -0,0 +1,136 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow + +.. _ipc: + +IPC: Fast Streaming and Serialization +===================================== + +Arrow defines two types of binary formats for serializing record batches: + +* **Streaming format**: for sending an arbitrary length sequence of record + batches. The format must be processed from start to end, and does not support + random access + +* **File or Random Access format**: for serializing a fixed number of record + batches. Supports random access, and thus is very useful when used with + memory maps + +To follow this section, make sure to first read the section on :ref:`Memory and +IO <io>`. + +Writing and Reading Streams +--------------------------- + +First, let's create a small record batch: + +.. ipython:: python + + import pyarrow as pa + + data = [ + pa.array([1, 2, 3, 4]), + pa.array(['foo', 'bar', 'baz', None]), + pa.array([True, None, False, True]) + ] + + batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2']) + batch.num_rows + batch.num_columns + +Now, we can begin writing a stream containing some number of these batches. For +this we use :class:`~pyarrow.StreamWriter`, which can write to a writeable +``NativeFile`` object or a writeable Python object: + +.. ipython:: python + + sink = pa.InMemoryOutputStream() + writer = pa.StreamWriter(sink, batch.schema) + +Here we used an in-memory Arrow buffer stream, but this could have been a +socket or some other IO sink. + +When creating the ``StreamWriter``, we pass the schema, since the schema +(column names and types) must be the same for all of the batches sent in this +particular stream. Now we can do: + +.. ipython:: python + + for i in range(5): + writer.write_batch(batch) + writer.close() + + buf = sink.get_result() + buf.size + +Now ``buf`` contains the complete stream as an in-memory byte buffer. We can +read such a stream with :class:`~pyarrow.StreamReader`: + +.. ipython:: python + + reader = pa.StreamReader(buf) + reader.schema + + batches = [b for b in reader] + len(batches) + +We can check the returned batches are the same as the original input: + +.. ipython:: python + + batches[0].equals(batch) + +An important point is that if the input source supports zero-copy reads +(e.g. like a memory map, or ``pyarrow.BufferReader``), then the returned +batches are also zero-copy and do not allocate any new memory on read. + +Writing and Reading Random Access Files +--------------------------------------- + +The :class:`~pyarrow.FileWriter` has the same API as +:class:`~pyarrow.StreamWriter`: + +.. ipython:: python + + sink = pa.InMemoryOutputStream() + writer = pa.FileWriter(sink, batch.schema) + + for i in range(10): + writer.write_batch(batch) + writer.close() + + buf = sink.get_result() + buf.size + +The difference between :class:`~pyarrow.FileReader` and +:class:`~pyarrow.StreamReader` is that the input source must have a ``seek`` +method for random access. The stream reader only requires read operations: + +.. ipython:: python + + reader = pa.FileReader(buf) + +Because we have access to the entire payload, we know the number of record +batches in the file, and can read any at random: + +.. ipython:: python + + reader.num_record_batches + b = reader.get_batch(3) + b.equals(batch) http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/jemalloc.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/jemalloc.rst.txt b/docs/python/_sources/jemalloc.rst.txt index 8d7a5dc..9389dcb 100644 --- a/docs/python/_sources/jemalloc.rst.txt +++ b/docs/python/_sources/jemalloc.rst.txt @@ -18,7 +18,7 @@ jemalloc MemoryPool =================== -Arrow's default :class:`~pyarrow.memory.MemoryPool` uses the system's allocator +Arrow's default :class:`~pyarrow.MemoryPool` uses the system's allocator through the POSIX APIs. Although this already provides aligned allocation, the POSIX interface doesn't support aligned reallocation. The default reallocation strategy is to allocate a new region, copy over the old data and free the @@ -27,10 +27,9 @@ the existing memory allocation to the requested size. While this may still be linear in the size of allocated memory, it is magnitudes faster as only the page mapping in the kernel is touched, not the actual data. -The :mod:`~pyarrow.jemalloc` allocator is not enabled by default to allow the -use of the system allocator and/or other allocators like ``tcmalloc``. You can -either explicitly make it the default allocator or pass it only to single -operations. +The jemalloc-based allocator is not enabled by default to allow the use of the +system allocator and/or other allocators like ``tcmalloc``. You can either +explicitly make it the default allocator or pass it only to single operations. .. code:: python
