http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/data.html ---------------------------------------------------------------------- diff --git a/docs/python/data.html b/docs/python/data.html new file mode 100644 index 0000000..e16f145 --- /dev/null +++ b/docs/python/data.html @@ -0,0 +1,524 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> + + +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + + <title>In-Memory Data Model — pyarrow documentation</title> + + <link rel="stylesheet" href="_static/sphinxdoc.css" type="text/css" /> + <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: './', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' + }; + </script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> + <link rel="index" title="Index" href="genindex.html" /> + <link rel="search" title="Search" href="search.html" /> + <link rel="next" title="IPC: Fast Streaming and Serialization" href="ipc.html" /> + <link rel="prev" title="Memory and IO Interfaces" href="memory.html" /> + </head> + <body role="document"> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="genindex.html" title="General Index" + accesskey="I">index</a></li> + <li class="right" > + <a href="ipc.html" title="IPC: Fast Streaming and Serialization" + accesskey="N">next</a> |</li> + <li class="right" > + <a href="memory.html" title="Memory and IO Interfaces" + accesskey="P">previous</a> |</li> + <li class="nav-item nav-item-0"><a href="index.html">pyarrow documentation</a> »</li> + </ul> + </div> + <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> + <div class="sphinxsidebarwrapper"> + <h3><a href="index.html">Table Of Contents</a></h3> + <ul> +<li><a class="reference internal" href="#">In-Memory Data Model</a><ul> +<li><a class="reference internal" href="#type-metadata">Type Metadata</a></li> +<li><a class="reference internal" href="#schemas">Schemas</a></li> +<li><a class="reference internal" href="#arrays">Arrays</a><ul> +<li><a class="reference internal" href="#dictionary-arrays">Dictionary Arrays</a></li> +</ul> +</li> +<li><a class="reference internal" href="#record-batches">Record Batches</a></li> +<li><a class="reference internal" href="#tables">Tables</a></li> +<li><a class="reference internal" href="#custom-schema-and-field-metadata">Custom Schema and Field Metadata</a></li> +</ul> +</li> +</ul> + + <h4>Previous topic</h4> + <p class="topless"><a href="memory.html" + title="previous chapter">Memory and IO Interfaces</a></p> + <h4>Next topic</h4> + <p class="topless"><a href="ipc.html" + title="next chapter">IPC: Fast Streaming and Serialization</a></p> + <div role="note" aria-label="source link"> + <h3>This Page</h3> + <ul class="this-page-menu"> + <li><a href="_sources/data.rst.txt" + rel="nofollow">Show Source</a></li> + </ul> + </div> +<div id="searchbox" style="display: none" role="search"> + <h3>Quick search</h3> + <form class="search" action="search.html" method="get"> + <div><input type="text" name="q" /></div> + <div><input type="submit" value="Go" /></div> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> +<script type="text/javascript">$('#searchbox').show(0);</script> + </div> + </div> + + <div class="document"> + <div class="documentwrapper"> + <div class="bodywrapper"> + <div class="body" role="main"> + + <div class="section" id="in-memory-data-model"> +<span id="data"></span><h1>In-Memory Data Model<a class="headerlink" href="#in-memory-data-model" title="Permalink to this headline">¶</a></h1> +<p>Apache Arrow defines columnar array data structures by composing type metadata +with memory buffers, like the ones explained in the documentation on +<a class="reference internal" href="memory.html#io"><span class="std std-ref">Memory and IO</span></a>. These data structures are exposed in Python through +a series of interrelated classes:</p> +<ul class="simple"> +<li><strong>Type Metadata</strong>: Instances of <code class="docutils literal"><span class="pre">pyarrow.DataType</span></code>, which describe a logical +array type</li> +<li><strong>Schemas</strong>: Instances of <code class="docutils literal"><span class="pre">pyarrow.Schema</span></code>, which describe a named +collection of types. These can be thought of as the column types in a +table-like object.</li> +<li><strong>Arrays</strong>: Instances of <code class="docutils literal"><span class="pre">pyarrow.Array</span></code>, which are atomic, contiguous +columnar data structures composed from Arrow Buffer objects</li> +<li><strong>Record Batches</strong>: Instances of <code class="docutils literal"><span class="pre">pyarrow.RecordBatch</span></code>, which are a +collection of Array objects with a particular Schema</li> +<li><strong>Tables</strong>: Instances of <code class="docutils literal"><span class="pre">pyarrow.Table</span></code>, a logical table data structure in +which each column consists of one or more <code class="docutils literal"><span class="pre">pyarrow.Array</span></code> objects of the +same type.</li> +</ul> +<p>We will examine these in the sections below in a series of examples.</p> +<div class="section" id="type-metadata"> +<span id="data-types"></span><h2>Type Metadata<a class="headerlink" href="#type-metadata" title="Permalink to this headline">¶</a></h2> +<p>Apache Arrow defines language agnostic column-oriented data structures for +array data. These include:</p> +<ul class="simple"> +<li><strong>Fixed-length primitive types</strong>: numbers, booleans, date and times, fixed +size binary, decimals, and other values that fit into a given number</li> +<li><strong>Variable-length primitive types</strong>: binary, string</li> +<li><strong>Nested types</strong>: list, struct, and union</li> +<li><strong>Dictionary type</strong>: An encoded categorical type (more on this later)</li> +</ul> +<p>Each logical data type in Arrow has a corresponding factory function for +creating an instance of that type object in Python:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [1]: </span><span class="kn">import</span> <span class="nn">pyarrow</span> <span class="kn">as</span> <span class="nn">pa</span> + +<span class="gp">In [2]: </span><span class="n">t1</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">int32</span><span class="p">()</span> + +<span class="gp">In [3]: </span><span class="n">t2</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">string</span><span class="p">()</span> + +<span class="gp">In [4]: </span><span class="n">t3</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">binary</span><span class="p">()</span> + +<span class="gp">In [5]: </span><span class="n">t4</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">binary</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> + +<span class="gp">In [6]: </span><span class="n">t5</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">timestamp</span><span class="p">(</span><span class="s1">'ms'</span><span class="p">)</span> + +<span class="gp">In [7]: </span><span class="n">t1</span> +<span class="gh">Out[7]: </span><span class="go">DataType(int32)</span> + +<span class="gp">In [8]: </span><span class="k">print</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> +<span class="go"> int32</span> + +<span class="gp">In [9]: </span><span class="k">print</span><span class="p">(</span><span class="n">t4</span><span class="p">)</span> +<span class="go"> fixed_size_binary[10]</span> + +<span class="gp">In [10]: </span><span class="k">print</span><span class="p">(</span><span class="n">t5</span><span class="p">)</span> +<span class="go"> timestamp[ms]</span> +</pre></div> +</div> +<p>We use the name <strong>logical type</strong> because the <strong>physical</strong> storage may be the +same for one or more types. For example, <code class="docutils literal"><span class="pre">int64</span></code>, <code class="docutils literal"><span class="pre">float64</span></code>, and +<code class="docutils literal"><span class="pre">timestamp[ms]</span></code> all occupy 64 bits per value.</p> +<p>These objects are <cite>metadata</cite>; they are used for describing the data in arrays, +schemas, and record batches. In Python, they can be used in functions where the +input data (e.g. Python objects) may be coerced to more than one Arrow type.</p> +<p>The <a class="reference internal" href="generated/pyarrow.Field.html#pyarrow.Field" title="pyarrow.Field"><code class="xref py py-class docutils literal"><span class="pre">Field</span></code></a> type is a type plus a name and optional +user-defined metadata:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [11]: </span><span class="n">f0</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s1">'int32_field'</span><span class="p">,</span> <span class="n">t1</span><span class="p">)</span> + +<span class="gp">In [12]: </span><span class="n">f0</span> +<span class="gh">Out[12]: </span><span class="go">pyarrow.Field<int32_field: int32></span> + +<span class="gp">In [13]: </span><span class="n">f0</span><span class="o">.</span><span class="n">name</span> +<span class="go"> Out[13]: 'int32_field'</span> + +<span class="gp">In [14]: </span><span class="n">f0</span><span class="o">.</span><span class="n">type</span> +<span class="go"> Out[14]: DataType(int32)</span> +</pre></div> +</div> +<p>Arrow supports <strong>nested value types</strong> like list, struct, and union. When +creating these, you must pass types or fields to indicate the data types of the +types’ children. For example, we can define a list of int32 values with:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [15]: </span><span class="n">t6</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">list_</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> + +<span class="gp">In [16]: </span><span class="n">t6</span> +<span class="gh">Out[16]: </span><span class="go">DataType(list<item: int32>)</span> +</pre></div> +</div> +<p>A <cite>struct</cite> is a collection of named fields:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [17]: </span><span class="n">fields</span> <span class="o">=</span> <span class="p">[</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s1">'s0'</span><span class="p">,</span> <span class="n">t1</span><span class="p">),</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s1">'s1'</span><span class="p">,</span> <span class="n">t2</span><span class="p">),</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s1">'s2'</span><span class="p">,</span> <span class="n">t4</span><span class="p">),</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">,</span> <span class="n">t6</span><span class="p">)</span> +<span class="gp"> ....: </span><span class="p">]</span> +<span class="gp"> ....: </span> + +<span class="gp">In [18]: </span><span class="n">t7</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">struct</span><span class="p">(</span><span class="n">fields</span><span class="p">)</span> + +<span class="gp">In [19]: </span><span class="k">print</span><span class="p">(</span><span class="n">t7</span><span class="p">)</span> +<span class="go">struct<s0: int32, s1: string, s2: fixed_size_binary[10], s3: list<item: int32>></span> +</pre></div> +</div> +<p>See <a class="reference internal" href="api.html#api-types"><span class="std std-ref">Data Types API</span></a> for a full listing of data type +functions.</p> +</div> +<div class="section" id="schemas"> +<span id="data-schema"></span><h2>Schemas<a class="headerlink" href="#schemas" title="Permalink to this headline">¶</a></h2> +<p>The <a class="reference internal" href="generated/pyarrow.Schema.html#pyarrow.Schema" title="pyarrow.Schema"><code class="xref py py-class docutils literal"><span class="pre">Schema</span></code></a> type is similar to the <code class="docutils literal"><span class="pre">struct</span></code> array type; it +defines the column names and types in a record batch or table data +structure. The <code class="docutils literal"><span class="pre">pyarrow.schema</span></code> factory function makes new Schema objects in +Python:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [20]: </span><span class="n">fields</span> <span class="o">=</span> <span class="p">[</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s1">'s0'</span><span class="p">,</span> <span class="n">t1</span><span class="p">),</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s1">'s1'</span><span class="p">,</span> <span class="n">t2</span><span class="p">),</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s1">'s2'</span><span class="p">,</span> <span class="n">t4</span><span class="p">),</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">,</span> <span class="n">t6</span><span class="p">)</span> +<span class="gp"> ....: </span><span class="p">]</span> +<span class="gp"> ....: </span> + +<span class="gp">In [21]: </span><span class="n">my_schema</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">schema</span><span class="p">(</span><span class="n">fields</span><span class="p">)</span> + +<span class="gp">In [22]: </span><span class="n">my_schema</span> +<span class="gh">Out[22]: </span><span class="go"></span> +<span class="go">s0: int32</span> +<span class="go">s1: string</span> +<span class="go">s2: fixed_size_binary[10]</span> +<span class="go">s3: list<item: int32></span> +</pre></div> +</div> +<p>In some applications, you may not create schemas directly, only using the ones +that are embedded in <a class="reference internal" href="ipc.html#ipc"><span class="std std-ref">IPC messages</span></a>.</p> +</div> +<div class="section" id="arrays"> +<span id="data-array"></span><h2>Arrays<a class="headerlink" href="#arrays" title="Permalink to this headline">¶</a></h2> +<p>For each data type, there is an accompanying array data structure for holding +memory buffers that define a single contiguous chunk of columnar array +data. When you are using PyArrow, this data may come from IPC tools, though it +can also be created from various types of Python sequences (lists, NumPy +arrays, pandas data).</p> +<p>A simple way to create arrays is with <code class="docutils literal"><span class="pre">pyarrow.array</span></code>, which is similar to +the <code class="docutils literal"><span class="pre">numpy.array</span></code> function:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [23]: </span><span class="n">arr</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span> + +<span class="gp">In [24]: </span><span class="n">arr</span> +<span class="gh">Out[24]: </span><span class="go"></span> +<span class="go"><pyarrow._array.Int64Array object at 0x2b0bcdd11318></span> +<span class="go">[</span> +<span class="go"> 1,</span> +<span class="go"> 2,</span> +<span class="go"> NA,</span> +<span class="go"> 3</span> +<span class="go">]</span> +</pre></div> +</div> +<p>The array’s <code class="docutils literal"><span class="pre">type</span></code> attribute is the corresponding piece of type metadata:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [25]: </span><span class="n">arr</span><span class="o">.</span><span class="n">type</span> +<span class="gh">Out[25]: </span><span class="go">DataType(int64)</span> +</pre></div> +</div> +<p>Each in-memory array has a known length and null count (which will be 0 if +there are no null values):</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [26]: </span><span class="nb">len</span><span class="p">(</span><span class="n">arr</span><span class="p">)</span> +<span class="gh">Out[26]: </span><span class="go">4</span> + +<span class="gp">In [27]: </span><span class="n">arr</span><span class="o">.</span><span class="n">null_count</span> +<span class="go"> Out[27]: 1</span> +</pre></div> +</div> +<p>Scalar values can be selected with normal indexing. <code class="docutils literal"><span class="pre">pyarrow.array</span></code> converts +<code class="docutils literal"><span class="pre">None</span></code> values to Arrow nulls; we return the special <code class="docutils literal"><span class="pre">pyarrow.NA</span></code> value for +nulls:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [28]: </span><span class="n">arr</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> +<span class="gh">Out[28]: </span><span class="go">1</span> + +<span class="gp">In [29]: </span><span class="n">arr</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> +<span class="go"> Out[29]: NA</span> +</pre></div> +</div> +<p>Arrow data is immutable, so values can be selected but not assigned.</p> +<p>Arrays can be sliced without copying:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [30]: </span><span class="n">arr</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span> +<span class="gh">Out[30]: </span><span class="go">3</span> +</pre></div> +</div> +<p><code class="docutils literal"><span class="pre">pyarrow.array</span></code> can create simple nested data structures like lists:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [31]: </span><span class="n">nested_arr</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">array</span><span class="p">([[],</span> <span class="bp">None</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span> <span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="mi">1</span><span class="p">]])</span> + +<span class="gp">In [32]: </span><span class="k">print</span><span class="p">(</span><span class="n">nested_arr</span><span class="o">.</span><span class="n">type</span><span class="p">)</span> +<span class="go">list<item: int64></span> +</pre></div> +</div> +<div class="section" id="dictionary-arrays"> +<h3>Dictionary Arrays<a class="headerlink" href="#dictionary-arrays" title="Permalink to this headline">¶</a></h3> +<p>The <strong>Dictionary</strong> type in PyArrow is a special array type that is similar to a +factor in R or a <code class="docutils literal"><span class="pre">pandas.Categorical</span></code>. It enables one or more record batches +in a file or stream to transmit integer <em>indices</em> referencing a shared +<strong>dictionary</strong> containing the distinct values in the logical array. This is +particularly often used with strings to save memory and improve performance.</p> +<p>The way that dictionaries are handled in the Apache Arrow format and the way +they appear in C++ and Python is slightly different. We define a special +<a class="reference internal" href="generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray" title="pyarrow.DictionaryArray"><code class="xref py py-class docutils literal"><span class="pre">DictionaryArray</span></code></a> type with a corresponding dictionary type. Let’s +consider an example:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [33]: </span><span class="n">indices</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="mi">2</span><span class="p">])</span> + +<span class="gp">In [34]: </span><span class="n">dictionary</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="s1">'foo'</span><span class="p">,</span> <span class="s1">'bar'</span><span class="p">,</span> <span class="s1">'baz'</span><span class="p">])</span> + +<span class="gp">In [35]: </span><span class="n">dict_array</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">DictionaryArray</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span><span class="n">indices</span><span class="p">,</span> <span class="n">dictionary</span><span class="p">)</span> + +<span class="gp">In [36]: </span><span class="n">dict_array</span> +<span class="gh">Out[36]: </span><span class="go"></span> +<span class="go"><pyarrow._array.DictionaryArray object at 0x2b0bcdd19588></span> +<span class="go">[</span> +<span class="go"> 'foo',</span> +<span class="go"> 'bar',</span> +<span class="go"> 'foo',</span> +<span class="go"> 'bar',</span> +<span class="go"> 'baz',</span> +<span class="go"> 'foo',</span> +<span class="go"> NA,</span> +<span class="go"> 'baz'</span> +<span class="go">]</span> +</pre></div> +</div> +<p>Here we have:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [37]: </span><span class="k">print</span><span class="p">(</span><span class="n">dict_array</span><span class="o">.</span><span class="n">type</span><span class="p">)</span> +<span class="go">dictionary<values=string, indices=int64></span> + +<span class="gp">In [38]: </span><span class="n">dict_array</span><span class="o">.</span><span class="n">indices</span> +<span class="go"> Out[38]: </span> +<span class="go"><pyarrow._array.Int64Array object at 0x2b0bcdd29908></span> +<span class="go">[</span> +<span class="go"> 0,</span> +<span class="go"> 1,</span> +<span class="go"> 0,</span> +<span class="go"> 1,</span> +<span class="go"> 2,</span> +<span class="go"> 0,</span> +<span class="go"> NA,</span> +<span class="go"> 2</span> +<span class="go">]</span> + +<span class="gp">In [39]: </span><span class="n">dict_array</span><span class="o">.</span><span class="n">dictionary</span> +<span class="go"> Out[39]: </span> +<span class="go"><pyarrow._array.StringArray object at 0x2b0bcdd29818></span> +<span class="go">[</span> +<span class="go"> 'foo',</span> +<span class="go"> 'bar',</span> +<span class="go"> 'baz'</span> +<span class="go">]</span> +</pre></div> +</div> +<p>When using <a class="reference internal" href="generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray" title="pyarrow.DictionaryArray"><code class="xref py py-class docutils literal"><span class="pre">DictionaryArray</span></code></a> with pandas, the analogue is +<code class="docutils literal"><span class="pre">pandas.Categorical</span></code> (more on this later):</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [40]: </span><span class="n">dict_array</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span> +<span class="gh">Out[40]: </span><span class="go"></span> +<span class="go">[foo, bar, foo, bar, baz, foo, NaN, baz]</span> +<span class="go">Categories (3, object): [foo, bar, baz]</span> +</pre></div> +</div> +</div> +</div> +<div class="section" id="record-batches"> +<span id="data-record-batch"></span><h2>Record Batches<a class="headerlink" href="#record-batches" title="Permalink to this headline">¶</a></h2> +<p>A <strong>Record Batch</strong> in Apache Arrow is a collection of equal-length array +instances. Let’s consider a collection of arrays:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [41]: </span><span class="n">data</span> <span class="o">=</span> <span class="p">[</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">]),</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="s1">'foo'</span><span class="p">,</span> <span class="s1">'bar'</span><span class="p">,</span> <span class="s1">'baz'</span><span class="p">,</span> <span class="bp">None</span><span class="p">]),</span> +<span class="gp"> ....: </span> <span class="n">pa</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="bp">True</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="bp">False</span><span class="p">,</span> <span class="bp">True</span><span class="p">])</span> +<span class="gp"> ....: </span><span class="p">]</span> +<span class="gp"> ....: </span> +</pre></div> +</div> +<p>A record batch can be created from this list of arrays using +<code class="docutils literal"><span class="pre">RecordBatch.from_arrays</span></code>:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [42]: </span><span class="n">batch</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">RecordBatch</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="p">[</span><span class="s1">'f0'</span><span class="p">,</span> <span class="s1">'f1'</span><span class="p">,</span> <span class="s1">'f2'</span><span class="p">])</span> + +<span class="gp">In [43]: </span><span class="n">batch</span><span class="o">.</span><span class="n">num_columns</span> +<span class="gh">Out[43]: </span><span class="go">3</span> + +<span class="gp">In [44]: </span><span class="n">batch</span><span class="o">.</span><span class="n">num_rows</span> +<span class="go"> Out[44]: 4</span> + +<span class="gp">In [45]: </span><span class="n">batch</span><span class="o">.</span><span class="n">schema</span> +<span class="go"> Out[45]: </span> +<span class="go">f0: int64</span> +<span class="go">f1: string</span> +<span class="go">f2: bool</span> +<span class="go">-- metadata --</span> + +<span class="gp">In [46]: </span><span class="n">batch</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> +<span class="go"> Out[46]: </span> +<span class="go"><pyarrow._array.StringArray object at 0x2b0bcdd36f98></span> +<span class="go">[</span> +<span class="go"> 'foo',</span> +<span class="go"> 'bar',</span> +<span class="go"> 'baz',</span> +<span class="go"> NA</span> +<span class="go">]</span> +</pre></div> +</div> +<p>A record batch can be sliced without copying memory like an array:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [47]: </span><span class="n">batch2</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">slice</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> + +<span class="gp">In [48]: </span><span class="n">batch2</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> +<span class="gh">Out[48]: </span><span class="go"></span> +<span class="go"><pyarrow._array.StringArray object at 0x2b0bcdd36ea8></span> +<span class="go">[</span> +<span class="go"> 'bar',</span> +<span class="go"> 'baz',</span> +<span class="go"> NA</span> +<span class="go">]</span> +</pre></div> +</div> +</div> +<div class="section" id="tables"> +<span id="data-table"></span><h2>Tables<a class="headerlink" href="#tables" title="Permalink to this headline">¶</a></h2> +<p>The PyArrow <a class="reference internal" href="generated/pyarrow.Table.html#pyarrow.Table" title="pyarrow.Table"><code class="xref py py-class docutils literal"><span class="pre">Table</span></code></a> type is not part of the Apache Arrow +specification, but is rather a tool to help with wrangling multiple record +batches and array pieces as a single logical dataset. As a relevant example, we +may receive multiple small record batches in a socket stream, then need to +concatenate them into contiguous memory for use in NumPy or pandas. The Table +object makes this efficient without requiring additional memory copying.</p> +<p>Considering the record batch we created above, we can create a Table containing +one or more copies of the batch using <code class="docutils literal"><span class="pre">Table.from_batches</span></code>:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [49]: </span><span class="n">batches</span> <span class="o">=</span> <span class="p">[</span><span class="n">batch</span><span class="p">]</span> <span class="o">*</span> <span class="mi">5</span> + +<span class="gp">In [50]: </span><span class="n">table</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">Table</span><span class="o">.</span><span class="n">from_batches</span><span class="p">(</span><span class="n">batches</span><span class="p">)</span> + +<span class="gp">In [51]: </span><span class="n">table</span> +<span class="gh">Out[51]: </span><span class="go"></span> +<span class="go">pyarrow.Table</span> +<span class="go">f0: int64</span> +<span class="go">f1: string</span> +<span class="go">f2: bool</span> +<span class="go">-- metadata --</span> + +<span class="gp">In [52]: </span><span class="n">table</span><span class="o">.</span><span class="n">num_rows</span> +<span class="go"> Out[52]: 20</span> +</pre></div> +</div> +<p>The table’s columns are instances of <a class="reference internal" href="generated/pyarrow.Column.html#pyarrow.Column" title="pyarrow.Column"><code class="xref py py-class docutils literal"><span class="pre">Column</span></code></a>, which is a container +for one or more arrays of the same type.</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [53]: </span><span class="n">c</span> <span class="o">=</span> <span class="n">table</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> + +<span class="gp">In [54]: </span><span class="n">c</span> +<span class="gh">Out[54]: </span><span class="go"><pyarrow._table.Column at 0x2b0bcda54f00></span> + +<span class="gp">In [55]: </span><span class="n">c</span><span class="o">.</span><span class="n">data</span> +<span class="go"> Out[55]: <pyarrow._table.ChunkedArray at 0x2b0bcda54ed0></span> + +<span class="gp">In [56]: </span><span class="n">c</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">num_chunks</span> +<span class="go"> Out[56]: 5</span> + +<span class="gp">In [57]: </span><span class="n">c</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">chunk</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> +<span class="go"> Out[57]: </span> +<span class="go"><pyarrow._array.Int64Array object at 0x2b0bcdd4b3b8></span> +<span class="go">[</span> +<span class="go"> 1,</span> +<span class="go"> 2,</span> +<span class="go"> 3,</span> +<span class="go"> 4</span> +<span class="go">]</span> +</pre></div> +</div> +<p>As you’ll see in the <span class="xref std std-ref">pandas section</span>, we can convert thee +objects to contiguous NumPy arrays for use in pandas:</p> +<div class="highlight-ipython"><div class="highlight"><pre><span></span><span class="gp">In [58]: </span><span class="n">c</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span> +<span class="gh">Out[58]: </span><span class="go"></span> +<span class="go">0 1</span> +<span class="go">1 2</span> +<span class="go">2 3</span> +<span class="go">3 4</span> +<span class="go">4 1</span> +<span class="go">5 2</span> +<span class="go">6 3</span> +<span class="go">7 4</span> +<span class="go">8 1</span> +<span class="go">9 2</span> +<span class="go">10 3</span> +<span class="go">11 4</span> +<span class="go">12 1</span> +<span class="go">13 2</span> +<span class="go">14 3</span> +<span class="go">15 4</span> +<span class="go">16 1</span> +<span class="go">17 2</span> +<span class="go">18 3</span> +<span class="go">19 4</span> +<span class="go">Name: f0, dtype: int64</span> +</pre></div> +</div> +</div> +<div class="section" id="custom-schema-and-field-metadata"> +<h2>Custom Schema and Field Metadata<a class="headerlink" href="#custom-schema-and-field-metadata" title="Permalink to this headline">¶</a></h2> +<p>TODO</p> +</div> +</div> + + + </div> + </div> + </div> + <div class="clearer"></div> + </div> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="genindex.html" title="General Index" + >index</a></li> + <li class="right" > + <a href="ipc.html" title="IPC: Fast Streaming and Serialization" + >next</a> |</li> + <li class="right" > + <a href="memory.html" title="Memory and IO Interfaces" + >previous</a> |</li> + <li class="nav-item nav-item-0"><a href="index.html">pyarrow documentation</a> »</li> + </ul> + </div> + <div class="footer" role="contentinfo"> + © Copyright 2016-2017 Apache Software Foundation. + Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 1.5.5. + </div> + </body> +</html> \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/development.html ---------------------------------------------------------------------- diff --git a/docs/python/development.html b/docs/python/development.html new file mode 100644 index 0000000..bfc4110 --- /dev/null +++ b/docs/python/development.html @@ -0,0 +1,291 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> + + +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + + <title>Development — pyarrow documentation</title> + + <link rel="stylesheet" href="_static/sphinxdoc.css" type="text/css" /> + <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: './', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' + }; + </script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> + <link rel="index" title="Index" href="genindex.html" /> + <link rel="search" title="Search" href="search.html" /> + <link rel="next" title="Memory and IO Interfaces" href="memory.html" /> + <link rel="prev" title="Install PyArrow" href="install.html" /> + </head> + <body role="document"> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="genindex.html" title="General Index" + accesskey="I">index</a></li> + <li class="right" > + <a href="memory.html" title="Memory and IO Interfaces" + accesskey="N">next</a> |</li> + <li class="right" > + <a href="install.html" title="Install PyArrow" + accesskey="P">previous</a> |</li> + <li class="nav-item nav-item-0"><a href="index.html">pyarrow documentation</a> »</li> + </ul> + </div> + <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> + <div class="sphinxsidebarwrapper"> + <h3><a href="index.html">Table Of Contents</a></h3> + <ul> +<li><a class="reference internal" href="#">Development</a><ul> +<li><a class="reference internal" href="#developing-with-conda">Developing with conda</a><ul> +<li><a class="reference internal" href="#linux-and-macos">Linux and macOS</a><ul> +<li><a class="reference internal" href="#system-requirements">System Requirements</a></li> +<li><a class="reference internal" href="#environment-setup-and-build">Environment Setup and Build</a></li> +</ul> +</li> +</ul> +</li> +<li><a class="reference internal" href="#windows">Windows</a><ul> +<li><a class="reference internal" href="#running-c-unit-tests-with-python">Running C++ unit tests with Python</a></li> +</ul> +</li> +</ul> +</li> +</ul> + + <h4>Previous topic</h4> + <p class="topless"><a href="install.html" + title="previous chapter">Install PyArrow</a></p> + <h4>Next topic</h4> + <p class="topless"><a href="memory.html" + title="next chapter">Memory and IO Interfaces</a></p> + <div role="note" aria-label="source link"> + <h3>This Page</h3> + <ul class="this-page-menu"> + <li><a href="_sources/development.rst.txt" + rel="nofollow">Show Source</a></li> + </ul> + </div> +<div id="searchbox" style="display: none" role="search"> + <h3>Quick search</h3> + <form class="search" action="search.html" method="get"> + <div><input type="text" name="q" /></div> + <div><input type="submit" value="Go" /></div> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> +<script type="text/javascript">$('#searchbox').show(0);</script> + </div> + </div> + + <div class="document"> + <div class="documentwrapper"> + <div class="bodywrapper"> + <div class="body" role="main"> + + <div class="section" id="development"> +<span id="id1"></span><h1>Development<a class="headerlink" href="#development" title="Permalink to this headline">¶</a></h1> +<div class="section" id="developing-with-conda"> +<h2>Developing with conda<a class="headerlink" href="#developing-with-conda" title="Permalink to this headline">¶</a></h2> +<div class="section" id="linux-and-macos"> +<h3>Linux and macOS<a class="headerlink" href="#linux-and-macos" title="Permalink to this headline">¶</a></h3> +<div class="section" id="system-requirements"> +<h4>System Requirements<a class="headerlink" href="#system-requirements" title="Permalink to this headline">¶</a></h4> +<p>On macOS, any modern XCode (6.4 or higher; the current version is 8.3.1) is +sufficient.</p> +<p>On Linux, for this guide, we recommend using gcc 4.8 or 4.9, or clang 3.7 or +higher. You can check your version by running</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>$ gcc --version +</pre></div> +</div> +<p>On Ubuntu 16.04 and higher, you can obtain gcc 4.9 with:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>$ sudo apt-get install g++-4.9 +</pre></div> +</div> +<p>Finally, set gcc 4.9 as the active compiler using:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span><span class="nb">export</span> <span class="nv">CC</span><span class="o">=</span>gcc-4.9 +<span class="nb">export</span> <span class="nv">CXX</span><span class="o">=</span>g++-4.9 +</pre></div> +</div> +</div> +<div class="section" id="environment-setup-and-build"> +<h4>Environment Setup and Build<a class="headerlink" href="#environment-setup-and-build" title="Permalink to this headline">¶</a></h4> +<p>First, let’s create a conda environment with all the C++ build and Python +dependencies from conda-forge:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>conda create -y -q -n pyarrow-dev <span class="se">\</span> + <span class="nv">python</span><span class="o">=</span><span class="m">3</span>.6 numpy six setuptools cython pandas pytest <span class="se">\</span> + cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib <span class="se">\</span> + brotli jemalloc -c conda-forge +<span class="nb">source</span> activate pyarrow-dev +</pre></div> +</div> +<p>Now, let’s clone the Arrow and Parquet git repositories:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>mkdir repos +<span class="nb">cd</span> repos +git clone https://github.com/apache/arrow.git +git clone https://github.com/apache/parquet-cpp.git +</pre></div> +</div> +<p>You should now see</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>$ ls -l +total <span class="m">8</span> +drwxrwxr-x <span class="m">12</span> wesm wesm <span class="m">4096</span> Apr <span class="m">15</span> <span class="m">19</span>:19 arrow/ +drwxrwxr-x <span class="m">12</span> wesm wesm <span class="m">4096</span> Apr <span class="m">15</span> <span class="m">19</span>:19 parquet-cpp/ +</pre></div> +</div> +<p>We need to set some environment variables to let Arrow’s build system know +about our build toolchain:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span><span class="nb">export</span> <span class="nv">ARROW_BUILD_TYPE</span><span class="o">=</span>release + +<span class="nb">export</span> <span class="nv">ARROW_BUILD_TOOLCHAIN</span><span class="o">=</span><span class="nv">$CONDA_PREFIX</span> +<span class="nb">export</span> <span class="nv">PARQUET_BUILD_TOOLCHAIN</span><span class="o">=</span><span class="nv">$CONDA_PREFIX</span> +<span class="nb">export</span> <span class="nv">ARROW_HOME</span><span class="o">=</span><span class="nv">$CONDA_PREFIX</span> +<span class="nb">export</span> <span class="nv">PARQUET_HOME</span><span class="o">=</span><span class="nv">$CONDA_PREFIX</span> +</pre></div> +</div> +<p>Now build and install the Arrow C++ libraries:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>mkdir arrow/cpp/build +<span class="nb">pushd</span> arrow/cpp/build + +cmake -DCMAKE_BUILD_TYPE<span class="o">=</span><span class="nv">$ARROW_BUILD_TYPE</span> <span class="se">\</span> + -DCMAKE_INSTALL_PREFIX<span class="o">=</span><span class="nv">$ARROW_HOME</span> <span class="se">\</span> + -DARROW_PYTHON<span class="o">=</span>on <span class="se">\</span> + -DARROW_BUILD_TESTS<span class="o">=</span>OFF <span class="se">\</span> + .. +make -j4 +make install +<span class="nb">popd</span> +</pre></div> +</div> +<p>Now, optionally build and install the Apache Parquet libraries in your +toolchain:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>mkdir parquet-cpp/build +<span class="nb">pushd</span> parquet-cpp/build + +cmake -DCMAKE_BUILD_TYPE<span class="o">=</span><span class="nv">$ARROW_BUILD_TYPE</span> <span class="se">\</span> + -DCMAKE_INSTALL_PREFIX<span class="o">=</span><span class="nv">$PARQUET_HOME</span> <span class="se">\</span> + -DPARQUET_BUILD_BENCHMARKS<span class="o">=</span>off <span class="se">\</span> + -DPARQUET_BUILD_EXECUTABLES<span class="o">=</span>off <span class="se">\</span> + -DPARQUET_ZLIB_VENDORED<span class="o">=</span>off <span class="se">\</span> + -DPARQUET_BUILD_TESTS<span class="o">=</span>off <span class="se">\</span> + .. + +make -j4 +make install +<span class="nb">popd</span> +</pre></div> +</div> +<p>Now, build pyarrow:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span><span class="nb">cd</span> arrow/python +python setup.py build_ext --build-type<span class="o">=</span><span class="nv">$ARROW_BUILD_TYPE</span> <span class="se">\</span> + --with-parquet --with-jemalloc --inplace +</pre></div> +</div> +<p>If you did not build parquet-cpp, you can omit <code class="docutils literal"><span class="pre">--with-parquet</span></code>.</p> +<p>You should be able to run the unit tests with:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>$ py.test <span class="nv">pyarrow</span> +<span class="o">================================</span> <span class="nb">test</span> session <span class="nv">starts</span> <span class="o">====================</span> +platform linux -- Python <span class="m">3</span>.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 +rootdir: /home/wesm/arrow-clone/python, inifile: +collected <span class="m">198</span> items + +pyarrow/tests/test_array.py ........... +pyarrow/tests/test_convert_builtin.py ..................... +pyarrow/tests/test_convert_pandas.py ............................. +pyarrow/tests/test_feather.py .......................... +pyarrow/tests/test_hdfs.py sssssssssssssss +pyarrow/tests/test_io.py .................. +pyarrow/tests/test_ipc.py ........ +pyarrow/tests/test_jemalloc.py ss +pyarrow/tests/test_parquet.py .................... +pyarrow/tests/test_scalars.py .......... +pyarrow/tests/test_schema.py ......... +pyarrow/tests/test_table.py ............. +pyarrow/tests/test_tensor.py ................ + +<span class="o">======================</span> <span class="m">181</span> passed, <span class="m">17</span> skipped in <span class="m">0</span>.98 <span class="nv">seconds</span> <span class="o">===========</span> +</pre></div> +</div> +</div> +</div> +</div> +<div class="section" id="windows"> +<h2>Windows<a class="headerlink" href="#windows" title="Permalink to this headline">¶</a></h2> +<p>First, make sure you can <a class="reference external" href="https://github.com/apache/arrow/blob/master/cpp/doc/Windows.md";>build the C++ library</a>.</p> +<p>Now, we need to build and install the C++ libraries someplace.</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>mkdir cpp<span class="se">\b</span>uild +<span class="nb">cd</span> cpp<span class="se">\b</span>uild +<span class="nb">set</span> <span class="nv">ARROW_HOME</span><span class="o">=</span>C:<span class="se">\t</span>hirdparty +cmake -G <span class="s2">"Visual Studio 14 2015 Win64"</span> ^ + -DCMAKE_INSTALL_PREFIX<span class="o">=</span>%ARROW_HOME% ^ + -DCMAKE_BUILD_TYPE<span class="o">=</span>Release ^ + -DARROW_BUILD_TESTS<span class="o">=</span>off ^ + -DARROW_PYTHON<span class="o">=</span>on .. +cmake --build . --target INSTALL --config Release +<span class="nb">cd</span> ..<span class="se">\.</span>. +</pre></div> +</div> +<p>After that, we must put the install directory’s bin path in our <code class="docutils literal"><span class="pre">%PATH%</span></code>:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span><span class="nb">set</span> <span class="nv">PATH</span><span class="o">=</span>%ARROW_HOME%<span class="se">\b</span>in<span class="p">;</span>%PATH% +</pre></div> +</div> +<p>Now, we can build pyarrow:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span><span class="nb">cd</span> python +python setup.py build_ext --inplace +</pre></div> +</div> +<div class="section" id="running-c-unit-tests-with-python"> +<h3>Running C++ unit tests with Python<a class="headerlink" href="#running-c-unit-tests-with-python" title="Permalink to this headline">¶</a></h3> +<p>Getting <code class="docutils literal"><span class="pre">python-test.exe</span></code> to run is a bit tricky because your +<code class="docutils literal"><span class="pre">%PYTHONPATH%</span></code> must be configured given the active conda environment:</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span><span class="nb">set</span> <span class="nv">CONDA_ENV</span><span class="o">=</span>C:<span class="se">\U</span>sers<span class="se">\w</span>esm<span class="se">\M</span>iniconda<span class="se">\e</span>nvs<span class="se">\a</span>rrow-test +<span class="nb">set</span> <span class="nv">PYTHONPATH</span><span class="o">=</span>%CONDA_ENV%<span class="se">\L</span>ib<span class="p">;</span>%CONDA_ENV%<span class="se">\L</span>ib<span class="se">\s</span>ite-packages<span class="p">;</span>%CONDA_ENV%<span class="se">\p</span>ython35.zip<span class="p">;</span>%CONDA_ENV%<span class="se">\D</span>LLs<span class="p">;</span>%CONDA_ENV% +</pre></div> +</div> +<p>Now <code class="docutils literal"><span class="pre">python-test.exe</span></code> or simply <code class="docutils literal"><span class="pre">ctest</span></code> (to run all tests) should work.</p> +</div> +</div> +</div> + + + </div> + </div> + </div> + <div class="clearer"></div> + </div> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="genindex.html" title="General Index" + >index</a></li> + <li class="right" > + <a href="memory.html" title="Memory and IO Interfaces" + >next</a> |</li> + <li class="right" > + <a href="install.html" title="Install PyArrow" + >previous</a> |</li> + <li class="nav-item nav-item-0"><a href="index.html">pyarrow documentation</a> »</li> + </ul> + </div> + <div class="footer" role="contentinfo"> + © Copyright 2016-2017 Apache Software Foundation. + Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 1.5.5. + </div> + </body> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/filesystems.html ---------------------------------------------------------------------- diff --git a/docs/python/filesystems.html b/docs/python/filesystems.html index c86da9e..be34885 100644 --- a/docs/python/filesystems.html +++ b/docs/python/filesystems.html @@ -1,174 +1,97 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> -<!DOCTYPE html> -<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> -<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> -<head> - <meta charset="utf-8"> - - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - - <title>File interfaces and Memory Maps — pyarrow documentation</title> - - - - - - - - - - - +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - - - - - - <link rel="stylesheet" href="_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="genindex.html"/> - <link rel="search" title="Search" href="search.html"/> - <link rel="top" title="pyarrow documentation" href="index.html"/> - <link rel="next" title="Reading/Writing Parquet files" href="parquet.html"/> - <link rel="prev" title="Pandas Interface" href="pandas.html"/> - - - <script src="_static/js/modernizr.min.js"></script> - -</head> - -<body class="wy-body-for-nav" role="document"> - - - <div class="wy-grid-for-nav"> - + <title>Filesystem Interfaces — pyarrow documentation</title> - <nav data-toggle="wy-nav-shift" class="wy-nav-side"> - <div class="wy-side-scroll"> - <div class="wy-side-nav-search"> - - - - <a href="index.html" class="icon icon-home"> pyarrow - - - - </a> - - - - - - - -<div role="search"> - <form id="rtd-search-form" class="wy-form" action="search.html" method="get"> - <input type="text" name="q" placeholder="Search docs" /> - <input type="hidden" name="check_keywords" value="yes" /> - <input type="hidden" name="area" value="default" /> - </form> -</div> - - - </div> - - <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> - - - - - - - <p class="caption"><span class="caption-text">Getting Started</span></p> -<ul class="current"> -<li class="toctree-l1"><a class="reference internal" href="install.html">Install PyArrow</a></li> -<li class="toctree-l1"><a class="reference internal" href="pandas.html">Pandas Interface</a></li> -<li class="toctree-l1 current"><a class="current reference internal" href="#">File interfaces and Memory Maps</a><ul> -<li class="toctree-l2"><a class="reference internal" href="#hadoop-file-system-hdfs">Hadoop File System (HDFS)</a></li> + <link rel="stylesheet" href="_static/sphinxdoc.css" type="text/css" /> + <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: './', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' + }; + </script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> + <link rel="index" title="Index" href="genindex.html" /> + <link rel="search" title="Search" href="search.html" /> + <link rel="next" title="Using PyArrow with pandas" href="pandas.html" /> + <link rel="prev" title="IPC: Fast Streaming and Serialization" href="ipc.html" /> + </head> + <body role="document"> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="genindex.html" title="General Index" + accesskey="I">index</a></li> + <li class="right" > + <a href="pandas.html" title="Using PyArrow with pandas" + accesskey="N">next</a> |</li> + <li class="right" > + <a href="ipc.html" title="IPC: Fast Streaming and Serialization" + accesskey="P">previous</a> |</li> + <li class="nav-item nav-item-0"><a href="index.html">pyarrow documentation</a> »</li> + </ul> + </div> + <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> + <div class="sphinxsidebarwrapper"> + <h3><a href="index.html">Table Of Contents</a></h3> + <ul> +<li><a class="reference internal" href="#">Filesystem Interfaces</a><ul> +<li><a class="reference internal" href="#hadoop-file-system-hdfs">Hadoop File System (HDFS)</a></li> </ul> </li> -<li class="toctree-l1"><a class="reference internal" href="parquet.html">Reading/Writing Parquet files</a></li> -<li class="toctree-l1"><a class="reference internal" href="api.html">API Reference</a></li> -<li class="toctree-l1"><a class="reference internal" href="getting_involved.html">Getting Involved</a></li> -</ul> -<p class="caption"><span class="caption-text">Additional Features</span></p> -<ul> -<li class="toctree-l1"><a class="reference internal" href="jemalloc.html">jemalloc MemoryPool</a></li> </ul> - - + <h4>Previous topic</h4> + <p class="topless"><a href="ipc.html" + title="previous chapter">IPC: Fast Streaming and Serialization</a></p> + <h4>Next topic</h4> + <p class="topless"><a href="pandas.html" + title="next chapter">Using PyArrow with pandas</a></p> + <div role="note" aria-label="source link"> + <h3>This Page</h3> + <ul class="this-page-menu"> + <li><a href="_sources/filesystems.rst.txt" + rel="nofollow">Show Source</a></li> + </ul> + </div> +<div id="searchbox" style="display: none" role="search"> + <h3>Quick search</h3> + <form class="search" action="search.html" method="get"> + <div><input type="text" name="q" /></div> + <div><input type="submit" value="Go" /></div> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> +<script type="text/javascript">$('#searchbox').show(0);</script> </div> </div> - </nav> - - <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - - - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href="index.html">pyarrow</a> - - </nav> - - - - <div class="wy-nav-content"> - <div class="rst-content"> - - - - - - - - - - - - - - - -<div role="navigation" aria-label="breadcrumbs navigation"> - - <ul class="wy-breadcrumbs"> - - <li><a href="index.html">Docs</a> »</li> - - <li>File interfaces and Memory Maps</li> - - - <li class="wy-breadcrumbs-aside"> - + <div class="document"> + <div class="documentwrapper"> + <div class="bodywrapper"> + <div class="body" role="main"> - <a href="_sources/filesystems.rst.txt" rel="nofollow"> View page source</a> - - - </li> - - </ul> - - - <hr/> -</div> - <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> - <div itemprop="articleBody"> - - <div class="section" id="file-interfaces-and-memory-maps"> -<h1>File interfaces and Memory Maps<a class="headerlink" href="#file-interfaces-and-memory-maps" title="Permalink to this headline">¶</a></h1> -<p>PyArrow features a number of file-like interfaces</p> + <div class="section" id="filesystem-interfaces"> +<h1>Filesystem Interfaces<a class="headerlink" href="#filesystem-interfaces" title="Permalink to this headline">¶</a></h1> +<p>In this section, we discuss filesystem-like interfaces in PyArrow.</p> <div class="section" id="hadoop-file-system-hdfs"> -<h2>Hadoop File System (HDFS)<a class="headerlink" href="#hadoop-file-system-hdfs" title="Permalink to this headline">¶</a></h2> +<span id="hdfs"></span><h2>Hadoop File System (HDFS)<a class="headerlink" href="#hadoop-file-system-hdfs" title="Permalink to this headline">¶</a></h2> <p>PyArrow comes with bindings to a C++-based interface to the Hadoop File System. You connect like so:</p> <div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pyarrow</span> <span class="kn">as</span> <span class="nn">pa</span> @@ -187,7 +110,7 @@ relies on some environment variables.</p> installed somewhere other than <code class="docutils literal"><span class="pre">$HADOOP_HOME/lib/native</span></code>.</li> <li><code class="docutils literal"><span class="pre">CLASSPATH</span></code>: must contain the Hadoop jars. You can set these using:</li> </ul> -<div class="highlight-shell"><div class="highlight"><pre><span></span><span class="nb">export</span> <span class="nv">CLASSPATH</span><span class="o">=</span><span class="sb">`</span>$HADOOP_HOME/bin/hdfs classpath --glob<span class="sb">`</span> +<div class="highlight-shell"><div class="highlight"><pre><span></span><span class="nb">export</span> <span class="nv">CLASSPATH</span><span class="o">=</span><span class="sb">`</span><span class="nv">$HADOOP_HOME</span>/bin/hdfs classpath --glob<span class="sb">`</span> </pre></div> </div> <p>You can also use libhdfs3, a thirdparty C++ library for HDFS from Pivotal Labs:</p> @@ -199,76 +122,29 @@ installed somewhere other than <code class="docutils literal"><span class="pre"> </div> - </div> - <div class="articleComments"> - - </div> </div> - <footer> - - <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> - - <a href="parquet.html" class="btn btn-neutral float-right" title="Reading/Writing Parquet files" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> - - - <a href="pandas.html" class="btn btn-neutral" title="Pandas Interface" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> - - </div> - - - <hr/> - - <div role="contentinfo"> - <p> - © Copyright 2016 Apache Software Foundation. - - </p> - </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. - -</footer> - </div> </div> - - </section> - - </div> - - - - - - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> - <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> - - - - - - <script type="text/javascript" src="_static/js/theme.js"></script> - - - - - <script type="text/javascript"> - jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); - }); - </script> - - -</body> + <div class="clearer"></div> + </div> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="genindex.html" title="General Index" + >index</a></li> + <li class="right" > + <a href="pandas.html" title="Using PyArrow with pandas" + >next</a> |</li> + <li class="right" > + <a href="ipc.html" title="IPC: Fast Streaming and Serialization" + >previous</a> |</li> + <li class="nav-item nav-item-0"><a href="index.html">pyarrow documentation</a> »</li> + </ul> + </div> + <div class="footer" role="contentinfo"> + © Copyright 2016-2017 Apache Software Foundation. + Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 1.5.5. + </div> + </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/generated/pyarrow.Array.html ---------------------------------------------------------------------- diff --git a/docs/python/generated/pyarrow.Array.html b/docs/python/generated/pyarrow.Array.html index c115b5e..2e7e0ac 100644 --- a/docs/python/generated/pyarrow.Array.html +++ b/docs/python/generated/pyarrow.Array.html @@ -1,208 +1,84 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> -<!DOCTYPE html> -<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> -<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> -<head> - <meta charset="utf-8"> - - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - - <title>pyarrow.Array — pyarrow documentation</title> - - - - - - - - - - - +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - - - - - - <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../genindex.html"/> - <link rel="search" title="Search" href="../search.html"/> - <link rel="top" title="pyarrow documentation" href="../index.html"/> - <link rel="up" title="API Reference" href="../api.html"/> - <link rel="next" title="pyarrow.BooleanArray" href="pyarrow.BooleanArray.html"/> - <link rel="prev" title="pyarrow.array" href="pyarrow.array.html"/> - - - <script src="../_static/js/modernizr.min.js"></script> - -</head> - -<body class="wy-body-for-nav" role="document"> - - - <div class="wy-grid-for-nav"> - + <title>pyarrow.Array — pyarrow documentation</title> - <nav data-toggle="wy-nav-shift" class="wy-nav-side"> - <div class="wy-side-scroll"> - <div class="wy-side-nav-search"> - - - - <a href="../index.html" class="icon icon-home"> pyarrow - - - - </a> - - - - - - - -<div role="search"> - <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> - <input type="text" name="q" placeholder="Search docs" /> - <input type="hidden" name="check_keywords" value="yes" /> - <input type="hidden" name="area" value="default" /> - </form> + <link rel="stylesheet" href="../_static/sphinxdoc.css" type="text/css" /> + <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: '../', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' + }; + </script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> + <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> + <link rel="index" title="Index" href="../genindex.html" /> + <link rel="search" title="Search" href="../search.html" /> + <link rel="next" title="pyarrow.BooleanArray" href="pyarrow.BooleanArray.html" /> + <link rel="prev" title="pyarrow.array" href="pyarrow.array.html" /> + </head> + <body role="document"> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="../genindex.html" title="General Index" + accesskey="I">index</a></li> + <li class="right" > + <a href="pyarrow.BooleanArray.html" title="pyarrow.BooleanArray" + accesskey="N">next</a> |</li> + <li class="right" > + <a href="pyarrow.array.html" title="pyarrow.array" + accesskey="P">previous</a> |</li> + <li class="nav-item nav-item-0"><a href="../index.html">pyarrow documentation</a> »</li> + <li class="nav-item nav-item-1"><a href="../api.html" accesskey="U">API Reference</a> »</li> + </ul> + </div> + <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> + <div class="sphinxsidebarwrapper"> + <h4>Previous topic</h4> + <p class="topless"><a href="pyarrow.array.html" + title="previous chapter">pyarrow.array</a></p> + <h4>Next topic</h4> + <p class="topless"><a href="pyarrow.BooleanArray.html" + title="next chapter">pyarrow.BooleanArray</a></p> + <div role="note" aria-label="source link"> + <h3>This Page</h3> + <ul class="this-page-menu"> + <li><a href="../_sources/generated/pyarrow.Array.rst.txt" + rel="nofollow">Show Source</a></li> + </ul> + </div> +<div id="searchbox" style="display: none" role="search"> + <h3>Quick search</h3> + <form class="search" action="../search.html" method="get"> + <div><input type="text" name="q" /></div> + <div><input type="submit" value="Go" /></div> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> </div> - - - </div> - - <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> - - - - - - - <p class="caption"><span class="caption-text">Getting Started</span></p> -<ul class="current"> -<li class="toctree-l1"><a class="reference internal" href="../install.html">Install PyArrow</a></li> -<li class="toctree-l1"><a class="reference internal" href="../pandas.html">Pandas Interface</a></li> -<li class="toctree-l1"><a class="reference internal" href="../filesystems.html">File interfaces and Memory Maps</a></li> -<li class="toctree-l1"><a class="reference internal" href="../parquet.html">Reading/Writing Parquet files</a></li> -<li class="toctree-l1 current"><a class="reference internal" href="../api.html">API Reference</a><ul class="current"> -<li class="toctree-l2"><a class="reference internal" href="../api.html#type-and-schema-factory-functions">Type and Schema Factory Functions</a></li> -<li class="toctree-l2"><a class="reference internal" href="../api.html#scalar-value-types">Scalar Value Types</a></li> -<li class="toctree-l2 current"><a class="reference internal" href="../api.html#array-types-and-constructors">Array Types and Constructors</a><ul class="current"> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.array.html">pyarrow.array</a></li> -<li class="toctree-l3 current"><a class="current reference internal" href="#">pyarrow.Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.BooleanArray.html">pyarrow.BooleanArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.DictionaryArray.html">pyarrow.DictionaryArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.FloatingPointArray.html">pyarrow.FloatingPointArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.IntegerArray.html">pyarrow.IntegerArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.Int8Array.html">pyarrow.Int8Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.Int16Array.html">pyarrow.Int16Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.Int32Array.html">pyarrow.Int32Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.Int64Array.html">pyarrow.Int64Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.NullArray.html">pyarrow.NullArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.NumericArray.html">pyarrow.NumericArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.UInt8Array.html">pyarrow.UInt8Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.UInt16Array.html">pyarrow.UInt16Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.UInt32Array.html">pyarrow.UInt32Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.UInt64Array.html">pyarrow.UInt64Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.BinaryArray.html">pyarrow.BinaryArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.FixedSizeBinaryArray.html">pyarrow.FixedSizeBinaryArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.StringArray.html">pyarrow.StringArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.Time32Array.html">pyarrow.Time32Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.Time64Array.html">pyarrow.Time64Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.Date32Array.html">pyarrow.Date32Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.Date64Array.html">pyarrow.Date64Array</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.TimestampArray.html">pyarrow.TimestampArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.DecimalArray.html">pyarrow.DecimalArray</a></li> -<li class="toctree-l3"><a class="reference internal" href="pyarrow.ListArray.html">pyarrow.ListArray</a></li> -</ul> -</li> -<li class="toctree-l2"><a class="reference internal" href="../api.html#tables-and-record-batches">Tables and Record Batches</a></li> -<li class="toctree-l2"><a class="reference internal" href="../api.html#tensor-type-and-functions">Tensor type and Functions</a></li> -<li class="toctree-l2"><a class="reference internal" href="../api.html#input-output-and-shared-memory">Input / Output and Shared Memory</a></li> -<li class="toctree-l2"><a class="reference internal" href="../api.html#interprocess-communication-and-messaging">Interprocess Communication and Messaging</a></li> -<li class="toctree-l2"><a class="reference internal" href="../api.html#memory-pools">Memory Pools</a></li> -<li class="toctree-l2"><a class="reference internal" href="../api.html#type-classes">Type Classes</a></li> -<li class="toctree-l2"><a class="reference internal" href="../api.html#apache-parquet">Apache Parquet</a></li> -</ul> -</li> -<li class="toctree-l1"><a class="reference internal" href="../getting_involved.html">Getting Involved</a></li> -</ul> -<p class="caption"><span class="caption-text">Additional Features</span></p> -<ul> -<li class="toctree-l1"><a class="reference internal" href="../jemalloc.html">jemalloc MemoryPool</a></li> -</ul> - - - +<script type="text/javascript">$('#searchbox').show(0);</script> </div> </div> - </nav> - - <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - - - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href="../index.html">pyarrow</a> - - </nav> - - - - <div class="wy-nav-content"> - <div class="rst-content"> - - - - - - - - - - - - - - - -<div role="navigation" aria-label="breadcrumbs navigation"> - - <ul class="wy-breadcrumbs"> - - <li><a href="../index.html">Docs</a> »</li> - - <li><a href="../api.html">API Reference</a> »</li> - - <li>pyarrow.Array</li> - - - <li class="wy-breadcrumbs-aside"> - - - <a href="../_sources/generated/pyarrow.Array.rst.txt" rel="nofollow"> View page source</a> - - - </li> - - </ul> - - - <hr/> -</div> - <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> - <div itemprop="articleBody"> + <div class="document"> + <div class="documentwrapper"> + <div class="bodywrapper"> + <div class="body" role="main"> <div class="section" id="pyarrow-array"> <h1>pyarrow.Array<a class="headerlink" href="#pyarrow-array" title="Permalink to this headline">¶</a></h1> @@ -212,7 +88,7 @@ <dd><dl class="method"> <dt id="pyarrow.Array.__init__"> <code class="descname">__init__</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyarrow.Array.__init__" title="Permalink to this definition">¶</a></dt> -<dd><p>x.__init__(...) initializes x; see help(type(x)) for signature</p> +<dd><p>Initialize self. See help(type(self)) for accurate signature.</p> </dd></dl> <p class="rubric">Methods</p> @@ -262,76 +138,30 @@ </div> - </div> - <div class="articleComments"> - - </div> </div> - <footer> - - <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> - - <a href="pyarrow.BooleanArray.html" class="btn btn-neutral float-right" title="pyarrow.BooleanArray" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> - - - <a href="pyarrow.array.html" class="btn btn-neutral" title="pyarrow.array" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> - - </div> - - - <hr/> - - <div role="contentinfo"> - <p> - © Copyright 2016 Apache Software Foundation. - - </p> - </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. - -</footer> - </div> </div> - - </section> - - </div> - - - - - - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'../', - VERSION:'', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="../_static/jquery.js"></script> - <script type="text/javascript" src="../_static/underscore.js"></script> - <script type="text/javascript" src="../_static/doctools.js"></script> - <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> - - - - - - <script type="text/javascript" src="../_static/js/theme.js"></script> - - - - - <script type="text/javascript"> - jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); - }); - </script> - - -</body> + <div class="clearer"></div> + </div> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="../genindex.html" title="General Index" + >index</a></li> + <li class="right" > + <a href="pyarrow.BooleanArray.html" title="pyarrow.BooleanArray" + >next</a> |</li> + <li class="right" > + <a href="pyarrow.array.html" title="pyarrow.array" + >previous</a> |</li> + <li class="nav-item nav-item-0"><a href="../index.html">pyarrow documentation</a> »</li> + <li class="nav-item nav-item-1"><a href="../api.html" >API Reference</a> »</li> + </ul> + </div> + <div class="footer" role="contentinfo"> + © Copyright 2016-2017 Apache Software Foundation. + Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 1.5.5. + </div> + </body> </html> \ No newline at end of file
