Added: avro/site/publish/docs/1.11.3/_print/index.html URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/_print/index.html?rev=1919887&view=auto ============================================================================== --- avro/site/publish/docs/1.11.3/_print/index.html (added) +++ avro/site/publish/docs/1.11.3/_print/index.html Wed Aug 14 13:30:06 2024 @@ -0,0 +1,828 @@ +<!doctype html><html lang=en class=no-js><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1,shrink-to-fit=no"><meta name=generator content="Hugo 0.113.0"><link rel=canonical type=text/html href=/docs/1.11.3/><link rel=alternate type=application/rss+xml href=/docs/1.11.3/index.xml><meta name=robots content="noindex, nofollow"><link rel=apple-touch-icon sizes=57x57 href=https://apache.org/favicons/apple-touch-icon-57x57.png><link rel=apple-touch-icon sizes=60x60 href=https://apache.org/favicons/apple-touch-icon-60x60.png><link rel=apple-touch-icon sizes=72x72 href=https://apache.org/favicons/apple-touch-icon-72x72.png><link rel=apple-touch-icon sizes=76x76 href=https://apache.org/favicons/apple-touch-icon-76x76.png><link rel=apple-touch-icon sizes=114x114 href=https://apache.org/favicons/apple-touch-icon-114x114.png><link rel=apple-touch-icon sizes=120x120 href=https://apache.org/favicons/apple-touch-icon-120x120.png><link rel=apple-touch-icon si zes=144x144 href=https://apache.org/favicons/apple-touch-icon-144x144.png><link rel=apple-touch-icon sizes=152x152 href=https://apache.org/favicons/apple-touch-icon-152x152.png><link rel=apple-touch-icon sizes=180x180 href=https://apache.org/favicons/apple-touch-icon-180x180.png><link rel=icon type=image/png href=https://apache.org/favicons/favicon-32x32.png sizes=32x32><link rel=icon type=image/png href=https://apache.org/favicons/favicon-194x194.png sizes=194x194><link rel=icon type=image/png href=https://apache.org/favicons/favicon-96x96.png sizes=96x96><link rel=icon type=image/png href=https://apache.org/favicons/android-chrome-192x192.png sizes=192x192><link rel=icon type=image/png href=https://apache.org/favicons/favicon-16x16.png sizes=16x16><link rel=manifest href=https://apache.org/favicons/manifest.json><link rel="shortcut icon" href=https://apache.org/favicons/favicon.ico><title>Apache Avro⢠1.11.3 Documentation | Apache Avro</title><meta name=description content><m eta property="og:title" content="Apache Avro⢠1.11.3 Documentation"><meta property="og:description" content><meta property="og:type" content="website"><meta property="og:url" content="/docs/1.11.3/"><meta itemprop=name content="Apache Avro⢠1.11.3 Documentation"><meta itemprop=description content><meta name=twitter:card content="summary"><meta name=twitter:title content="Apache Avro⢠1.11.3 Documentation"><meta name=twitter:description content><link rel=preload href=/scss/main.min.e6198ccfb5f488caccc5f70e1d5d01bec7ba61523a1be10b1acc6ce8c9358f56.css as=style><link href=/scss/main.min.e6198ccfb5f488caccc5f70e1d5d01bec7ba61523a1be10b1acc6ce8c9358f56.css rel=stylesheet integrity><script src=https://code.jquery.com/jquery-3.5.1.min.js integrity="sha256-9/aliU8dGd2tb6OSsuzixeV4y/faTqgFtohetphbbj0=" crossorigin=anonymous></script> +<link rel=stylesheet href=/css/prism.css></head><body class=td-section><header><nav class="js-navbar-scroll navbar navbar-expand navbar-dark flex-column flex-md-row td-navbar"><a class=navbar-brand href=/><span class=navbar-logo><img src=/docs/1.11.3/logo.svg width=100 height=30 style="margin:0 10px"></span><span class="text-uppercase font-weight-bold">Apache Avro</span></a><div class="td-navbar-nav-scroll ml-md-auto" id=main_navbar><ul class="navbar-nav mt-2 mt-lg-0"><li class="nav-item mr-4 mb-2 mb-lg-0"><a class=nav-link href=/project/><span>Project</span></a></li><li class="nav-item mr-4 mb-2 mb-lg-0"><a class=nav-link href=/blog/><span>Blog</span></a></li><li class="nav-item mr-4 mb-2 mb-lg-0"><a class=nav-link href=/community/><span>Community</span></a></li><li class="nav-item dropdown mr-4 d-none d-lg-block"><a class="nav-link dropdown-toggle" href=# id=navbarDropdown role=button data-toggle=dropdown aria-haspopup=true aria-expanded=false>Documentation</a><div class=dropdown- menu aria-labelledby=navbarDropdownMenuLink><a class=dropdown-item href=/docs/1.11.3/>1.11.3 (Current)</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.11.0/>1.11.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.10.2/>1.10.2</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.10.1/>1.10.1</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.10.0/>1.10.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.9.2/>1.9.2</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.9.1/>1.9.1</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.9.0/>1.9.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.8.2/>1.8.2</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.8.1/>1.8.1</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.8.0/>1.8.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.7.7/>1.7.7</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.7.6/>1.7.6</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.7.5/>1.7.5</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.7.4/>1.7.4</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.7.3/>1.7.3</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.7.2/>1.7.2</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.7.1/>1.7.1</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.7.0/>1.7.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.6.3/>1.6.3</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.6.2/>1.6.2</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.6.1/>1.6.1</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.6.0/>1.6.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.5.4/>1.5.4</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.5.3/>1.5.3</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.5.2/>1.5.2</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.5.1/>1.5.1</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.5.0/>1.5.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.4.1/>1.4.1</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.4.0/>1.4.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.3.3/>1.3.3</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.3.2/>1.3.2</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.3.1/>1.3.1</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.3.0/>1.3.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.2.0/>1.2.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.1.0/>1.1.0</a> +<a class=dropdown-item href=https://avro.apache.org/docs/1.0.0/>1.0.0</a></div></li></ul></div><div class="navbar-nav d-none d-lg-block"></div></nav></header><div class="container-fluid td-outer"><div class=td-main><div class="row flex-xl-nowrap"><main class="col-12 col-md-9 col-xl-8 pl-md-5" role=main><div class=td-content><div class="pageinfo pageinfo-primary d-print-none"><p>This is the multi-page printable view of this section. +<a href=# onclick="return print(),!1">Click here to print</a>.</p><p><a href=/docs/1.11.3/>Return to the regular view of this page</a>.</p></div><h1 class=title>Apache Avro⢠1.11.3 Documentation</h1><ul><li>1: <a href=#pg-655753e4815eb9b13f98b73aabb0ffd3>Getting Started (Java)</a></li><ul></ul><li>2: <a href=#pg-4f2642fff58292dde451ff409d479720>Getting Started (Python)</a></li><ul></ul><li>3: <a href=#pg-d9956c22c61ec53d9e37fab869629872>Specification</a></li><ul></ul><li>4: <a href=#pg-1c5c47ac952ea6cc621e98b4a8cad26c>Java API</a></li><li>5: <a href=#pg-291cee60e4e1987850a1b954ba1f30e1>C API</a></li><li>6: <a href=#pg-bcbcac4b420e051fab661f0704d02e76>C++ API</a></li><li>7: <a href=#pg-7ffc5b23b834d5b2b890f962c7b9ce63>C# API</a></li><li>8: <a href=#pg-dd932686ac60cf5aa786c6b48519d3c9>Python API</a></li><li>9: <a href=#pg-2b97fbe7574a73d2403b40427372a243>MapReduce guide</a></li><ul></ul><li>10: <a href=#pg-6b7c552261a057985add7d501e95a5ef>IDL Language</a></li><ul></ul><li>11: <a href=#pg-9b34b66d18b175f8dff6cb86f2480be4>SASL profile</a></li><ul></ul></ul><div class=content><h2 id=introduction>Introduction</h2><p>Apache Avro⢠is a data serialization system.</p><p>Avro provides:</p><ul><li>Rich data structures.</li><li>A compact, fast, binary data format.</li><li>A container file, to store persistent data.</li><li>Remote procedure call (RPC).</li><li>Simple integration with dynamic languages. Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages.</li></ul><h2 id=schemas>Schemas</h2><p>Avro relies on schemas. When Avro data is read, the schema used when writing it is always present. This permits each datum to be written with no per-value overheads, making serialization both fast and small. This also facilitates use with dynamic, scripting languages, since data, together with its schema, is fully self-describing.< /p><p>When Avro data is stored in a file, its schema is stored with it, so that files may be processed later by any program. If the program reading the data expects a different schema this can be easily resolved, since both schemas are present.</p><p>When Avro is used in RPC, the client and server exchange schemas in the connection handshake. (This can be optimized so that, for most calls, no schemas are actually transmitted.) Since both client and server both have the other’s full schema, correspondence between same named fields, missing fields, extra fields, etc. can all be easily resolved.</p><p>Avro schemas are defined with JSON . This facilitates implementation in languages that already have JSON libraries.</p><h2 id=comparison-with-other-systems>Comparison with other systems</h2><p>Avro provides functionality similar to systems such as <a href=https://thrift.apache.org/>Thrift</a>, <a href=https://code.google.com/p/protobuf/>Protocol Buffers</a>, etc. Avro differs from t hese systems in the following fundamental aspects.</p><ul><li>Dynamic typing: Avro does not require that code be generated. Data is always accompanied by a schema that permits full processing of that data without code generation, static datatypes, etc. This facilitates construction of generic data-processing systems and languages.</li><li>Untagged data: Since the schema is present when data is read, considerably less type information need be encoded with data, resulting in smaller serialization size.</li><li>No manually-assigned field IDs: When a schema changes, both the old and new schema are always present when processing data, so differences may be resolved symbolically, using field names.</li></ul></div></div><div class=td-content style=page-break-before:always><h1 id=pg-655753e4815eb9b13f98b73aabb0ffd3>1 - Getting Started (Java)</h1><p>This is a short guide for getting started with Apache Avro⢠using Java. This guide only covers using Avro for data serialization; see Patri ck Hunt’s <a href=https://github.com/phunt/avro-rpc-quickstart>Avro RPC Quick Start</a> for a good introduction to using Avro for RPC.</p><h2 id=download>Download</h2><p>Avro implementations for C, C++, C#, Java, PHP, Python, and Ruby can be downloaded from the <a href=/project/download/>Apache Avro⢠Download</a> page. This guide uses Avro 1.11.3, the latest version at the time of writing. For the examples in this guide, download avro-1.11.3.jar and avro-tools-1.11.3.jar.</p><p>Alternatively, if you are using Maven, add the following dependency to your POM:</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-xml data-lang=xml><span style=display:flex><span><span style=color:#204a87;font-weight:700><dependency></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><groupId></span>org.apache.avro<span style=color:#204a87;font-weight:700></groupId></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><artifactId></span>avro<span style=color:#204a87;font-weight:700></artifactId></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><version></span>1.11.3<span style=color:#204a87;font-weight:700></version></span> +</span></span><span style=display:flex><span><span style=color:#204a87;font-weight:700></dependency></span> +</span></span></code></pre></div><p>As well as the Avro Maven plugin (for performing code generation):</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-xml data-lang=xml><span style=display:flex><span><span style=color:#204a87;font-weight:700><plugin></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><groupId></span>org.apache.avro<span style=color:#204a87;font-weight:700></groupId></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><artifactId></span>avro-maven-plugin<span style=color:#204a87;font-weight:700></artifactId></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><version></span>1.11.3<span style=color:#204a87;font-weight:700></version></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><executions></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><execution></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><phase></span>generate-sources<span style=color:#204a87;font-weight:700></phase></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><goals></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><goal></span>schema<span style=color:#204a87;font-weight:700></goal></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700></goals></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><configuration></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><sourceDirectory></span>${project.basedir}/src/main/avro/<span style=color:#204a87;font-weight:700></sourceDirectory></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><outputDirectory></span>${project.basedir}/src/main/java/<span style=color:#204a87;font-weight:700></outputDirectory></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700></configuration></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700></execution></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700></executions></span> +</span></span><span style=display:flex><span><span style=color:#204a87;font-weight:700></plugin></span> +</span></span><span style=display:flex><span><span style=color:#204a87;font-weight:700><plugin></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><groupId></span>org.apache.maven.plugins<span style=color:#204a87;font-weight:700></groupId></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><artifactId></span>maven-compiler-plugin<span style=color:#204a87;font-weight:700></artifactId></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><configuration></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><source></span>1.8<span style=color:#204a87;font-weight:700></source></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700><target></span>1.8<span style=color:#204a87;font-weight:700></target></span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700></configuration></span> +</span></span><span style=display:flex><span><span style=color:#204a87;font-weight:700></plugin></span> +</span></span></code></pre></div><p>You may also build the required Avro jars from source. Building Avro is beyond the scope of this guide; see the Build Documentation page in the wiki for more information.</p><h2 id=defining-a-schema>Defining a schema</h2><p>Avro schemas are defined using JSON. Schemas are composed of primitive types (null, boolean, int, long, float, double, bytes, and string) and complex types (record, enum, array, map, union, and fixed). You can learn more about Avro schemas and types from the specification, but for now let’s start with a simple schema example, user.avsc:</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-json data-lang=json><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"namespace"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"example.avro&# 34;</span><span style=color:#000;font-weight:700>,</span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700>"type"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"record"</span><span style=color:#000;font-weight:700>,</span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"User"</span><span style=color:#000;font-weight:700>,</span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700>"fields"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#000;font-weight:700>[</span> +</span></span><span style=display:flex><span> <span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"name"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"type"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"string"</span><span style=color:#000;font-weight:700>},</span> +</span></span><span style=display:flex><span> <span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"favorite_number"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"type"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#000;font-weight:700>[</span><span style=color:#4e9a06>"int"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"null"</span><span style=color:#000;font-weight:700>]},</span> +</span></span><span style=display:flex><span> <span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"favorite_color"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"type"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#000;font-weight:700>[</span><span style=color:#4e9a06>"string"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"null"</span><span style=color:#000;font-weight:700>]}</span> +</span></span><span style=display:flex><span> <span style=color:#000;font-weight:700>]</span> +</span></span><span style=display:flex><span><span style=color:#000;font-weight:700>}</span> +</span></span></code></pre></div><p>This schema defines a record representing a hypothetical user. (Note that a schema file can only contain a single schema definition.) At minimum, a record definition must include its type (“type”: “record”), a name (“name”: “User”), and fields, in this case name, favorite_number, and favorite_color. We also define a namespace (“namespace”: “example.avro”), which together with the name attribute defines the “full name” of the schema (example.avro.User in this case).</p><p>Fields are defined via an array of objects, each of which defines a name and type (other attributes are optional, see the record specification for more details). The type attribute of a field is another schema object, which can be either a primitive or complex type. For example, the name field of our User schema is the primitive type string, whereas the favorite_number and favorite_color fields are both unions, represented by JSON arrays. unions are a complex type that can be any of the types listed in the array; e.g., favorite_number can either be an int or null, essentially making it an optional field.</p><h2 id=serializing-and-deserializing-with-code-generation>Serializing and deserializing with code generation</h2><h3 id=compiling-the-schema>Compiling the schema</h3><p>Code generation allows us to automatically create classes based on our previously-defined schema. Once we have defined the relevant classes, there is no need to use the schema directly in our programs. We use the avro-tools jar to generate code as follows:</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-shell data-lang=shell><span style=display:flex><span>java -jar /path/to/avro-tools-1.11.3.jar compile schema <schema file> <destination> +</span></span></code></pre></div><p>This will generate the appropriate source files in a package based on the schema’s namespace in the provided destination folder. For instance, to generate a User class in package example.avro from the schema defined above, run</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-shell data-lang=shell><span style=display:flex><span>java -jar /path/to/avro-tools-1.11.3.jar compile schema user.avsc . +</span></span></code></pre></div><p>Note that if you using the Avro Maven plugin, there is no need to manually invoke the schema compiler; the plugin automatically performs code generation on any .avsc files present in the configured source directory.</p><h3 id=creating-users>Creating Users</h3><p>Now that we’ve completed the code generation, let’s create some Users, serialize them to a data file on disk, and then read back the file and deserialize the User objects.</p><p>First let’s create some Users and set their fields.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#000>User</span> <span style=color:#000>user1</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>();</span> +</span></span><span style=display:flex><span><span style=color:#000>user1</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>setName</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"Alyssa"</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>user1</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>setFavoriteNumber</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#0000cf;font-weight:700>256</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// Leave favorite color null +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic></span> +</span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// Alternate constructor +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic></span><span style=color:#000>User</span> <span style=color:#000>user2</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"Ben"</span><span style=color:#ce5c00;font-weight:700>,</span> <span style=color:#0000cf;font-weight:700>7</span><span style=color:#ce5c00;font-weight:700>,</span> <span style=color:#4e9a06>"red"</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span> +</span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// Construct via builder +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic></span><span style=color:#000>User</span> <span style=color:#000>user3</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>newBuilder</span><span style=color:#ce5c00;font-weight:700>()</span> +</span></span><span style=display:flex><span> <span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>setName</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"Charlie"</span><span style=color:#ce5c00;font-weight:700>)</span> +</span></span><span style=display:flex><span> <span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>setFavoriteColor</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"blue"</span><span style=color:#ce5c00;font-weight:700>)</span> +</span></span><span style=display:flex><span> <span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>setFavoriteNumber</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#204a87;font-weight:700>null</span><span style=color:#ce5c00;font-weight:700>)</span> +</span></span><span style=display:flex><span> <span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>build</span><span style=color:#ce5c00;font-weight:700>();</span> +</span></span></code></pre></div><p>As shown in this example, Avro objects can be created either by invoking a constructor directly or by using a builder. Unlike constructors, builders will automatically set any default values specified in the schema. Additionally, builders validate the data as it set, whereas objects constructed directly will not cause an error until the object is serialized. However, using constructors directly generally offers better performance, as builders create a copy of the datastructure before it is written.</p><p>Note that we do not set user1’s favorite color. Since that record is of type [“string”, “null”], we can either set it to a string or leave it null; it is essentially optional. Similarly, we set user3’s favorite number to null (using a builder requires setting all fields, even if they are null).</p><h3 id=serializing>Serializing</h3><p>Now let’s serialize our Users to disk.</p><div class=highlight><pre tabi ndex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// Serialize user1, user2 and user3 to disk +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic></span><span style=color:#000>DatumWriter</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>></span> <span style=color:#000>userDatumWriter</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>SpecificDatumWriter</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>>(</span><span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>class</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>DataFileWriter</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>></span> <span style=color:#000>dataFileWriter</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>DataFileWriter</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>>(</span><span style=color:#000>userDatumWriter</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>dataFileWriter</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>create</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>user1</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>getSchema</span><span style=color:#ce5c00;font-weight:700>(),</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>File</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"users.avro"</span><span style=color:#ce5c00;font-weight:700>));</span> +</span></span><span style=display:flex><span><span style=color:#000>dataFileWriter</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>append</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>user1</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>dataFileWriter</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>append</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>user2</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>dataFileWriter</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>append</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>user3</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>dataFileWriter</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>close</span><span style=color:#ce5c00;font-weight:700>();</span> +</span></span></code></pre></div><p>We create a DatumWriter, which converts Java objects into an in-memory serialized format. The SpecificDatumWriter class is used with generated classes and extracts the schema from the specified generated type.</p><p>Next we create a DataFileWriter, which writes the serialized records, as well as the schema, to the file specified in the dataFileWriter.create call. We write our users to the file via calls to the dataFileWriter.append method. When we are done writing, we close the data file.</p><h3 id=deserializing>Deserializing</h3><p>Finally, let’s deserialize the data file we just created.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// Deserialize Users from disk +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic></span><span style=color:#000>DatumReader</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>></span> <span style=color:#000>userDatumReader</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>SpecificDatumReader</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>>(</span><span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>class</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>DataFileReader</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>></span> <span style=color:#000>dataFileReader</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>DataFileReader</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>User</span><span style=color:#ce5c00;font-weight:700>>(</span><span style=color:#000>file</span><span style=color:#ce5c00;font-weight:700>,</span> <span style=color:#000>userDatumReader</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>User</span> <span style=color:#000>user</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>null</span><span style=color:#ce5c00;font-weight:700>;</span> +</span></span><span style=display:flex><span><span style=color:#204a87;font-weight:700>while</span> <span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>dataFileReader</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>hasNext</span><span style=color:#ce5c00;font-weight:700>())</span> <span style=color:#ce5c00;font-weight:700>{</span> +</span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// Reuse user object by passing it to next(). This saves us from +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// allocating and garbage collecting many objects for files with +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// many items. +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic></span><span style=color:#000>user</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#000>dataFileReader</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>next</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>user</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>System</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>out</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>println</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>user</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#ce5c00;font-weight:700>}</span> +</span></span></code></pre></div><p>This snippet will output:</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-json data-lang=json><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"Alyssa"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"favorite_number"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#0000cf;font-weight:700>256</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"favorite_color"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#204a87;font-weight:700>null</span><span style=color:#000;font-weight:700>}</span> +</span></span><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"Ben"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"favorite_number"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#0000cf;font-weight:700>7</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"favorite_color"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"red"</span><span style=color:#000;font-weight:700>}</span> +</span></span><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"Charlie"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"favorite_number"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#204a87;font-weight:700>null</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"favorite_color"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"blue"</span><span style=color:#000;font-weight:700>}</span> +</span></span></code></pre></div><p>Deserializing is very similar to serializing. We create a SpecificDatumReader, analogous to the SpecificDatumWriter we used in serialization, which converts in-memory serialized items into instances of our generated class, in this case User. We pass the DatumReader and the previously created File to a DataFileReader, analogous to the DataFileWriter, which reads both the schema used by the writer as well as the data from the file on disk. The data will be read using the writer’s schema included in the file and the schema provided by the reader, in this case the User class. The writer’s schema is needed to know the order in which fields were written, while the reader’s schema is needed to know what fields are expected and how to fill in default values for fields added since the file was written. If there are differences between the two schemas, they are resolved according to the Schema Resolution specification.</p><p>Next we use th e DataFileReader to iterate through the serialized Users and print the deserialized object to stdout. Note how we perform the iteration: we create a single User object which we store the current deserialized user in, and pass this record object to every call of dataFileReader.next. This is a performance optimization that allows the DataFileReader to reuse the same User object rather than allocating a new User for every iteration, which can be very expensive in terms of object allocation and garbage collection if we deserialize a large data file. While this technique is the standard way to iterate through a data file, it’s also possible to use for (User user : dataFileReader) if performance is not a concern.</p><h3 id=compiling-and-running-the-example-code>Compiling and running the example code</h3><p>This example code is included as a Maven project in the examples/java-example directory in the Avro docs. From this directory, execute the following commands to build and run the example:</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-shell data-lang=shell><span style=display:flex><span>$ mvn compile <span style=color:#8f5902;font-style:italic># includes code generation via Avro Maven plugin</span> +</span></span><span style=display:flex><span>$ mvn -q exec:java -Dexec.mainClass<span style=color:#ce5c00;font-weight:700>=</span>example.SpecificMain +</span></span></code></pre></div><h3 id=beta-feature-generating-faster-code>Beta feature: Generating faster code</h3><p>In release 1.9.0, we introduced a new approach to generating code that speeds up decoding of objects by more than 10% and encoding by more than 30% (future performance enhancements are underway). To ensure a smooth introduction of this change into production systems, this feature is controlled by a feature flag, the system property org.apache.avro.specific.use_custom_coders. In this first release, this feature is off by default. To turn it on, set the system flag to true at runtime. In the sample above, for example, you could enable the fater coders as follows:</p><p>$ mvn -q exec:java -Dexec.mainClass=example.SpecificMain<br>-Dorg.apache.avro.specific.use_custom_coders=true</p><p>Note that you do not have to recompile your Avro schema to have access to this feature. The feature is compiled and built into your code, and you turn it on and off at runtime using the f eature flag. As a result, you can turn it on during testing, for example, and then off in production. Or you can turn it on in production, and quickly turn it off if something breaks.</p><p>We encourage the Avro community to exercise this new feature early to help build confidence. (For those paying one-demand for compute resources in the cloud, it can lead to meaningful cost savings.) As confidence builds, we will turn this feature on by default, and eventually eliminate the feature flag (and the old code).</p><h2 id=serializing-and-deserializing-without-code-generation>Serializing and deserializing without code generation</h2><p>Data in Avro is always stored with its corresponding schema, meaning we can always read a serialized item regardless of whether we know the schema ahead of time. This allows us to perform serialization and deserialization without code generation.</p><p>Let’s go over the same example as in the previous section, but without using code generation: we&rs quo;ll create some users, serialize them to a data file on disk, and then read back the file and deserialize the users objects.</p><h3 id=creating-users-1>Creating users</h3><p>First, we use a Parser to read our schema definition and create a Schema object.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#000>Schema</span> <span style=color:#000>schema</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>Schema</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>Parser</span><span style=color:#ce5c00;font-weight:700>().</span><span style=color:#c4a000>parse</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>File</span><span style=color:#ce5c00 ;font-weight:700>(</span><span style=color:#4e9a06>"user.avsc"</span><span style=color:#ce5c00;font-weight:700>));</span> +</span></span></code></pre></div><p>Using this schema, let’s create some users.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#000>GenericRecord</span> <span style=color:#000>user1</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>GenericData</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>Record</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>schema</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>user1</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>put</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"name"</span><span style=color:#ce5c00;font-weight:700>,</span> <span style=color:#4e9a06>"Alyssa"</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>user1</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>put</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"favorite_number"</span><span style=color:#ce5c00;font-weight:700>,</span> <span style=color:#0000cf;font-weight:700>256</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// Leave favorite color null +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic></span> +</span></span><span style=display:flex><span><span style=color:#000>GenericRecord</span> <span style=color:#000>user2</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>GenericData</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>Record</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>schema</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>user2</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>put</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"name"</span><span style=color:#ce5c00;font-weight:700>,</span> <span style=color:#4e9a06>"Ben"</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>user2</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>put</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"favorite_number"</span><span style=color:#ce5c00;font-weight:700>,</span> <span style=color:#0000cf;font-weight:700>7</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>user2</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>put</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"favorite_color"</span><span style=color:#ce5c00;font-weight:700>,</span> <span style=color:#4e9a06>"red"</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span></code></pre></div><p>Since we’re not using code generation, we use GenericRecords to represent users. GenericRecord uses the schema to verify that we only specify valid fields. If we try to set a non-existent field (e.g., user1.put(“favorite_animal”, “cat”)), we’ll get an AvroRuntimeException when we run the program.</p><p>Note that we do not set user1’s favorite color. Since that record is of type [“string”, “null”], we can either set it to a string or leave it null; it is essentially optional.</p><h3 id=serializing-1>Serializing</h3><p>Now that we’ve created our user objects, serializing and deserializing them is almost identical to the example above which uses code generation. The main difference is that we use generic instead of specific readers and writers.</p><p>First we’ll serialize our users to a data file on disk.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f 8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// Serialize user1 and user2 to disk +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic></span><span style=color:#000>File</span> <span style=color:#000>file</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>File</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#4e9a06>"users.avro"</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>DatumWriter</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>GenericRecord</span><span style=color:#ce5c00;font-weight:700>></span> <span style=color:#000>datumWriter</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>GenericDatumWriter</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>GenericRecord</span><span style=color:#ce5c00;font-weight:700>>(</span><span style=color:#000>schema</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>DataFileWriter</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>GenericRecord</span><span style=color:#ce5c00;font-weight:700>></span> <span style=color:#000>dataFileWriter</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>DataFileWriter</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>GenericRecord</span><span style=color:#ce5c00;font-weight:700>>(</span><span style=color:#000>datumWriter</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>dataFileWriter</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>create</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>schema</span><span style=color:#ce5c00;font-weight:700>,</span> <span style=color:#000>file</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>dataFileWriter</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>append</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>user1</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>dataFileWriter</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>append</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>user2</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>dataFileWriter</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>close</span><span style=color:#ce5c00;font-weight:700>();</span> +</span></span></code></pre></div><p>We create a DatumWriter, which converts Java objects into an in-memory serialized format. Since we are not using code generation, we create a GenericDatumWriter. It requires the schema both to determine how to write the GenericRecords and to verify that all non-nullable fields are present.</p><p>As in the code generation example, we also create a DataFileWriter, which writes the serialized records, as well as the schema, to the file specified in the dataFileWriter.create call. We write our users to the file via calls to the dataFileWriter.append method. When we are done writing, we close the data file.</p><h3 id=deserializing-1>Deserializing</h3><p>Finally, we’ll deserialize the data file we just created.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// Deseriali ze users from disk +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic></span><span style=color:#000>DatumReader</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>GenericRecord</span><span style=color:#ce5c00;font-weight:700>></span> <span style=color:#000>datumReader</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>GenericDatumReader</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>GenericRecord</span><span style=color:#ce5c00;font-weight:700>>(</span><span style=color:#000>schema</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>DataFileReader</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>GenericRecord</span><span style=color:#ce5c00;font-weight:700>></span> <span style=color:#000>dataFileReader</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>new</span> <span style=color:#000>DataFileReader</span><span style=color:#ce5c00;font-weight:700><</span><span style=color:#000>GenericRecord</span><span style=color:#ce5c00;font-weight:700>>(</span><span style=color:#000>file</span><span style=color:#ce5c00;font-weight:700>,</span> <span style=color:#000>datumReader</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>GenericRecord</span> <span style=color:#000>user</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#204a87;font-weight:700>null</span><span style=color:#ce5c00;font-weight:700>;</span> +</span></span><span style=display:flex><span><span style=color:#204a87;font-weight:700>while</span> <span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>dataFileReader</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>hasNext</span><span style=color:#ce5c00;font-weight:700>())</span> <span style=color:#ce5c00;font-weight:700>{</span> +</span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// Reuse user object by passing it to next(). This saves us from +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// allocating and garbage collecting many objects for files with +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic>// many items. +</span></span></span><span style=display:flex><span><span style=color:#8f5902;font-style:italic></span><span style=color:#000>user</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#000>dataFileReader</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>next</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>user</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span><span style=display:flex><span><span style=color:#000>System</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>out</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#c4a000>println</span><span style=color:#ce5c00;font-weight:700>(</span><span style=color:#000>user</span><span style=color:#ce5c00;font-weight:700>);</span> +</span></span></code></pre></div><p>This outputs:</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-json data-lang=json><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"Alyssa"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"favorite_number"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#0000cf;font-weight:700>256</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"favorite_color"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#204a87;font-weight:700>null</span><span style=color:#000;font-weight:700>}</span> +</span></span><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"Ben"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"favorite_number"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#0000cf;font-weight:700>7</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"favorite_color"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"red"</span><span style=color:#000;font-weight:700>}</span> +</span></span></code></pre></div><p>Deserializing is very similar to serializing. We create a GenericDatumReader, analogous to the GenericDatumWriter we used in serialization, which converts in-memory serialized items into GenericRecords. We pass the DatumReader and the previously created File to a DataFileReader, analogous to the DataFileWriter, which reads both the schema used by the writer as well as the data from the file on disk. The data will be read using the writer’s schema included in the file, and the reader’s schema provided to the GenericDatumReader. The writer’s schema is needed to know the order in which fields were written, while the reader’s schema is needed to know what fields are expected and how to fill in default values for fields added since the file was written. If there are differences between the two schemas, they are resolved according to the Schema Resolution specification.</p><p>Next, we use the DataFileReader to iterate through the serialized users and print the deserialized object to stdout. Note how we perform the iteration: we create a single GenericRecord object which we store the current deserialized user in, and pass this record object to every call of dataFileReader.next. This is a performance optimization that allows the DataFileReader to reuse the same record object rather than allocating a new GenericRecord for every iteration, which can be very expensive in terms of object allocation and garbage collection if we deserialize a large data file. While this technique is the standard way to iterate through a data file, it’s also possible to use for (GenericRecord user : dataFileReader) if performance is not a concern.</p><h3 id=compiling-and-running-the-example-code-1>Compiling and running the example code</h3><p>This example code is included as a Maven project in the examples/java-example directory in the Avro docs. From this directory, execute the following commands to build and run the example: </p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-shell data-lang=shell><span style=display:flex><span>$ mvn compile +</span></span><span style=display:flex><span>$ mvn -q exec:java -Dexec.mainClass<span style=color:#ce5c00;font-weight:700>=</span>example.GenericMain +</span></span></code></pre></div></div><div class=td-content style=page-break-before:always><h1 id=pg-4f2642fff58292dde451ff409d479720>2 - Getting Started (Python)</h1><p>This is a short guide for getting started with Apache Avro⢠using Python. This guide only covers using Avro for data serialization; see Patrick Hunt’s Avro RPC Quick Start for a good introduction to using Avro for RPC.</p><h2 id=notice-for-python-3-users>Notice for Python 3 users</h2><p>A package called “avro-python3” had been provided to support Python 3 previously, but the codebase was consolidated into the “avro” package and that supports both Python 2 and 3 now. The avro-python3 package will be removed in the near future, so users should use the “avro” package instead. They are mostly API compatible, but there’s a few minor difference (e.g., function name capitalization, such as avro.schema.Parse vs avro.schema.parse).</p><h2 id=download>Download</h2><p>For P ython, the easiest way to get started is to install it from PyPI. Python’s Avro API is available over PyPi.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-shell data-lang=shell><span style=display:flex><span>$ python3 -m pip install avro +</span></span></code></pre></div><p>The official releases of the Avro implementations for C, C++, C#, Java, PHP, Python, and Ruby can be downloaded from the Apache Avro⢠Releases page. This guide uses Avro 1.11.3, the latest version at the time of writing. Download and unzip avro-1.11.3.tar.gz, and install via python setup.py (this will probably require root privileges). Ensure that you can import avro from a Python prompt.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-shell data-lang=shell><span style=display:flex><span>$ tar xvf avro-1.11.3.tar.gz +</span></span><span style=display:flex><span>$ <span style=color:#204a87>cd</span> avro-1.11.3 +</span></span><span style=display:flex><span>$ python setup.py install +</span></span><span style=display:flex><span>$ python +</span></span><span style=display:flex><span>>>> import avro <span style=color:#8f5902;font-style:italic># should not raise ImportError</span> +</span></span></code></pre></div><p>Alternatively, you may build the Avro Python library from source. From your the root Avro directory, run the commands</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-shell data-lang=shell><span style=display:flex><span>$ <span style=color:#204a87>cd</span> lang/py/ +</span></span><span style=display:flex><span>$ python3 -m pip install -e . +</span></span><span style=display:flex><span>$ python +</span></span></code></pre></div><h2 id=defining-a-schema>Defining a schema</h2><p>Avro schemas are defined using JSON. Schemas are composed of primitive types (null, boolean, int, long, float, double, bytes, and string) and complex types (record, enum, array, map, union, and fixed). You can learn more about Avro schemas and types from the specification, but for now let’s start with a simple schema example, user.avsc:</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-json data-lang=json><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"namespace"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"example.avro"</span><span style=color:#000;font-weight:700>,</span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700>"type"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"record"</span><span style=color:#000;font-weight:700>,</span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"User"</span><span style=color:#000;font-weight:700>,</span> +</span></span><span style=display:flex><span> <span style=color:#204a87;font-weight:700>"fields"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#000;font-weight:700>[</span> +</span></span><span style=display:flex><span> <span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"name"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"type"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"string"</span><span style=color:#000;font-weight:700>},</span> +</span></span><span style=display:flex><span> <span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"favorite_number"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"type"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#000;font-weight:700>[</span><span style=color:#4e9a06>"int"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"null"</span><span style=color:#000;font-weight:700>]},</span> +</span></span><span style=display:flex><span> <span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"favorite_color"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#204a87;font-weight:700>"type"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#000;font-weight:700>[</span><span style=color:#4e9a06>"string"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"null"</span><span style=color:#000;font-weight:700>]}</span> +</span></span><span style=display:flex><span> <span style=color:#000;font-weight:700>]</span> +</span></span><span style=display:flex><span><span style=color:#000;font-weight:700>}</span> +</span></span></code></pre></div><p>This schema defines a record representing a hypothetical user. (Note that a schema file can only contain a single schema definition.) At minimum, a record definition must include its type (“type”: “record”), a name (“name”: “User”), and fields, in this case name, favorite_number, and favorite_color. We also define a namespace (“namespace”: “example.avro”), which together with the name attribute defines the “full name” of the schema (example.avro.User in this case).</p><p>Fields are defined via an array of objects, each of which defines a name and type (other attributes are optional, see the record specification for more details). The type attribute of a field is another schema object, which can be either a primitive or complex type. For example, the name field of our User schema is the primitive type string, whereas the favorite_number and favorite_color fields are both unions, represented by JSON arrays. unions are a complex type that can be any of the types listed in the array; e.g., favorite_number can either be an int or null, essentially making it an optional field.</p><h2 id=serializing-and-deserializing-without-code-generation>Serializing and deserializing without code generation</h2><p>Data in Avro is always stored with its corresponding schema, meaning we can always read a serialized item, regardless of whether we know the schema ahead of time. This allows us to perform serialization and deserialization without code generation. Note that the Avro Python library does not support code generation.</p><p>Try running the following code snippet, which serializes two users to a data file on disk, and then reads back and deserializes the data file:</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span><span st yle=color:#204a87;font-weight:700>import</span> <span style=color:#000>avro.schema</span> +</span></span><span style=display:flex><span><span style=color:#204a87;font-weight:700>from</span> <span style=color:#000>avro.datafile</span> <span style=color:#204a87;font-weight:700>import</span> <span style=color:#000>DataFileReader</span><span style=color:#000;font-weight:700>,</span> <span style=color:#000>DataFileWriter</span> +</span></span><span style=display:flex><span><span style=color:#204a87;font-weight:700>from</span> <span style=color:#000>avro.io</span> <span style=color:#204a87;font-weight:700>import</span> <span style=color:#000>DatumReader</span><span style=color:#000;font-weight:700>,</span> <span style=color:#000>DatumWriter</span> +</span></span><span style=display:flex><span> +</span></span><span style=display:flex><span><span style=color:#000>schema</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#000>avro</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>schema</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>parse</span><span style=color:#000;font-weight:700>(</span><span style=color:#204a87>open</span><span style=color:#000;font-weight:700>(</span><span style=color:#4e9a06>"user.avsc"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"rb"</span><span style=color:#000;font-weight:700>)</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>read</span><span style=color:#000;font-weight:700>())</span> +</span></span><span style=display:flex><span> +</span></span><span style=display:flex><span><span style=color:#000>writer</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#000>DataFileWriter</span><span style=color:#000;font-weight:700>(</span><span style=color:#204a87>open</span><span style=color:#000;font-weight:700>(</span><span style=color:#4e9a06>"users.avro"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"wb"</span><span style=color:#000;font-weight:700>),</span> <span style=color:#000>DatumWriter</span><span style=color:#000;font-weight:700>(),</span> <span style=color:#000>schema</span><span style=color:#000;font-weight:700>)</span> +</span></span><span style=display:flex><span><span style=color:#000>writer</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>append</span><span style=color:#000;font-weight:700>({</span><span style=color:#4e9a06>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"Alyssa"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"favorite_number"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#0000cf;font-weight:700>256</span><span style=color:#000;font-weight:700>})</span> +</span></span><span style=display:flex><span><span style=color:#000>writer</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>append</span><span style=color:#000;font-weight:700>({</span><span style=color:#4e9a06>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"Ben"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"favorite_number"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#0000cf;font-weight:700>7</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"favorite_color"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"red"</span><span style=color:#000;font-weight:700>})</span> +</span></span><span style=display:flex><span><span style=color:#000>writer</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>close</span><span style=color:#000;font-weight:700>()</span> +</span></span><span style=display:flex><span> +</span></span><span style=display:flex><span><span style=color:#000>reader</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#000>DataFileReader</span><span style=color:#000;font-weight:700>(</span><span style=color:#204a87>open</span><span style=color:#000;font-weight:700>(</span><span style=color:#4e9a06>"users.avro"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"rb"</span><span style=color:#000;font-weight:700>),</span> <span style=color:#000>DatumReader</span><span style=color:#000;font-weight:700>())</span> +</span></span><span style=display:flex><span><span style=color:#204a87;font-weight:700>for</span> <span style=color:#000>user</span> <span style=color:#204a87;font-weight:700>in</span> <span style=color:#000>reader</span><span style=color:#000;font-weight:700>:</span> +</span></span><span style=display:flex><span> <span style=color:#204a87>print</span><span style=color:#000;font-weight:700>(</span><span style=color:#000>user</span><span style=color:#000;font-weight:700>)</span> +</span></span><span style=display:flex><span><span style=color:#000>reader</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>close</span><span style=color:#000;font-weight:700>()</span> +</span></span></code></pre></div><p>This outputs:</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-json data-lang=json><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#a40000>u'favorite_color':</span> <span style=color:#a40000>None,</span> <span style=color:#a40000>u'favorite_number':</span> <span style=color:#a40000>256,</span> <span style=color:#a40000>u'name':</span> <span style=color:#a40000>u'Alyssa'</span><span style=color:#000;font-weight:700>}</span> +</span></span><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#a40000>u'favorite_color':</span> <span style=color:#a40000>u'red',</span> <span style=color:#a40000>u'favorite_number':</span> <span style=color:#a40000>7,</span> <span style=color:#a40000>u'name':</span> <span style=color:#a40000>u'Ben'</span><span style=color:#000;font-weight:700>}</span> +</span></span></code></pre></div><p>Do make sure that you open your files in binary mode (i.e. using the modes wb or rb respectively). Otherwise you might generate corrupt files due to automatic replacement of newline characters with the platform-specific representations.</p><p>Let’s take a closer look at what’s going on here.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span><span style=color:#000>schema</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#000>avro</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>schema</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>parse</span><span style=color:#000;font-weight:700>(</span><span style=color:#204a87>open</span><span style=color:#000;font-weight:700>(</span><span style=color:#4e9a06>"user.avsc" ;</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"rb"</span><span style=color:#000;font-weight:700>)</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>read</span><span style=color:#000;font-weight:700>())</span> +</span></span></code></pre></div><p>avro.schema.parse takes a string containing a JSON schema definition as input and outputs a avro.schema.Schema object (specifically a subclass of Schema, in this case RecordSchema). We’re passing in the contents of our user.avsc schema file here.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span><span style=color:#000>writer</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#000>DataFileWriter</span><span style=color:#000;font-weight:700>(</span><span style=color:#204a87>open</span><span style=color:#000;font-weight:700>(</span><span style=color:#4e9a06>"users.avro"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"wb"</span><span style=color:#000;font-weight:700>),</span> <span style=color:#000>DatumWriter</span><span style=color:#00 0;font-weight:700>(),</span> <span style=color:#000>schema</span><span style=color:#000;font-weight:700>)</span> +</span></span></code></pre></div><p>We create a DataFileWriter, which we’ll use to write serialized items to a data file on disk. The DataFileWriter constructor takes three arguments:</p><ul><li>The file we’ll serialize to</li><li>A DatumWriter, which is responsible for actually serializing the items to Avro’s binary format (DatumWriters can be used separately from DataFileWriters, e.g., to perform IPC with Avro).</li><li>The schema we’re using. The DataFileWriter needs the schema both to write the schema to the data file, and to verify that the items we write are valid items and write the appropriate fields.</li></ul><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span><span style=color:#000>writer</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>append</span><span style=color:#000;font-weight:700>( {</span><span style=color:#4e9a06>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"Alyssa"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"favorite_number"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#0000cf;font-weight:700>256</span><span style=color:#000;font-weight:700>})</span> +</span></span><span style=display:flex><span><span style=color:#000>writer</span><span style=color:#ce5c00;font-weight:700>.</span><span style=color:#000>append</span><span style=color:#000;font-weight:700>({</span><span style=color:#4e9a06>"name"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"Ben"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"favorite_number"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#0000cf;font-weight:700>7</span><span style=color:#000;font-weight:700>,</span> <span style=color:#4e9a06>"favorite_color"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"red"</span><span style=color:#000;font-weight:700>})</span> +</span></span></code></pre></div><p>We use DataFileWriter.append to add items to our data file. Avro records are represented as Python dicts. Since the field favorite_color has type [“string”, “null”], we are not required to specify this field, as shown in the first append. Were we to omit the required name field, an exception would be raised. Any extra entries not corresponding to a field are present in the dict are ignored.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span><span style=color:#000>reader</span> <span style=color:#ce5c00;font-weight:700>=</span> <span style=color:#000>DataFileReader</span><span style=color:#000;font-weight:700>(</span><span style=color:#204a87>open</span><span style=color:#000;font-weight:700>(</span><span style=color:#4e9a06>"users.avro"</span><span style=color:#000;font-weight:700> ,</span> <span style=color:#4e9a06>"rb"</span><span style=color:#000;font-weight:700>),</span> <span style=color:#000>DatumReader</span><span style=color:#000;font-weight:700>())</span> +</span></span></code></pre></div><p>We open the file again, this time for reading back from disk. We use a DataFileReader and DatumReader analagous to the DataFileWriter and DatumWriter above.</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span><span style=color:#204a87;font-weight:700>for</span> <span style=color:#000>user</span> <span style=color:#204a87;font-weight:700>in</span> <span style=color:#000>reader</span><span style=color:#000;font-weight:700>:</span> +</span></span><span style=display:flex><span> <span style=color:#204a87>print</span><span style=color:#000;font-weight:700>(</span><span style=color:#000>user</span><span style=color:#000;font-weight:700>)</span> +</span></span></code></pre></div><p>The DataFileReader is an iterator that returns dicts corresponding to the serialized items.</p></div><div class=td-content style=page-break-before:always><h1 id=pg-d9956c22c61ec53d9e37fab869629872>3 - Specification</h1><h2 id=introduction>Introduction</h2><p>This document defines Apache Avro. It is intended to be the authoritative specification. Implementations of Avro must adhere to this document.</p><h2 id=schema-declaration>Schema Declaration</h2><p>A Schema is represented in <a href=https://www.json.org/>JSON</a> by one of:</p><ul><li>A JSON string, naming a defined type.</li><li>A JSON object, of the form:</li></ul><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-js data-lang=js><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#4e9a06>"type"</span><span style=color:#ce5c00;font-weight:700>:</span> <span style =color:#4e9a06>"typeName"</span><span style=color:#000;font-weight:700>,</span> <span style=color:#000;font-weight:700>...</span><span style=color:#000>attributes</span><span style=color:#000;font-weight:700>...}</span> +</span></span></code></pre></div><p>where <em>typeName</em> is either a primitive or derived type name, as defined below. Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.</p><ul><li>A JSON array, representing a union of embedded types.</li></ul><h2 id=primitive-types>Primitive Types</h2><p>The set of primitive type names is:</p><ul><li><em>null</em>: no value</li><li><em>boolean</em>: a binary value</li><li><em>int</em>: 32-bit signed integer</li><li><em>long</em>: 64-bit signed integer</li><li><em>float</em>: single precision (32-bit) IEEE 754 floating-point number</li><li><em>double</em>: double precision (64-bit) IEEE 754 floating-point number</li><li><em>bytes</em>: sequence of 8-bit unsigned bytes</li><li><em>string</em>: unicode character sequence</li></ul><p>Primitive types have no specified attributes.</p><p>Primitive type names are also defined type names. Thus, for example, the schema “string” is equivalent to:</p><div class=highlight><pre tabindex=0 style=background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-json data-lang=json><span style=display:flex><span><span style=color:#000;font-weight:700>{</span><span style=color:#204a87;font-weight:700>"type"</span><span style=color:#000;font-weight:700>:</span> <span style=color:#4e9a06>"string"</span><span style=color:#000;font-weight:700>}</span> +</span></span></code></pre></div><h2 id=complex-types>Complex Types</h2><p>Avro supports six kinds of complex types: <em>records</em>, <em>enums</em>, <em>arrays</em>, <em>maps</em>, <em>unions</em> and <em>fixed</em>.</p><h3 id=schema-record>Records</h3><p>Records use the type name “record” and support the following attributes:</p><ul><li><em>name</em>: a JSON string providing the name of the record (required).</li><li><em>namespace</em>, a JSON string that qualifies the name (optional);</li><li><em>doc</em>: a JSON string providing documentation to the user of this schema (optional).</li><li><em>aliases</em>: a JSON array of strings, providing alternate names for this record (optional).</li><li><em>fields</em>: a JSON array, listing fields (required). Each field is a JSON object with the following attributes:<ul><li><em>name</em>: a JSON string providing the name of the field (required), and</li><li><em>doc</em>: a JSON string describing this field for users (optional) .</li><li><em>type</em>: a <a href=#schema-declaration title="Schema declaration">schema</a>, as defined above</li><li><em>order</em>: specifies how this field impacts sort ordering of this record (optional). Valid values are “ascending” (the default), “descending”, or “ignore”. For more details on how this is used, see the sort order section below.</li><li><em>aliases</em>: a JSON array of strings, providing alternate names for this field (optional).</li><li><em>default</em>: A default value for this field, only used when reading instances that lack the field for schema evolution purposes. The presence of a default value does not make the field optional at encoding time. Permitted values depend on the field’s schema type, according to the table below. Default values for union fields correspond to the first schema in the union. Default values for bytes and fixed fields are JSON strings, where Unicode code points 0-255 are mapped to unsigned 8-bit byte values 0-255. Avro encodes a field even if its value is equal to its default.</li></ul></li></ul><p><em>field default values</em></p><table><thead><tr><th><strong>avro type</strong></th><th><strong>json type</strong></th><th><strong>example</strong></th></tr></thead><tbody><tr><td>null</td><td>null</td><td><code>null</code></td></tr><tr><td>boolean</td><td>boolean</td><td><code>true</code></td></tr><tr><td>int,long</td><td>integer</td><td><code>1</code></td></tr><tr><td>float,double</td><td>number</td><td><code>1.1</code></td></tr><tr><td>bytes</td><td>string</td><td><code>"\u00FF"</code></td></tr><tr><td>string</td><td>string</td><td><code>"foo"</code></td></tr><tr><td>record</td><td>object</td><td><code>{"a": 1}</code></td></tr><tr><td>enum</td><td>string</td><td><code>"FOO"</code></td></tr><tr><td>array</td><td>array</td><td><code>[1]</code></td></tr><tr><td>map</td><td>object</td><td><code>{"a": 1}</code></td></tr><tr><td>fixed</td><td>string</td><td><code>"\u00ff"</c ode></td></tr></tbody></table><p>For example, a linked-list of 64-bit values may be defined with:</p><pre tabindex=0><code class=language-jsonc data-lang=jsonc>{ + "type": "record", + "name": "LongList", + "aliases": ["LinkedLongs"], // old name for this + "fields" : [ + {"name": "value", "type": "long"}, // each element has a long + {"name": "next", "type": ["null", "LongList"]} // optional next element + ] +}
[... 628 lines stripped ...]
