Added: avro/site/publish/docs/1.11.3/specification/index.xml URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/specification/index.xml?rev=1919887&view=auto ============================================================================== --- avro/site/publish/docs/1.11.3/specification/index.xml (added) +++ avro/site/publish/docs/1.11.3/specification/index.xml Wed Aug 14 13:30:06 2024 @@ -0,0 +1 @@ +<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom";><channel><title>Apache Avro â Specification</title><link>/docs/1.11.3/specification/</link><description>Recent content in Specification on Apache Avro</description><generator>Hugo -- gohugo.io</generator><language>en</language><lastBuildDate>Mon, 25 Oct 2021 00:00:00 +0000</lastBuildDate><atom:link href="/docs/1.11.3/specification/index.xml" rel="self" type="application/rss+xml"/></channel></rss> \ No newline at end of file
Added: avro/site/publish/docs/1.11.3/trevni/css/maven-base.css URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/css/maven-base.css?rev=1919887&view=auto ============================================================================== --- avro/site/publish/docs/1.11.3/trevni/css/maven-base.css (added) +++ avro/site/publish/docs/1.11.3/trevni/css/maven-base.css Wed Aug 14 13:30:06 2024 @@ -0,0 +1,156 @@ +body { + margin: 0px; + padding: 0px; +} +img { + border:none; +} +table { + padding:0px; + width: 100%; + margin-left: -2px; + margin-right: -2px; +} +acronym { + cursor: help; + border-bottom: 1px dotted #feb; +} +table.bodyTable th, table.bodyTable td { + padding: 2px 4px 2px 4px; + vertical-align: top; +} +div.clear{ + clear:both; + visibility: hidden; +} +div.clear hr{ + display: none; +} +#bannerLeft, #bannerRight { + font-size: xx-large; + font-weight: bold; +} +#bannerLeft img, #bannerRight img { + margin: 0px; +} +.xleft, #bannerLeft img { + float:left; +} +.xright, #bannerRight { + float:right; +} +#banner { + padding: 0px; +} +#banner img { + border: none; +} +#breadcrumbs { + padding: 3px 10px 3px 10px; +} +#leftColumn { + width: 170px; + float:left; + overflow: auto; + padding-bottom: 3px; /* IE-9 scrollbar-fix */ +} +#bodyColumn { + margin-right: 1.5em; + margin-left: 197px; +} +#legend { + padding: 8px 0 8px 0; +} +#navcolumn { + padding: 8px 4px 0 8px; +} +#navcolumn h5 { + margin: 0; + padding: 0; + font-size: small; +} +#navcolumn ul { + margin: 0; + padding: 0; + font-size: small; +} +#navcolumn li { + list-style-type: none; + background-image: none; + background-repeat: no-repeat; + background-position: 0 0.4em; + padding-left: 16px; + list-style-position: outside; + line-height: 1.2em; + font-size: smaller; +} +#navcolumn li.expanded { + background-image: url(../images/expanded.gif); +} +#navcolumn li.collapsed { + background-image: url(../images/collapsed.gif); +} +#navcolumn li.none { + text-indent: -1em; + margin-left: 1em; +} +#poweredBy { + text-align: center; +} +#navcolumn img { + margin-top: 10px; + margin-bottom: 3px; +} +#poweredBy img { + display:block; + margin: 20px 0 20px 17px; +} +#search img { + margin: 0px; + display: block; +} +#search #q, #search #btnG { + border: 1px solid #999; + margin-bottom:10px; +} +#search form { + margin: 0px; +} +#lastPublished { + font-size: x-small; +} +.navSection { + margin-bottom: 2px; + padding: 8px; +} +.navSectionHead { + font-weight: bold; + font-size: x-small; +} +.section { + padding: 4px; +} +#footer { + padding: 3px 10px 3px 10px; + font-size: x-small; +} +#breadcrumbs { + font-size: x-small; + margin: 0pt; +} +.source { + padding: 12px; + margin: 1em 7px 1em 7px; +} +.source pre { + margin: 0px; + padding: 0px; +} +#navcolumn img.imageLink, .imageLink{ + padding-left: 0px; + padding-bottom: 0px; + padding-top: 0px; + padding-right: 2px; + border: 0px; + margin: 0px; +} Added: avro/site/publish/docs/1.11.3/trevni/css/maven-theme.css URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/css/maven-theme.css?rev=1919887&view=auto ============================================================================== --- avro/site/publish/docs/1.11.3/trevni/css/maven-theme.css (added) +++ avro/site/publish/docs/1.11.3/trevni/css/maven-theme.css Wed Aug 14 13:30:06 2024 @@ -0,0 +1,217 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +body { + background-color: #fff; + font-family: Verdana, Helvetica, Arial, sans-serif; + margin-left: auto; + margin-right: auto; + background-repeat: repeat-y; + font-size: 13px; + padding: 0px; +} +td, select, input, li{ + font-family: Verdana, Helvetica, Arial, sans-serif; + font-size: 12px; + color:#333333; +} +code{ + font-size: 12px; +} +a { + text-decoration: none; +} +a:link { + color:#47a; +} +a:visited { + color:#68a; +} +a:active, a:hover { + color:#990000; +} +#legend li.externalLink { + background: url(../images/external.png) left top no-repeat; + padding-left: 18px; +} +a.externalLink, a.externalLink:link, a.externalLink:visited, a.externalLink:active, a.externalLink:hover { + background: url(../images/external.png) right center no-repeat; + padding-right: 18px; +} +#legend li.newWindow { + background: url(../images/newwindow.png) left top no-repeat; + padding-left: 18px; +} +a.newWindow, a.newWindow:link, a.newWindow:visited, a.newWindow:active, a.newWindow:hover { + background: url(../images/newwindow.png) right center no-repeat; + padding-right: 18px; +} +h2 { + font-size: 17px; + color: #333333; +} +h3 { + padding: 4px 4px 4px 24px; + color: #666; + background-color: #ccc; + font-weight: bold; + font-size: 14px; + background-image: url(../images/h3.jpg); + background-repeat: no-repeat; + background-position: left bottom; +} +p { + line-height: 1.3em; + font-size: 12px; + color: #000; +} +#breadcrumbs { + height: 13px; + background-image: url(../images/breadcrumbs.jpg); + padding: 5px 10px 14px 20px; +} +* html #breadcrumbs { + padding-bottom: 8px; +} +#leftColumn { + margin: 10px 0 10px 0; + border-top-color: #ccc; + border-top-style: solid; + border-top-width: 1px; + border-right-color: #ccc; + border-right-style: solid; + border-right-width: 1px; + border-bottom-color: #ccc; + border-bottom-style: solid; + border-bottom-width: 1px; + padding-right: 5px; + padding-left: 5px; +} +#navcolumn h5 { + font-size: smaller; + border-bottom: 1px solid #aaaaaa; + padding-top: 2px; + padding-left: 9px; + color: #49635a; + background-image: url(../images/h5.jpg); + background-repeat: no-repeat; + background-position: left bottom; +} + +table.bodyTable th { + color: white; + background-color: #bbb; + text-align: left; + font-weight: bold; +} + +table.bodyTable th, table.bodyTable td { + font-size: 11px; +} + +table.bodyTable tr.a { + background-color: #ddd; +} + +table.bodyTable tr.b { + background-color: #eee; +} + +table.bodyTable tr.a:hover, table.bodyTable tr.b:hover { + background: #dfe3db; +} + +.source { + border: 1px solid #999; + overflow:auto +} +dt { + padding: 4px 4px 4px 24px; + color: #333333; + background-color: #ccc; + font-weight: bold; + font-size: 14px; + background-image: url(../images/h3.jpg); + background-repeat: no-repeat; + background-position: left bottom; +} +.subsectionTitle { + font-size: 13px; + font-weight: bold; + color: #666; + +} + +table { + font-size: 10px; +} +.xright a:link, .xright a:visited, .xright a:active { + color: #666; +} +.xright a:hover { + color: #003300; +} +#banner { + height: 93px; + background: url(../images/banner.jpg); +} +#navcolumn ul { + margin: 5px 0 15px -0em; +} +#navcolumn ul a { + color: #333333; +} +#navcolumn ul a:hover { + color: red; +} +#intro { + border: solid #ccc 1px; + margin: 6px 0px 0px 0px; + padding: 10px 40px 10px 40px; +} +.subsection { + margin-left: 3px; + color: #333333; +} + +.subsection p { + font-size: 12px; +} +#footer { + padding: 10px; + margin: 20px 0px 20px 0px; + border-top: solid #ccc 1px; + color: #333333; +} + +.errormark, .warningmark, .donemark, .infomark { + background: url(../images/icon_error_sml.gif) no-repeat; +} + +.warningmark { + background-image: url(../images/icon_warning_sml.gif); +} + +.donemark { + background-image: url(../images/icon_success_sml.gif); +} + +.infomark { + background-image: url(../images/icon_info_sml.gif); +} Added: avro/site/publish/docs/1.11.3/trevni/css/print.css URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/css/print.css?rev=1919887&view=auto ============================================================================== --- avro/site/publish/docs/1.11.3/trevni/css/print.css (added) +++ avro/site/publish/docs/1.11.3/trevni/css/print.css Wed Aug 14 13:30:06 2024 @@ -0,0 +1,7 @@ +#banner, #footer, #leftcol, #breadcrumbs, .docs #toc, .docs .courtesylinks, #leftColumn, #navColumn { + display: none !important; +} +#bodyColumn, body.docs div.docs { + margin: 0 !important; + border: none !important +} Added: avro/site/publish/docs/1.11.3/trevni/css/site.css URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/css/site.css?rev=1919887&view=auto ============================================================================== --- avro/site/publish/docs/1.11.3/trevni/css/site.css (added) +++ avro/site/publish/docs/1.11.3/trevni/css/site.css Wed Aug 14 13:30:06 2024 @@ -0,0 +1,31 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to You under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* https://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +#banner { + height: 93px; + background: none; +} + +#bannerLeft img { + height: 90px; + margin-left: 30px; + margin-top: 4px; +} + +#bannerRight img { + margin: 17px; +} + Added: avro/site/publish/docs/1.11.3/trevni/images/apache-maven-project-2.png URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/apache-maven-project-2.png?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/apache-maven-project-2.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/banner.jpg URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/banner.jpg?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/banner.jpg ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/bg.jpg URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/bg.jpg?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/bg.jpg ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/breadcrumbs.jpg URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/breadcrumbs.jpg?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/breadcrumbs.jpg ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/collapsed.gif URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/collapsed.gif?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/collapsed.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/expanded.gif URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/expanded.gif?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/expanded.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/external.png URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/external.png?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/external.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/h3.jpg URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/h3.jpg?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/h3.jpg ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/h5.jpg URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/h5.jpg?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/h5.jpg ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/icon_error_sml.gif URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/icon_error_sml.gif?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/icon_error_sml.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/icon_info_sml.gif URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/icon_info_sml.gif?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/icon_info_sml.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/icon_success_sml.gif URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/icon_success_sml.gif?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/icon_success_sml.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/icon_warning_sml.gif URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/icon_warning_sml.gif?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/icon_warning_sml.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/logo_apache.jpg URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/logo_apache.jpg?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/logo_apache.jpg ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/logo_maven.jpg URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/logo_maven.jpg?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/logo_maven.jpg ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/logos/build-by-maven-black.png URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/logos/build-by-maven-black.png?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/logos/build-by-maven-black.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/logos/build-by-maven-white.png URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/logos/build-by-maven-white.png?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/logos/build-by-maven-white.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/logos/maven-feather.png URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/logos/maven-feather.png?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/logos/maven-feather.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/maven-logo-2.gif URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/maven-logo-2.gif?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/maven-logo-2.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/images/newwindow.png URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/images/newwindow.png?rev=1919887&view=auto ============================================================================== Binary file - no diff available. Propchange: avro/site/publish/docs/1.11.3/trevni/images/newwindow.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: avro/site/publish/docs/1.11.3/trevni/spec.html URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.11.3/trevni/spec.html?rev=1919887&view=auto ============================================================================== --- avro/site/publish/docs/1.11.3/trevni/spec.html (added) +++ avro/site/publish/docs/1.11.3/trevni/spec.html Wed Aug 14 13:30:06 2024 @@ -0,0 +1,277 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> +<!-- + | Generated by Apache Maven Doxia at 2023-09-15 + | Rendered using Apache Maven Stylus Skin 1.5 +--> +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <title>Trevni Specification – Trevni: A Column File Format</title> + <style type="text/css" media="all"> + @import url("./css/maven-base.css"); + @import url("./css/maven-theme.css"); + @import url("./css/site.css"); + </style> + <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" /> + <meta name="Date-Revision-yyyymmdd" content="20230915" /> + <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> + </head> + <body class="composite"> + <div id="banner"> + <span id="bannerLeft"> + Trevni Specification + </span> + <div class="clear"> + <hr/> + </div> + </div> + <div id="breadcrumbs"> + + <div class="xleft"> + Last Published: 2023-09-15 + | Version: 1.11.3 + </div> + <div class="xright"> + </div> + <div class="clear"> + <hr/> + </div> + </div> + <div id="leftColumn"> + <div id="navcolumn"> + + <h5>Trevni</h5> + <ul> + <li class="none"> + <strong>Spec</strong> + </li> + </ul> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img alt="Built by Maven" src="./images/logos/maven-feather.png"/> + </a> + + </div> + </div> + <div id="bodyColumn"> + <div id="contentBox"> + <section> +<h2><a name="Trevni:_A_Column_File_Format"></a>Trevni: A Column File Format</h2> +<p>Version 0.1</p> +<p>DRAFT</p> +<p>This document is the authoritative specification of a file format. Its intent is to permit compatible, independent implementations that read and/or write files in this format.</p></section><section> +<h2><a name="Introduction"></a>Introduction</h2> +<p>Data sets are often described as a <i>table</i> composed of <i>rows</i> and <i>columns</i>. Each record in the dataset is considered a row, with each field of the record occupying a different column. Writing records to a file one-by-one as they are created results in a <i>row-major</i> format, like Hadoop’s SequenceFile or Avro data files.</p> +<p>In many cases higher query performance may be achieved if the data is instead organized in a <i>column-major</i> format, where multiple values of a given column are stored adjacently. This document defines such a column-major file format for datasets.</p> +<p>To permit scalable, distributed query evaluation, datasets are partitioned into row groups, containing distinct collections of rows. Each row group is organized in column-major order, while row groups form a row-major partitioning of the entire dataset.</p></section><section> +<h2><a name="Rationale"></a>Rationale</h2><section> +<h3><a name="Goals"></a>Goals</h3> +<p>The format is meant satisfy the following goals:</p> +<ol style="list-style-type: decimal"> +<li>Maximize the size of row groups. Disc drives are used most efficiently when sequentially accessing data. Consider a drive that takes 10ms to seek and transfers at 100MB/second. If a 10-column dataset whose values are all the same size is split into 10MB row groups, then accessing a single column will require a sequence of seek+1MB reads, for a cost of 20ms/MB processed. If the same dataset is split into 100MB row groups then this drops to 11ms/MB processed. This effect is exaggerated for datasets with larger numbers of columns and with columns whose values are smaller than average. So we’d prefer row groups that are 100MB or greater.</li> +<li>Permit random access within a row group. Some queries will first examine one column, and, only when certain relatively rare criteria are met, examine other columns. Rather than iterating through selected columns of the row-group in parallel, one might iterate through one column and randomly access another. This is called support for WHERE clauses, after the SQL operator of that name.</li> +<li>Minimize the number of files per dataset. HDFS is a primary intended deployment platform for these files. The HDFS Namenode requires memory for each file in the filesystem, thus for a format to be HDFS-friendly it should strive to require the minimum number of distinct files.</li> +<li>Support co-location of columns within row-groups. Row groups are the unit of parallel operation on a column dataset. For efficient file i/o, the entirety of a row-group should ideally reside on the host that is evaluating the query in order to avoid network latencies and bottlenecks.</li> +<li>Data integrity. The format should permit applications to detect data corruption. Many file systems may prevent corruption, but files may be moved between filesystems and be subject to corruption at points in that process. It is best if the data in a file can be validated independently.</li> +<li>Extensibility. The format should permit applications to store additional annotations about a datasets in the files, such as type information, origin, etc. Some environments may have metadata stores for such information, but not all do, and files might be moved among systems with different metadata systems. The ability to keep such information within the file simplifies the coordination of such information.</li> +<li>Minimal overhead. The column format should not make datasets appreciably larger. Storage is a primary cost and a choice to use this format should not require additional storage.</li> +<li>Primary format. The column format should be usable as a primary format for datasets, not as an auxiliary, accelerated format. Applications that process a dataset in row-major order should be able to easily consume column files and applications that produce datasets in row-major order should be able to easily generate column files.</li></ol></section><section> +<h3><a name="Design"></a>Design</h3> +<p>To meet these goals we propose the following design.</p> +<ol style="list-style-type: decimal"> +<li>Each row group is a separate file. All values of a column in a file are written contiguously. This maximizes the row group size, optimizing performance when querying few and small columns.</li> +<li>Each file occupies a single HDFS block. A larger than normal block size may be specified, e.g., ~1GB instead of the typical ~100MB. This guarantees co-location and eliminates network use when query processing can be co-located with the file. This also moderates the memory impact on the HDFS Namenode since no small files are written.</li> +<li>Each column in a file is written as a sequence of ~64kB compressed blocks. The sequence is prefixed by a table describing all of the blocks in the column to permit random access within the column.</li> +<li>Application-specific metadata may be added at the file, column, and block levels.</li> +<li>Checksums are included with each block, providing data integrity.</li></ol></section><section> +<h3><a name="Discussion"></a>Discussion</h3> +<p>The use of a single block per file achieves the same effect as the custom block placement policy described in the <a href="#CIF">CIF</a> paper, but while still permitting HDFS rebalancing and not increasing the number of files in the namespace.</p></section></section><section> +<h2><a name="Format_Specification"></a>Format Specification</h2> +<p>This section formally describes the proposed column file format.</p><section> +<h3><a name="Data_Model"></a>Data Model</h3> +<p>We assume a simple data model, where a record is a set of named fields, and the value of each field is a sequence of untyped bytes. A type system may be layered on top of this, as specified in the Type Mapping section below.</p></section><section> +<h3><a name="Primitive_Values"></a>Primitive Values</h3> +<p>We define the following primitive value types:</p> +<ul> +<li>Signed 64-bit <b>long</b> values are written using a variable-length zig-zag coding, where the high-order bit in each byte determines whether subsequent bytes are present. For example: +<table border="0" class="bodyTable"> +<tr class="a"> +<td align="center">decimal value</td> +<td align="center">hex bytes</td></tr> +<tr class="b"> +<td align="center">0</td> +<td align="center">00</td></tr> +<tr class="a"> +<td align="center">-1</td> +<td align="center">01</td></tr> +<tr class="b"> +<td align="center">1</td> +<td align="center">02</td></tr> +<tr class="a"> +<td align="center">...</td> +<td align="center"></td></tr> +<tr class="b"> +<td align="center">-64</td> +<td align="center">7f</td></tr> +<tr class="a"> +<td align="center">64</td> +<td align="center">80 01</td></tr> +<tr class="b"> +<td align="center">...</td> +<td align="center"></td></tr></table></li> +<li><b>bytes</b> are encoded as a <i>long</i> followed by that many bytes of data.</li> +<li>a <b>string</b> is encoded as a <i>long</i> followed by that many bytes of UTF-8 encoded character data. +<p>For example, the three-character string "foo" would be encoded as the <i>long</i> value 3 (encoded as hex 06) followed by the UTF-8 encoding of 'f', 'o', and 'o' (the hex bytes 66 6f 6f): 06 66 6f 6f</p></li></ul></section><section> +<h3><a name="Type_Names"></a>Type Names</h3> +<p>The following type names are used to describe column values:</p> +<ul> +<li><b>null</b>, requires zero bytes. Sometimes used in array columns.</li> +<li><b>boolean</b>, one bit, packed into bytes, little-endian;</li> +<li><b>int</b>, like <i>long</i>, but restricted to 32-bit signed values</li> +<li><b>long</b> 64-bit signed values, represented as above</li> +<li><b>fixed32</b> 32-bit values stored as four bytes, little-endian.</li> +<li><b>fixed64</b> 64-bit values stored as eight bytes, little-endian.</li> +<li><b>float</b> 32-bit IEEE floating point value, little-endian</li> +<li><b>double</b> 64-bit IEEE floating point value, little-endian</li> +<li><b>string</b> as above</li> +<li><b>bytes</b> as above, may be used to encapsulate more complex objects</li></ul> +<p>Type names are represented as <i>strings</i> (UTF-8 encoded, length-prefixed).</p></section><section> +<h3><a name="Metadata"></a>Metadata</h3> +<p><b>Metadata</b> consists of:</p> +<ul> +<li>A <i>long</i> indicating the number of metadata key/value pairs.</li> +<li>For each pair, a <i>string</i> key and <i>bytes</i> value.</li></ul> +<p>All metadata properties that start with "trevni." are reserved.</p><section> +<h4><a name="File_Metadata"></a>File Metadata</h4> +<p>The following file metadata properties are defined:</p> +<ul> +<li><b>trevni.codec</b> the name of the default compression codec used to compress blocks, as a <i>string</i>. Implementations are required to support the "null" codec. Optional. If absent, it is assumed to be "null". Codecs are described in more detail below.</li> +<li><b>trevni.checksum</b> the name of the checksum algorithm used in this file, as a <i>string</i>. Implementations are required to support the "crc-32” checksum. Optional. If absent, it is assumed to be "null". Checksums are described in more detail below.</li></ul></section><section> +<h4><a name="Column_Metadata"></a>Column Metadata</h4> +<p>The following column metadata properties are defined:</p> +<ul> +<li><b>trevni.codec</b> the name of the compression codec used to compress the blocks of this column, as a <i>string</i>. Implementations are required to support the "null" codec. Optional. If absent, it is assumed to be "null". Codecs are described in more detail below.</li> +<li><b>trevni.name</b> the name of the column, as a <i>string</i>. Required.</li> +<li><b>trevni.type</b> the type of data in the column. One of the type names above. Required.</li> +<li><b>trevni.values</b> if present, indicates that the initial value of each block in this column will be stored in the block’s descriptor. Not permitted for array columns or columns that specify a parent.</li> +<li><b>trevni.array</b> if present, indicates that each row in this column contains a sequence of values of the named type rather than just a single value. An integer length precedes each sequence of values indicating the count of values in the sequence. If the length is negative then it indicates a sequence of zero or one lengths, where -1 indicates two zeros, -2 two ones, -3 three zeros, -4 three ones, etc.</li> +<li><b>trevni.parent</b> if present, the name of an <i>array</i> column whose lengths are also used by this column. Thus values of this column are sequences but no lengths are stored in this column.</li></ul> +<p>For example, consider the following row, as JSON, where all values are primitive types, but one has multiple values.</p> +<div> +<pre>{"id"=566, "date"=23423234234 + "from"="[email protected]", + "to"=["[email protected]", "[email protected]"], + "content"="Hi!"}</pre></div> +<p>The columns for this might be specified as:</p> +<div> +<pre>name=id type=int +name=date type=long +name=from type=string +name=to type=string array=true +name=content type=string </pre></div> +<p>If a row contains an array of records, e.g. "received" in the following:</p> +<div> +<pre>{"id"=566, "date"=23423234234 + "from"="[email protected]", + "to"=["[email protected]", "[email protected]"], + "content"="Hi!" + "received"=[{"date"=234234234234, "host"="192.168.0.0.1"}, + {"date"=234234545645, "host"="192.168.0.0.2"}] +}</pre></div> +<p>Then one can define a parent column followed by a column for each field in the record, adding the following columns:</p> +<div> +<pre>name=received type=null array=true +name=date type=long parent=received +name=host type=string parent=received</pre></div> +<p>If an array value itself contains an array, e.g. the "sigs" below:</p> +<div> +<pre>{"id"=566, "date"=23423234234 + "from"="[email protected]", + "to"=["[email protected]", "[email protected]"], + "content"="Hi!" + "received"=[{"date"=234234234234, "host"="192.168.0.0.1", + "sigs"=[{"algo"="weak", "value"="0af345de"}]}, + {"date"=234234545645, "host"="192.168.0.0.2", + "sigs"=[]}] +}</pre></div> +<p>Then a parent column may be defined that itself has a parent column.</p> +<div> +<pre>name=sigs type=null array=true parent=received +name=algo type=string parent=sigs +name=value type=string parent=sigs</pre></div></section><section> +<h4><a name="Block_Metadata"></a>Block Metadata</h4> +<p>No block metadata properties are currently defined.</p></section></section><section> +<h3><a name="File_Format"></a>File Format</h3> +<p>A <b>file</b> consists of:</p> +<ul> +<li>A <i>file header</i>, followed by</li> +<li>one or more <i>columns</i>.</li></ul> +<p>A <b>file header</b> consists of:</p> +<ul> +<li>Four bytes, ASCII 'T', 'r', 'v', followed by 0x02.</li> +<li>a <i>fixed64</i> indicating the number of rows in the file</li> +<li>a <i>fixed32</i> indicating the number of columns in the file</li> +<li>file <i>metadata</i>.</li> +<li>for each column, its <i>column metadata</i></li> +<li>for each column, its starting position in the file as a <i>fixed64</i>.</li></ul> +<p>A <b>column</b> consists of:</p> +<ul> +<li>A <i>fixed32</i> indicating the number of blocks in this column.</li> +<li>For each block, a <i>block descriptor</i></li> +<li>One or more <i>blocks</i>.</li></ul> +<p>A <b>block descriptor</b> consists of:</p> +<ul> +<li>A <i>fixed32</i> indicating the number of rows in the block</li> +<li>A <i>fixed32</i> indicating the size in bytes of the block before the codec is applied (excluding checksum).</li> +<li>A <i>fixed32</i> indicating the size in bytes of the block after the codec is applied (excluding checksum).</li> +<li>If this column’s metadata declares it to include values, the first value in the column, serialized according to this column's type.</li></ul> +<p>A <b>block</b> consists of:</p> +<ul> +<li>The serialized column values. If a column is an array column then value sequences are preceded by their length, as an <i>int</i>. If a codec is specified, the values and lengths are compressed by that codec.</li> +<li>The checksum, as determined by the file metadata.</li></ul></section><section> +<h3><a name="Codecs"></a>Codecs</h3> +<dl> +<dt>null</dt> +<dd>The "null" codec simply passes data through uncompressed.</dd> +<dt>deflate</dt> +<dd>The "deflate" codec writes the data block using the deflate algorithm as specified in RFC 1951.</dd> +<dt>snappy</dt> +<dd>The "snappy" codec uses Google's Snappy compression library.</dd></dl></section><section> +<h3><a name="Checksum_algorithms"></a>Checksum algorithms</h3> +<dl> +<dt>null</dt> +<dd>The "null" checksum contains zero bytes.</dd> +<dt>crc-32</dt> +<dd>Each "crc-32" checksum contains the four bytes of an ISO 3309 CRC-32 checksum of the uncompressed block data as a fixed32.</dd></dl></section><section> +<h3><a name="Type_Mappings"></a>Type Mappings</h3> +<p>We define a standard mapping for how types defined in various serialization systems are represented in a column file. Records from these systems are <i>shredded</i> into columns. When records are nested, a depth-first recursive walk can assign a separate column for each primitive value.</p><section> +<h4><a name="Avro"></a>Avro</h4></section><section> +<h4><a name="Protocol_Buffers"></a>Protocol Buffers</h4></section><section> +<h4><a name="Thrift"></a>Thrift</h4></section></section></section><section> +<h2><a name="Implementation_Notes"></a>Implementation Notes</h2> +<p>Some possible techniques for writing column files include:</p> +<ol style="list-style-type: decimal"> +<li>Use a standard ~100MB block, buffer in memory up to the block size, then flush the file directly to HDFS. A single reduce task might create multiple output files. The namenode requires memory proportional to the number of names and blocks*replication. This would increase the number of names but not blocks, so this should still be much better than a file per column.</li> +<li>Spill each column to a separate local, temporary file then, when the file is closed, append these files, writing a single file to HDFS whose block size is set to be that of the entire file. This would be a bit slower than and may have trouble when the local disk is full, but it would better use HDFS namespace and further reduce seeks when processing columns whose values are small.</li> +<li>Use a separate mapreduce job to convert row-major files to column-major. The map output would output a by (row#, column#, value) tuple, partitioned by row# but sorted by column# then row#. The reducer could directly write the column file. But the column file format would need to be changed to write counts, descriptors, etc. at the end of files rather than at the front.</li></ol> +<p>(1) is the simplest to implement and most implementations should start with it.</p><section> +<h3><a name="References"></a>References</h3> +<p><a name="CIF">CIF</a> <a class="externalLink" href="https://arxiv.org/pdf/1105.4252.pdf";><i>Column-Oriented Storage Techniques for MapReduce</i></a>, Floratou, Patel, Shekita, & Tata, VLDB 2011.</p> +<p><a name="DREMEL">DREMEL</a> <a class="externalLink" href="https://research.google.com/pubs/archive/36632.pdf";><i>Dremel: Interactive Analysis of Web-Scale Datasets</i></a>, Melnik, Gubarev, Long, Romer, Shivakumar, & Tolton, VLDB 2010.</p></section></section> + </div> + </div> + <div class="clear"> + <hr/> + </div> + <div id="footer"> + <div class="xright"> + © 2009-2023 + The Apache Software Foundation + + - <a href="http://maven.apache.org/privacy-policy.html";>Privacy Policy</a>. + Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation. + </div> + <div class="clear"> + <hr/> + </div> + </div> + </body> +</html>
