This is an automated email from the ASF dual-hosted git repository. jonwei pushed a commit to branch 20rc2_update in repository https://gitbox.apache.org/repos/asf/druid-website-src.git
commit bd206f9f00961d660fc43f6b38d98b6c9dbd261e Author: jon-wei <[email protected]> AuthorDate: Thu Oct 8 23:36:01 2020 -0700 0.20.0-rc2 updates --- docs/0.20.0/development/extensions-core/avro.html | 13 ++++++++++++- docs/0.20.0/ingestion/data-formats.html | 9 +++++++++ docs/latest/development/extensions-core/avro.html | 13 ++++++++++++- docs/latest/ingestion/data-formats.html | 9 +++++++++ 4 files changed, 42 insertions(+), 2 deletions(-) diff --git a/docs/0.20.0/development/extensions-core/avro.html b/docs/0.20.0/development/extensions-core/avro.html index b12a1e7..38a808e 100644 --- a/docs/0.20.0/development/extensions-core/avro.html +++ b/docs/0.20.0/development/extensions-core/avro.html @@ -82,8 +82,19 @@ two Avro Parsers for stream ingestion and Hadoop batch ingestion. See <a href="/docs/0.20.0/ingestion/data-formats.html#avro-hadoop-parser">Avro Hadoop Parser</a> and <a href="/docs/0.20.0/ingestion/data-formats.html#avro-stream-parser">Avro Stream Parser</a> for more details about how to use these in an ingestion spec.</p> +<p>Additionally, it provides an InputFormat for reading Avro OCF files when using +<a href="/docs/0.20.0/ingestion/native-batch.html">native batch indexing</a>, see <a href="/docs/0.20.0/ingestion/data-formats.html#avro-ocf">Avro OCF</a> +for details on how to ingest OCF files.</p> <p>Make sure to <a href="/docs/0.20.0/development/extensions.html#loading-extensions">include</a> <code>druid-avro-extensions</code> as an extension.</p> -</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/0.20.0/development/extensions-core/approximate-histograms.html"><span class="arrow-prev">← </span><span>Approximate Histogram aggregators</span></a><a class="docs-next button" href="/docs/0.20.0/development/extensions-core/azure.html"><span>Microsoft Azure</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#avro-extension" [...] +<h3><a class="anchor" aria-hidden="true" id="avro-types"></a><a href="#avro-types" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1 [...] +<p>Druid supports most Avro types natively, there are however some exceptions which are detailed here.</p> +<p><code>union</code> types which aren't of the form <code>[null, otherType]</code> aren't supported at this time.</p> +<p><code>bytes</code> and <code>fixed</code> Avro types will be returned by default as base64 encoded strings unless the <code>binaryAsString</code> option is enabled on the Avro parser. +This setting will decode these types as UTF-8 strings.</p> +<p><code>enum</code> types will be returned as <code>string</code> of the enum symbol.</p> +<p><code>record</code> and <code>map</code> types representing nested data can be ingested using <a href="/docs/0.20.0/ingestion/data-formats.html#flattenspec">flattenSpec</a> on the parser.</p> +<p>Druid doesn't currently support Avro logical types, they will be ignored and fields will be handled according to the underlying primitive type.</p> +</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/0.20.0/development/extensions-core/approximate-histograms.html"><span class="arrow-prev">← </span><span>Approximate Histogram aggregators</span></a><a class="docs-next button" href="/docs/0.20.0/development/extensions-core/azure.html"><span>Microsoft Azure</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#avro-extension" [...] document.addEventListener('keyup', function(e) { if (e.target !== document.body) { return; diff --git a/docs/0.20.0/ingestion/data-formats.html b/docs/0.20.0/ingestion/data-formats.html index 7846c68..9553c2b 100644 --- a/docs/0.20.0/ingestion/data-formats.html +++ b/docs/0.20.0/ingestion/data-formats.html @@ -262,6 +262,9 @@ please read <a href="/docs/0.20.0/development/extensions-core/orc.html#migration <blockquote> <p>You need to include the <a href="/docs/0.20.0/development/extensions-core/avro.html"><code>druid-avro-extensions</code></a> as an extension to use the Avro OCF input format.</p> </blockquote> +<blockquote> +<p>See the <a href="/docs/0.20.0/development/extensions-core/avro.html#avro-types">Avro Types</a> section for how Avro types are handled in Druid</p> +</blockquote> <p>The <code>inputFormat</code> to load data of Avro OCF format. An example is:</p> <pre><code class="hljs css language-json">"ioConfig": { "inputFormat": { @@ -383,6 +386,9 @@ Each line can be further parsed using <a href="#parsespec"><code>parseSpec</code <blockquote> <p>You need to include the <a href="/docs/0.20.0/development/extensions-core/avro.html"><code>druid-avro-extensions</code></a> as an extension to use the Avro Hadoop Parser.</p> </blockquote> +<blockquote> +<p>See the <a href="/docs/0.20.0/development/extensions-core/avro.html#avro-types">Avro Types</a> section for how Avro types are handled in Druid</p> +</blockquote> <p>This parser is for <a href="/docs/0.20.0/ingestion/hadoop.html">Hadoop batch ingestion</a>. The <code>inputFormat</code> of <code>inputSpec</code> in <code>ioConfig</code> must be set to <code>"org.apache.druid.data.input.avro.AvroValueInputFormat"</code>. You may want to set Avro reader's schema in <code>jobProperties</code> in <code>tuningConfig</code>, @@ -880,6 +886,9 @@ an explicitly defined <a href="http://www.joda.org/joda-time/apidocs/org/joda/ti <blockquote> <p>You need to include the <a href="/docs/0.20.0/development/extensions-core/avro.html"><code>druid-avro-extensions</code></a> as an extension to use the Avro Stream Parser.</p> </blockquote> +<blockquote> +<p>See the <a href="/docs/0.20.0/development/extensions-core/avro.html#avro-types">Avro Types</a> section for how Avro types are handled in Druid</p> +</blockquote> <p>This parser is for <a href="/docs/0.20.0/ingestion/index.html#streaming">stream ingestion</a> and reads Avro data from a stream directly.</p> <table> <thead> diff --git a/docs/latest/development/extensions-core/avro.html b/docs/latest/development/extensions-core/avro.html index 1b8a32b..32e689b 100644 --- a/docs/latest/development/extensions-core/avro.html +++ b/docs/latest/development/extensions-core/avro.html @@ -82,8 +82,19 @@ two Avro Parsers for stream ingestion and Hadoop batch ingestion. See <a href="/docs/latest/ingestion/data-formats.html#avro-hadoop-parser">Avro Hadoop Parser</a> and <a href="/docs/latest/ingestion/data-formats.html#avro-stream-parser">Avro Stream Parser</a> for more details about how to use these in an ingestion spec.</p> +<p>Additionally, it provides an InputFormat for reading Avro OCF files when using +<a href="/docs/latest/ingestion/native-batch.html">native batch indexing</a>, see <a href="/docs/latest/ingestion/data-formats.html#avro-ocf">Avro OCF</a> +for details on how to ingest OCF files.</p> <p>Make sure to <a href="/docs/latest/development/extensions.html#loading-extensions">include</a> <code>druid-avro-extensions</code> as an extension.</p> -</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/latest/development/extensions-core/approximate-histograms.html"><span class="arrow-prev">← </span><span>Approximate Histogram aggregators</span></a><a class="docs-next button" href="/docs/latest/development/extensions-core/azure.html"><span>Microsoft Azure</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#avro-extension" [...] +<h3><a class="anchor" aria-hidden="true" id="avro-types"></a><a href="#avro-types" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1 [...] +<p>Druid supports most Avro types natively, there are however some exceptions which are detailed here.</p> +<p><code>union</code> types which aren't of the form <code>[null, otherType]</code> aren't supported at this time.</p> +<p><code>bytes</code> and <code>fixed</code> Avro types will be returned by default as base64 encoded strings unless the <code>binaryAsString</code> option is enabled on the Avro parser. +This setting will decode these types as UTF-8 strings.</p> +<p><code>enum</code> types will be returned as <code>string</code> of the enum symbol.</p> +<p><code>record</code> and <code>map</code> types representing nested data can be ingested using <a href="/docs/latest/ingestion/data-formats.html#flattenspec">flattenSpec</a> on the parser.</p> +<p>Druid doesn't currently support Avro logical types, they will be ignored and fields will be handled according to the underlying primitive type.</p> +</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/latest/development/extensions-core/approximate-histograms.html"><span class="arrow-prev">← </span><span>Approximate Histogram aggregators</span></a><a class="docs-next button" href="/docs/latest/development/extensions-core/azure.html"><span>Microsoft Azure</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#avro-extension" [...] document.addEventListener('keyup', function(e) { if (e.target !== document.body) { return; diff --git a/docs/latest/ingestion/data-formats.html b/docs/latest/ingestion/data-formats.html index efe7249..192e9c0 100644 --- a/docs/latest/ingestion/data-formats.html +++ b/docs/latest/ingestion/data-formats.html @@ -262,6 +262,9 @@ please read <a href="/docs/latest/development/extensions-core/orc.html#migration <blockquote> <p>You need to include the <a href="/docs/latest/development/extensions-core/avro.html"><code>druid-avro-extensions</code></a> as an extension to use the Avro OCF input format.</p> </blockquote> +<blockquote> +<p>See the <a href="/docs/latest/development/extensions-core/avro.html#avro-types">Avro Types</a> section for how Avro types are handled in Druid</p> +</blockquote> <p>The <code>inputFormat</code> to load data of Avro OCF format. An example is:</p> <pre><code class="hljs css language-json">"ioConfig": { "inputFormat": { @@ -383,6 +386,9 @@ Each line can be further parsed using <a href="#parsespec"><code>parseSpec</code <blockquote> <p>You need to include the <a href="/docs/latest/development/extensions-core/avro.html"><code>druid-avro-extensions</code></a> as an extension to use the Avro Hadoop Parser.</p> </blockquote> +<blockquote> +<p>See the <a href="/docs/latest/development/extensions-core/avro.html#avro-types">Avro Types</a> section for how Avro types are handled in Druid</p> +</blockquote> <p>This parser is for <a href="/docs/latest/ingestion/hadoop.html">Hadoop batch ingestion</a>. The <code>inputFormat</code> of <code>inputSpec</code> in <code>ioConfig</code> must be set to <code>"org.apache.druid.data.input.avro.AvroValueInputFormat"</code>. You may want to set Avro reader's schema in <code>jobProperties</code> in <code>tuningConfig</code>, @@ -880,6 +886,9 @@ an explicitly defined <a href="http://www.joda.org/joda-time/apidocs/org/joda/ti <blockquote> <p>You need to include the <a href="/docs/latest/development/extensions-core/avro.html"><code>druid-avro-extensions</code></a> as an extension to use the Avro Stream Parser.</p> </blockquote> +<blockquote> +<p>See the <a href="/docs/latest/development/extensions-core/avro.html#avro-types">Avro Types</a> section for how Avro types are handled in Druid</p> +</blockquote> <p>This parser is for <a href="/docs/latest/ingestion/index.html#streaming">stream ingestion</a> and reads Avro data from a stream directly.</p> <table> <thead> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
