[jira] [Commented] (DRILL-5913) DrillReduceAggregatesRule mixed the same functions of the same inputRef which have different dataTypes
[
https://issues.apache.org/jira/browse/DRILL-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249111#comment-16249111
]
ASF GitHub Bot commented on DRILL-5913:
---
Github user weijietong commented on the issue:
https://github.com/apache/drill/pull/1016
@amansinha100 maybe you are familiar with this part of codes . Could you
give a review ? anyone else will also be welcome.
> DrillReduceAggregatesRule mixed the same functions of the same inputRef which
> have different dataTypes
> ---
>
> Key: DRILL-5913
> URL: https://issues.apache.org/jira/browse/DRILL-5913
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> sample query:
> {code:java}
> select stddev_samp(cast(employee_id as int)) as col1, sum(cast(employee_id as
> int)) as col2 from cp.`employee.json`
> {code}
> error info:
> {code:java}
> org.apache.drill.exec.rpc.RpcException:
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> AssertionError: Type mismatch:
> rel rowtype:
> RecordType(INTEGER $f0, INTEGER $f1, BIGINT NOT NULL $f2, INTEGER $f3) NOT
> NULL
> equivRel rowtype:
> RecordType(INTEGER $f0, INTEGER $f1, BIGINT NOT NULL $f2, BIGINT $f3) NOT NULL
> [Error Id: f5114e62-a57b-46b1-afe8-ae652f390896 on localhost:31010]
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: Internal error: Error while applying rule
> DrillReduceAggregatesRule, args
> [rel#29:LogicalAggregate.NONE.ANY([]).[](input=rel#28:Subset#3.NONE.ANY([]).[],group={},agg#0=SUM($1),agg#1=SUM($0),agg#2=COUNT($0),agg#3=$SUM0($0))]
> org.apache.drill.exec.work.foreman.Foreman.run():294
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745
> Caused By (java.lang.AssertionError) Internal error: Error while applying
> rule DrillReduceAggregatesRule, args
> [rel#29:LogicalAggregate.NONE.ANY([]).[](input=rel#28:Subset#3.NONE.ANY([]).[],group={},agg#0=SUM($1),agg#1=SUM($0),agg#2=COUNT($0),agg#3=$SUM0($0))]
> org.apache.calcite.util.Util.newInternal():792
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():811
> {code}
> The reason is that stddev_samp(cast(employee_id as int)) will be reduced as
> sum($0) ,sum($1) ,count($0) while the sum(cast(employee_id as int)) will be
> reduced as sum0($0) by the DrillReduceAggregatesRule's first time matching.
> The second time's matching will reduce stddev_samp's sum($0) to sum0($0) too
> . But this sum0($0) 's data type is different from the first time's sum0($0)
> : one is integer ,the other is bigint . But Calcite's addAggCall method treat
> them as the same by ignoring their data type. This leads to the bigint
> sum0($0) be replaced by the integer sum0($0).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5957) Wire protocol versioning, version negotiation
[ https://issues.apache.org/jira/browse/DRILL-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249084#comment-16249084 ] Paul Rogers commented on DRILL-5957: [~tdunning], good points as usual! The end user is really the driver of version compatibility support. Drill users have been fairly silent on this front, so perhaps version compatibility is not necessary in this particular case. But, based on experience with other products, folks in production often need to: * Upgrade clients and servers on slightly different schedules, forcing the need for older clients to talk with newer servers or visa-versa. * If clients are installed on desktops (such as ODBC for Tableau), then the clients may actually be remote (or on a plane or at a customer site) at the time of server upgrades, forcing the need for older clients to connect to new servers. * If users have multiple Drill clusters, then they may have different upgrade schedules, requiring that a single client be able to speak with multiple server versions. (In the most obvious case, a user may have version X in production, while trying out version (X+1) in a test environment prior to upgrade.) Perhaps there are alternatives in the big data world. Maybe two versions of the Drill client can coexist in the same Tableau or other app? (For JDBC, this would mean that the clients must be in separate name spaces, much as SQuirreL does, but SQLline does not do.) Will there be a performance hit to "transcode" vectors across versions when the version changes? Of course. The question is, is the temporary performance hit an acceptable cost to allow a staged upgrade? Or, would the users prefer to do an all-at-once upgrade in order to avoid the performance hit? (And, of course, the performance hit creates a very good incentive to upgrade...) Finally, note that the "dbody" portion of of a Drill message exists outside of the Protobuf structure. A Drill message has four parts: * Message ID * P-body (Protobuf body) length * P-body (Serialized Protobuf content) * D-body (data body) length * D-body (serialized value vectors) For this reason, Protobuf formats don't help us with vector serialization. Note that vector data may be GB in size, so sending two copies is a worse performance impact than transcoding... All this said, if staged upgrades and version compatibility is not a concern for Drill users at present, then there is no barrier to upgrading our vector formats; we just require new clients be used with the new Drill version. This would, of course, be the simplest solution by far. > Wire protocol versioning, version negotiation > - > > Key: DRILL-5957 > URL: https://issues.apache.org/jira/browse/DRILL-5957 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.11.0 >Reporter: Paul Rogers > > Drill has very limited support for evolving its wire protocol. As Drill > becomes more widely deployed, this limitation will constrain the project's > ability to rapidly evolve the wire protocol based on user experience to > improve simplicitly, performance or minimize resource use. > Proposed is a standard mechanism to version the API and negotiate the API > version between client and server at connect time. The focus here is between > Drill clients (JDBC, ODBC) and the Drill server. The same mechanism can also > be used between servers to support rolling upgrades. > This proposal is an outline; it is not a detailed design. The purpose here is > to drive understanding of the problem. Once we have that, we can focus on the > implementation details. > h4. Problem Statement > The problem we wish to address here concerns both the _syntax_ and > _semantics_ of API messages. Syntax concerns: > * The set of messages and their sequence > * The format of bytes on the wire > * The format of message packets > Semantics concerns: > * The meaning of each field. > * The layout of non-message data (vectors, in Drill.) > We wish to introduce a system whereby both syntax and semantics can be > evolved in a controlled, known manner such that: > * A client of version x can connect to, and interoperate with, a server in a > range of versions (x-y, x+z) for some values of y and z. > For example, version x of the Drill client is deployed in the field. It must > connect to the oldest Drill cluster available to that client. (That is it > must connect to servers up to y versions old.) During an upgrade, the server > may be upgraded before the client. Thus, the client must also work with > servers up to z versions newer than the client. > If we wish to tackle rolling upgrades, then y and z can both be 1 for > server-to-server APIs. A version x server will talk with (x-1) servers when > the cluster upgrades to x, and will talk to (x+1) servers when the cluster is >
[jira] [Commented] (DRILL-4779) Kafka storage plugin support
[ https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249068#comment-16249068 ] ASF GitHub Bot commented on DRILL-4779: --- Github user kameshb commented on the issue: https://github.com/apache/drill/pull/1027 @paul-rogers @arina-ielchiieva @vrozov Thanks for reviewing. Anil & I have addressed review comments. Could you please go through the changes and also rest of the Kafka storage codebase. > Kafka storage plugin support > > > Key: DRILL-4779 > URL: https://issues.apache.org/jira/browse/DRILL-4779 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Labels: doc-impacting > Fix For: 1.12.0 > > > Implement Kafka storage plugin will enable the strong SQL support for Kafka. > Initially implementation can target for supporting json and avro message types -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5957) Wire protocol versioning, version negotiation
[ https://issues.apache.org/jira/browse/DRILL-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249063#comment-16249063 ] Ted Dunning commented on DRILL-5957: This suggestion has the virtue that only breaking changes will cause a version update, but it still has the problem that the version has to move no matter what part of the protocol changes. This is reminiscent of the old CORBA versioning nightmares. Also, is there really any way to negotiate the value vector format without having a reformatting step inserted with fairly catastrophic performance hit? I don't see a consideration of the cost of maintaining old version compatibility, either. If old client versions work, then there will be no incentive to upgrade. That will increase pressure to keep adding multiple protocol support to the server and will seemingly lock down any real progress just as much as client/server lockstepping. It seems that the short term desire here is to allow the vector format to change. What about making the current dvector parts be optional and adding alternative (optional) dvector parts in new formats? This effectively allows versioning of only the dvector stuff, leaving all the rest of the protocol to be soft-versioned as is currently done. The client advertised version could be used to trigger one format or the other and the incentive to upgrade is in the form of much slower transfer for the old format due to transcoding. > Wire protocol versioning, version negotiation > - > > Key: DRILL-5957 > URL: https://issues.apache.org/jira/browse/DRILL-5957 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.11.0 >Reporter: Paul Rogers > > Drill has very limited support for evolving its wire protocol. As Drill > becomes more widely deployed, this limitation will constrain the project's > ability to rapidly evolve the wire protocol based on user experience to > improve simplicitly, performance or minimize resource use. > Proposed is a standard mechanism to version the API and negotiate the API > version between client and server at connect time. The focus here is between > Drill clients (JDBC, ODBC) and the Drill server. The same mechanism can also > be used between servers to support rolling upgrades. > This proposal is an outline; it is not a detailed design. The purpose here is > to drive understanding of the problem. Once we have that, we can focus on the > implementation details. > h4. Problem Statement > The problem we wish to address here concerns both the _syntax_ and > _semantics_ of API messages. Syntax concerns: > * The set of messages and their sequence > * The format of bytes on the wire > * The format of message packets > Semantics concerns: > * The meaning of each field. > * The layout of non-message data (vectors, in Drill.) > We wish to introduce a system whereby both syntax and semantics can be > evolved in a controlled, known manner such that: > * A client of version x can connect to, and interoperate with, a server in a > range of versions (x-y, x+z) for some values of y and z. > For example, version x of the Drill client is deployed in the field. It must > connect to the oldest Drill cluster available to that client. (That is it > must connect to servers up to y versions old.) During an upgrade, the server > may be upgraded before the client. Thus, the client must also work with > servers up to z versions newer than the client. > If we wish to tackle rolling upgrades, then y and z can both be 1 for > server-to-server APIs. A version x server will talk with (x-1) servers when > the cluster upgrades to x, and will talk to (x+1) servers when the cluster is > upgraded to version (x+1). > h4. Current State > Drill currently provides some ad-hoc version compatibility: > * Slow change. Drill's APIs have not changed much since Drill 1.0, thereby > avoiding the issue. > * Protobuf support. Drill uses Protobuf for message bodies, leveraging that > format's ability to absorb the additional or deprecation of individual fields. > * API version number. The API holds a version number, though the code to use > it is rather ad-hoc. > The above has allowed clever coding to handle some version changes, but each > is a one-off, ad-hoc collision. The recent security work is an example that, > with enough effort, ad-hoc solutions can be found. > The above cannot handle: > * Change in the message order > * Change in the "pbody/dbody" structure of each message. > * Change in the structure of serialized value vectors. > As a result, the current structure prevents any change to Drill's core > mechanism, value vectors, as there is no way or clients and servers to > negotiate the vector wire format. For example, Drill cannot adopt Arrow > because a pre-Arrow client would not understand
[jira] [Commented] (DRILL-4779) Kafka storage plugin support
[
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249039#comment-16249039
]
ASF GitHub Bot commented on DRILL-4779:
---
Github user kameshb commented on a diff in the pull request:
https://github.com/apache/drill/pull/1027#discussion_r150435308
--- Diff:
contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaRecordReader.java
---
@@ -0,0 +1,178 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.kafka;
+
+import static
org.apache.drill.exec.store.kafka.DrillKafkaConfig.DRILL_KAFKA_POLL_TIMEOUT;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.kafka.KafkaSubScan.KafkaSubScanSpec;
+import org.apache.drill.exec.store.kafka.decoders.MessageReader;
+import org.apache.drill.exec.store.kafka.decoders.MessageReaderFactory;
+import org.apache.drill.exec.util.Utilities;
+import org.apache.drill.exec.vector.complex.impl.VectorContainerWriter;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+public class KafkaRecordReader extends AbstractRecordReader {
+ private static final Logger logger =
LoggerFactory.getLogger(KafkaRecordReader.class);
+ public static final long DEFAULT_MESSAGES_PER_BATCH = 4000;
+
+ private VectorContainerWriter writer;
+ private MessageReader messageReader;
+
+ private boolean unionEnabled;
+ private KafkaConsumer kafkaConsumer;
+ private KafkaStoragePlugin plugin;
+ private KafkaSubScanSpec subScanSpec;
+ private long kafkaPollTimeOut;
+ private long endOffset;
+
+ private long currentOffset;
+ private long totalFetchTime = 0;
+
+ private List partitions;
+ private final boolean enableAllTextMode;
+ private final boolean readNumbersAsDouble;
+
+ private Iterator> messageIter;
+
+ public KafkaRecordReader(KafkaSubScan.KafkaSubScanSpec subScanSpec,
List projectedColumns,
+ FragmentContext context, KafkaStoragePlugin plugin) {
+setColumns(projectedColumns);
+this.enableAllTextMode =
context.getOptions().getOption(ExecConstants.KAFKA_ALL_TEXT_MODE).bool_val;
+this.readNumbersAsDouble = context.getOptions()
+
.getOption(ExecConstants.KAFKA_READER_READ_NUMBERS_AS_DOUBLE).bool_val;
+this.unionEnabled =
context.getOptions().getOption(ExecConstants.ENABLE_UNION_TYPE);
+this.plugin = plugin;
+this.subScanSpec = subScanSpec;
+this.endOffset = subScanSpec.getEndOffset();
+this.kafkaPollTimeOut =
Long.valueOf(plugin.getConfig().getDrillKafkaProps().getProperty(DRILL_KAFKA_POLL_TIMEOUT));
+ }
+
+ @Override
+ protected Collection transformColumns(Collection
projectedColumns) {
+Set transformed = Sets.newLinkedHashSet();
+if (!isStarQuery()) {
+ for (SchemaPath column : projectedColumns) {
+transformed.add(column);
+ }
+} else {
+ transformed.add(Utilities.STAR_COLUMN);
+}
[jira] [Updated] (DRILL-5958) Revisit the List and RepeatedList vectors
[
https://issues.apache.org/jira/browse/DRILL-5958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5958:
---
Description:
Drill provides a List vector used when reading JSON data. The semantics of this
vector are somewhat obscure and overly complex. This ticket asks to clean up
the design and implementation of this vector.
h4. Current Behavior
Drill contains two kinds of repeated types:
* Repeated vectors, which exist for all Drill types.
* List vectors, which exist outside the repeated vector system.
Lists are rather hard to explain.
Drill has 38 types. Each type comes in three cardinalities: Required (0),
Optional (0, 1) or Repeated (0..n). Thus, there is an {{IntVector}}, a
{{NullableIntVector}} and a {{RepeatedIntVector}}.
Lists are an an odd duck and exist outside of this system. A list is not simply
another level of repetition (a {{RepeatedRepeatedIntVector}}. Rather, a list is
heterogeneous: it is just a list of something.
A list is a semi-union: it starts as a list of something. As work progresses,
we may realize that we need to store some other thing. At that point, the
contents of the list morph from the original type to the union type. The result
of this decision is that, when inspecting the metadata for a list (in the form
of a {{MaterializedField}}), there is no information about list type.
Presumably the code must inspect the "data vector" associated with the list to
learn the type of the items within the list. (If done at write time, then, of
course, the type can change as described above.)
For this reason, the List type is closely associated with the Union type: a
list is, essentially, a "repeated Union", though it starts of as a "repeated
something", then evolves to become a repeated union. (Drill has no
{{RepeatedUnionVector}}, presumably because {{ListVector}} is the repeated form
of the union type.)
Strangely, Drill also has a {{RepeatedListVector}}, which introduces all manner
of ambiguity. Combining these, the cardinality hierarchy for unions is:
* {{UnionVector}} (like an optional union type)
* {{ListVector}} (repeated semi-union)
* {{RepeatedListVector}} (a 2D union array)
* {{RepeatedListVector}} which contains a {{ListVector}} (a 3D union grid. Note
that this could also be implemented as a {{ListVector}} that contains a
{{RepeatedListVector}} or even a {{ListVector}} which contains a {{ListVector}}
which contains a {{ListVector}} which implicitly contains a {{UnionVector}}.)
* {{RepeatedListVector}} which contains a {{RepeatedListVector}} (a 4D hyper
grid.)
* And so on.
For a primitive type, such as Int, we have:
* {{IntVector}} or {{NullableIntVector}} (cardinality of 1 or (0,1))
* {{RepeatedIntVector}} (a 1D list of Int)
* {{ListVector}} which contains a {{RepeatedIntVector}} (a 2D array of ints.
Not that this could have been a {{RepeatedListVector}} that stores only ints.)
* {{RepeatedListVector}} which contains a {{RepeatedIntVector}} (a 3D cube of
ints. This could also be formed by a {{ListVector}} that contains a
{{ListVector}} that contains a {{RepeatedIntVector}} along with several other
combinations.)
h4. Examples of Current Behavior
Lists and repeated types appeared to evolve to support JSON-like structures.
For example:
{code}
{a: 10} {a: null}
{code}
Here, `a` is a nullable scalar and is represented as a {{NullableIntVector}}.
{code}
{a: [10, 20]}
{code}
Here, `a` is a list of Int and is represented as a {{RepeatedIntVector}}.
Drill does not allow nulls in such vectors, so we cannot represent:
{code}
{a: [10, null, 20]}
{code}
Once we go beyond 1D, we need lists:
{code}
{a: [[10, 20], [30, 40]]}
{code}
The above requires a {{ListVector}} that contains a {{RepeatedIntVector}}.
{code}
{a: [[[110, 120], [130, 140]], [210, 220], [230, 240]]}
{code}
The above requires a {{RepeatedListVector}} that contains a
{{RepeatedIntVector}}.
Similarly, since lists can hold any type (just like a union), we can have
repeated objects:
{code}
{a: [[{x: 0, y: 0}, {x: 1, y: 0}], [{x: 4, y: 0}, {x: 4, y: 1}]]}
{code}
The above would be represented as a {{ListVector}} that contains a
{{RepeatedMapVector}}. (Or, equivalently, a {{RepeatedListVector}} that
contains a {{MapVector}}.)
Because the List vector is a union type, it can (presumably) also handle
heterogeneous lists (though this needs to be checked to see if the code
actually supports this case):
{code}
{a: [10, "fred", 123.45, null]}
{code}
Since unions support combinations of not just scalars, but also scalars and
complex types, Drill can also support:
{code}
{a: [10, {b: "foo"}, null, [10, "bob"]]}
{code}
h4. Muddy Semantics
The above show a number of problems that make lists (and unions) far more
complex than necessary:
* Ambiguity of when to use a {{ListVector}} of {{FooVector}} vs. a
{{RepeatedFooVector}}.
* Ambiguity of when to use a {{ListVector}} of
[jira] [Updated] (DRILL-5958) Revisit the List and RepeatedList vectors
[
https://issues.apache.org/jira/browse/DRILL-5958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5958:
---
Description:
Drill provides a List vector used when reading JSON data. The semantics of this
vector are somewhat obscure and overly complex. This ticket asks to clean up
the design and implementation of this vector.
h4. Current Behavior
Drill contains two kinds of repeated types:
* Repeated vectors, which exist for all Drill types.
* List vectors, which exist outside the repeated vector system.
Lists are rather hard to explain.
Drill has 38 types. Each type comes in three cardinalities: Required (0),
Optional (0, 1) or Repeated (0..n). Thus, there is an {{IntVector}}, a
{{NullableIntVector}} and a {{RepeatedIntVector}}.
Lists are an an odd duck and exist outside of this system. A list is not simply
another level of repetition (a {{RepeatedRepeatedIntVector}}. Rather, a list is
heterogeneous: it is just a list of something.
A list is a semi-union: it starts as a list of something. As work progresses,
we may realize that we need to store some other thing. At that point, the
contents of the list morph from the original type to the union type. The result
of this decision is that, when inspecting the metadata for a list (in the form
of a {{MaterializedField}}), there is no information about list type.
Presumably the code must inspect the "data vector" associated with the list to
learn the type of the items within the list. (If done at write time, then, of
course, the type can change as described above.)
For this reason, the List type is closely associated with the Union type: a
list is, essentially, a "repeated Union", though it starts of as a "repeated
something", then evolves to become a repeated union. (Drill has no
{{RepeatedUnionVector}}, presumably because {{ListVector}} is the repeated form
of the union type.)
Strangely, Drill also has a {{RepeatedListVector}}, which introduces all manner
of ambiguity. Combining these, the cardinality hierarchy for unions is:
* {{UnionVector}} (like an optional union type)
* {{ListVector}} (repeated semi-union)
* {{RepeatedListVector}} (a 2D union array)
* {{RepeatedListVector}} which contains a {{ListVector}} (a 3D union grid. Note
that this could also be implemented as a {{ListVector}} that contains a
{{RepeatedListVector}}.)
* {{RepeatedListVector}} which contains a {{RepeatedListVector}} (a 4D hyper
grid.)
* And so on.
For a primitive type, such as Int, we have:
* {{IntVector}} or {{NullableIntVector}} (cardinality of 1 or (0,1))
* {{RepeatedIntVector}} (a 1D list of Int)
* {{ListVector}} which contains a {{RepeatedIntVector}} (a 2D array of ints.
Not that this could have been a {{RepeatedListVector}} that stores only ints.)
* {{RepeatedListVector}} which contains a {{RepeatedIntVector}} (a 3D cube of
ints. This could also be formed by a {{ListVector}} that contains a
{{ListVector}} that contains a {{RepeatedIntVector}} along with several other
combinations.)
h4. Examples of Current Behavior
Lists and repeated types appeared to evolve to support JSON-like structures.
For example:
{code}
{a: 10} {a: null}
{code}
Here, `a` is a nullable scalar and is represented as a {{NullableIntVector}}.
{code}
{a: [10, 20]}
{code}
Here, `a` is a list of Int and is represented as a {{RepeatedIntVector}}.
Drill does not allow nulls in such vectors, so we cannot represent:
{code}
{a: [10, null, 20]}
{code}
Once we go beyond 1D, we need lists:
{code}
{a: [[10, 20], [30, 40]]}
{code}
The above requires a {{ListVector}} that contains a {{RepeatedIntVector}}.
{code}
{a: [[[110, 120], [130, 140]], [210, 220], [230, 240]]}
{code}
The above requires a {{RepeatedListVector}} that contains a
{{RepeatedIntVector}}.
Similarly, since lists can hold any type (just like a union), we can have
repeated objects:
{code}
{a: [[{x: 0, y: 0}, {x: 1, y: 0}], [{x: 4, y: 0}, {x: 4, y: 1}]]}
{code}
The above would be represented as a {{ListVector}} that contains a
{{RepeatedMapVector}}. (Or, equivalently, a {{RepeatedListVector}} that
contains a {{MapVector}}.)
Because the List vector is a union type, it can (presumably) also handle
heterogeneous lists (though this needs to be checked to see if the code
actually supports this case):
{code}
{a: [10, "fred", 123.45, null]}
{code}
Since unions support combinations of not just scalars, but also scalars and
complex types, Drill can also support:
{code}
{a: [10, {b: "foo"}, null, [10, "bob"]]}
{code}
h4. Muddy Semantics
The above show a number of problems that make lists (and unions) far more
complex than necessary:
* Ambiguity of when to use a {{ListVector}} of {{FooVector}} vs. a
{{RepeatedFooVector}}.
* Ambiguity of when to use a {{ListVector}} of {{RepeatedFooVector}} vs. a
{{RepeatedListVector}} of {{FooVector}}.
The same solution used to handle extra layers of repetition is used to handle
[jira] [Updated] (DRILL-5955) Revisit Union Vectors
[
https://issues.apache.org/jira/browse/DRILL-5955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5955:
---
Description:
Drill supports a “Union Vector” type that allows a single column to hold values
of multiple types. Conceptually, each column value is a (type, value) pair. For
example, row 0 might be an Int, row 1 a Varchar and row 2 a NULL value.
The name refers to a C “union” in which the same bit of memory is used to
represent one of a set of defined types.
Drill implements the union vector a bit like a map: as a collection of typed
vectors. Each value is keyed by type. The result is that a union vector is more
like a C “struct” than a C “union”: every vector takes space, but only one of
the vectors is used for each row. For the example above, the union vector
contains an Int vector, a Varchar vector and a type vector. For each row,
either the Int or the Varchar is used. For NULL values, neither vector is used.
h4. Memory Footprint Concerns
The current representation, despite its name, makes very inefficient use of
memory because it requires the sum of the storage for each included type. (That
is, if we store 1000 rows, we need 1000 slots for integers, another 1000 for
Varchar and yet another 1000 for the type vector.)
Drill poorly supports the union type. One operator that does support it is the
sort. If the union type is enabled, and the sort sees a schema change, the sort
will create a new union vector that combines the two types. The result is a
sudden, unplanned increase in memory usage. Since the sort can buffer many
hundreds of batches, this unplanned memory increase can cause the sort to run
out of memory.
h4. Muddy Semantics
The union vector is closely tied with the List vector: a list vector is,
essentially, an array of unions. (See DRILL-5958). The list type is used to
model JSON in which a list can hold anything: another list, an object or
scalars. For this reason, the union vector also can hold any type. And, indeed,
it can hold a union of any of these types: a Map and an Int, or a List and a
Map.
Drill is a relational, SQL-based tool. Work is required to bring non-relational
structures into Drill. As discussed below, a union of scalars can be made to
work. But, a union of structured types (lists, arrays or Maps) makes no sense.
h4. High Complexity
The union vector, as implemented is quite complex. It contains member variables
for every other vector type (except, strangely, the decimal types.) Access to
typed members is by type-specific methods, meaning that the client code must
include a separate call for every type, resulting in very complex client code.
The complexity allowed the union type to be made to work, but causes this one
type to consume a disproportionate amount of the vector and client code.
h4. Proposed Revision to Structure: The Variant Type
Given the above, we can now present the proposed changes. First let us
recognize that a union vector need not hold structured types; there are other
solutions as discussed in DRILL-. This leaves the union vector to hold just
scalars.
h4. Proposed Revision to Storage
This in turn lets us adopt the [Variant
type|https://en.wikipedia.org/wiki/Variant_type] originally introduced in
Visual Basic. Variant “is a tagged union that can be used to represent any
other data type”. The Variant type was designed to be compact by building on
the idea of a tagged union in C.
{code}
struct {
int tag; // type
union {
int intValue;
long longValue;
…
}
}
{code}
When implemented as a vector, the format could consume just a single
variable-width vector with each entry of the form: {{\[type value]}}. The
vector is simply a sequence of these (type, value) pairs.
The type is a single-byte that encodes the MinorType that describes the value.
That is, the type byte is like the existing type vector, but stored in the same
location as the data. The data is simply the serialized format of data. (Four
bytes for an Int, 8 bytes for a Float8 and so on.)
Variable-width types require an extra field: the type field: {{\[type length
value]}}. For example, a Varchar would be encoded as {{\[Varchar 27 byte0-26]}}.
A writer uses the type to drive the serialization. A reader uses the type to
drive deserialization.
Note that the type field must include a special marker for nulls. Today, the
union type uses 0 to indicate a null value. (Note that, in a union and variant,
a null value is not a null of some type, both the type and value are null.)
That form should be used in the variant representation as well. But, note that
the 0 value in the MajorType enum is not Null but is instead Late. This is an
unpleasant messiness that the union (and variant )encoding must handle.
An offset vector provides the location of each field, as is done with
variable-length vectors today.
The result is huge compaction
[jira] [Created] (DRILL-5958) Revisit the List and RepeatedList vectors
Paul Rogers created DRILL-5958:
--
Summary: Revisit the List and RepeatedList vectors
Key: DRILL-5958
URL: https://issues.apache.org/jira/browse/DRILL-5958
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.11.0
Reporter: Paul Rogers
Drill provides a List vector used when reading JSON data. The semantics of this
vector are somewhat obscure and overly complex. This ticket asks to clean up
the design and implementation of this vector.
h4. Current Behavior
Drill contains two kinds of repeated types:
* Repeated vectors, which exist for all Drill types.
* List vectors, which exist outside the repeated vector system.
Lists are rather hard to explain.
Drill has 38 types. Each type comes in three cardinalities: Required (0),
Optional (0, 1) or Repeated (0..n). Thus, there is an {{IntVector}}, a
{{NullableIntVector}} and a {{RepeatedIntVector}}.
Lists are an an odd duck and exist outside of this system. A list is not simply
another level of repetition (a {{RepeatedRepeatedIntVector}}. Rather, a list is
heterogeneous: it is just a list of something.
For this reason, the List type is closely associated with the Union type: a
list is, essentially, a "repeated Union", though it is not implemented that way.
Strangely, Drill does have a {{RepeatedListVector}}, which introduces all
manner of ambiguity. Combining these, the cardinality hierarchy for unions is:
* {{UnionVector}} (like an optional union type)
* {{ListVector}} (repeated union)
* {{RepeatedListVector}} (a 2D union array)
* {{RepeatedListVector}} which contains a {{ListVector}} (a 3D union grid. Note
that this could also be implemented as a {{ListVector}} that contains a
{{RepeatedListVector}}.)
* {{RepeatedListVector}} which contains a {{RepeatedListVector}} (a 4D hyper
grid.)
* And so on.
For a primitive type, such as Int, we have:
* {{IntVector}} or {{NullableIntVector}} (cardinality of 1 or (0,1))
* {{RepeatedIntVector}} (a 1D list of Int)
* {{ListVector}} which contains a {{RepeatedIntVector}} (a 2D array of ints.
Not that this could have been a {{RepeatedListVector}} that stores only ints.)
* {{RepeatedListVector}} which contains a {{RepeatedIntVector}} (a 3D cube of
ints. This could also be formed by a {{ListVector}} that contains a
{{ListVector}} that contains a {{RepeatedIntVector}} along with several other
combinations.)
h4. Examples of Current Behavior
Lists and repeated types appeared to evolve to support JSON-like structures.
For example:
{code}
{a: 10} {a: null}
{code}
Here, `a` is a nullable scalar and is represented as a {{NullableIntVector}}.
{code}
{a: [10, 20]}
{code}
Here, `a` is a list of Int and is represented as a {{RepeatedIntVector}}.
Drill does not allow nulls in such vectors, so we cannot represent:
{code}
{a: [10, null, 20]}
{code}
Once we go beyond 1D, we need lists:
{code}
{a: [[10, 20], [30, 40]]}
{code}
The above requires a {{ListVector}} that contains a {{RepeatedIntVector}}.
{code}
{a: [[[110, 120], [130, 140]], [210, 220], [230, 240]]}
{code}
The above requires a {{RepeatedListVector}} that contains a
{{RepeatedIntVector}}.
Similarly, since lists can hold any type (just like a union), we can have
repeated objects:
{code}
{a: [[{x: 0, y: 0}, {x: 1, y: 0}], [{x: 4, y: 0}, {x: 4, y: 1}]]}
{code}
The above would be represented as a {{ListVector}} that contains a
{{RepeatedMapVector}}. (Or, equivalently, a {{RepeatedListVector}} that
contains a {{MapVector}}.)
Because the List vector is a union type, it can (presumably) also handle
heterogeneous lists (though this needs to be checked to see if the code
actually supports this case):
{code}
{a: [10, "fred", 123.45, null]}
{code}
Since unions support combinations of not just scalars, but also scalars and
complex types, Drill can also support:
{code}
{a: [10, {b: "foo"}, null, [10, "bob"]]}
{code}
h4. Muddy Semantics
The above show a number of problems that make lists (and unions) far more
complex than necessary:
* Ambiguity of when to use a {{ListVector}} of {{FooVector}} vs. a
{{RepeatedFooVector}}.
* Ambiguity of when to use a {{ListVector}} of {{RepeatedFooVector}} vs. a
{{RepeatedListVector}} of {{FooVector}}.
The same solution used to handle extra layers of repetition is used to handle
variant types (DRILL-5955):
* Lists can handle any combination of scalars.
* Lists can handle any structure type (map, repeated map, list, repeated list).
* Lists are thus not typed. They are not a "List of Int", they are just a List.
h4. Mapping to SQL
The above muddy semantics give rise to this question. Drill is a SQL engine,
how do we map the List semantics to a relational schema? If we don't have a
clean answer, then the List type, while clever, does not have a useful purpose
and is instead distracting us from the larger question of how we map JSON-like
structures to a relational schema.
[jira] [Updated] (DRILL-5955) Revisit Union Vectors
[
https://issues.apache.org/jira/browse/DRILL-5955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5955:
---
Description:
Drill supports a “Union Vector” type that allows a single column to hold values
of multiple types. Conceptually, each column value is a (type, value) pair. For
example, row 0 might be an Int, row 1 a Varchar and row 2 a NULL value.
The name refers to a C “union” in which the same bit of memory is used to
represent one of a set of defined types.
Drill implements the union vector a bit like a map: as a collection of typed
vectors. Each value is keyed by type. The result is that a union vector is more
like a C “struct” than a C “union”: every vector takes space, but only one of
the vectors is used for each row. For the example above, the union vector
contains an Int vector, a Varchar vector and a type vector. For each row,
either the Int or the Varchar is used. For NULL values, neither vector is used.
h4. Memory Footprint Concerns
The current representation, despite its name, makes very inefficient use of
memory because it requires the sum of the storage for each included type. (That
is, if we store 1000 rows, we need 1000 slots for integers, another 1000 for
Varchar and yet another 1000 for the type vector.)
Drill poorly supports the union type. One operator that does support it is the
sort. If the union type is enabled, and the sort sees a schema change, the sort
will create a new union vector that combines the two types. The result is a
sudden, unplanned increase in memory usage. Since the sort can buffer many
hundreds of batches, this unplanned memory increase can cause the sort to run
out of memory.
h4. Muddy Semantics
The union vector is closely tied with the List vector: a list vector is,
essentially, an array of unions. (See DRILL-). The list type is used to
model JSON in which a list can hold anything: another list, an object or
scalars. For this reason, the union vector also can hold any type. And, indeed,
it can hold a union of any of these types: a Map and an Int, or a List and a
Map.
Drill is a relational, SQL-based tool. Work is required to bring non-relational
structures into Drill. As discussed below, a union of scalars can be made to
work. But, a union of structured types (lists, arrays or Maps) makes no sense.
h4. High Complexity
The union vector, as implemented is quite complex. It contains member variables
for every other vector type (except, strangely, the decimal types.) Access to
typed members is by type-specific methods, meaning that the client code must
include a separate call for every type, resulting in very complex client code.
The complexity allowed the union type to be made to work, but causes this one
type to consume a disproportionate amount of the vector and client code.
h4. Proposed Revision to Structure: The Variant Type
Given the above, we can now present the proposed changes. First let us
recognize that a union vector need not hold structured types; there are other
solutions as discussed in DRILL-. This leaves the union vector to hold just
scalars.
h4. Proposed Revision to Storage
This in turn lets us adopt the [Variant
type|https://en.wikipedia.org/wiki/Variant_type] originally introduced in
Visual Basic. Variant “is a tagged union that can be used to represent any
other data type”. The Variant type was designed to be compact by building on
the idea of a tagged union in C.
{code}
struct {
int tag; // type
union {
int intValue;
long longValue;
…
}
}
{code}
When implemented as a vector, the format could consume just a single
variable-width vector with each entry of the form: {{\[type value]}}. The
vector is simply a sequence of these (type, value) pairs.
The type is a single-byte that encodes the MinorType that describes the value.
That is, the type byte is like the existing type vector, but stored in the same
location as the data. The data is simply the serialized format of data. (Four
bytes for an Int, 8 bytes for a Float8 and so on.)
Variable-width types require an extra field: the type field: {{\[type length
value]}}. For example, a Varchar would be encoded as {{\[Varchar 27 byte0-26]}}.
A writer uses the type to drive the serialization. A reader uses the type to
drive deserialization.
Note that the type field must include a special marker for nulls. Today, the
union type uses 0 to indicate a null value. (Note that, in a union and variant,
a null value is not a null of some type, both the type and value are null.)
That form should be used in the variant representation as well. But, note that
the 0 value in the MajorType enum is not Null but is instead Late. This is an
unpleasant messiness that the union (and variant )encoding must handle.
An offset vector provides the location of each field, as is done with
variable-length vectors today.
The result is huge compaction
[jira] [Updated] (DRILL-5957) Wire protocol versioning, version negotiation
[ https://issues.apache.org/jira/browse/DRILL-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers updated DRILL-5957: --- Description: Drill has very limited support for evolving its wire protocol. As Drill becomes more widely deployed, this limitation will constrain the project's ability to rapidly evolve the wire protocol based on user experience to improve simplicitly, performance or minimize resource use. Proposed is a standard mechanism to version the API and negotiate the API version between client and server at connect time. The focus here is between Drill clients (JDBC, ODBC) and the Drill server. The same mechanism can also be used between servers to support rolling upgrades. This proposal is an outline; it is not a detailed design. The purpose here is to drive understanding of the problem. Once we have that, we can focus on the implementation details. h4. Problem Statement The problem we wish to address here concerns both the _syntax_ and _semantics_ of API messages. Syntax concerns: * The set of messages and their sequence * The format of bytes on the wire * The format of message packets Semantics concerns: * The meaning of each field. * The layout of non-message data (vectors, in Drill.) We wish to introduce a system whereby both syntax and semantics can be evolved in a controlled, known manner such that: * A client of version x can connect to, and interoperate with, a server in a range of versions (x-y, x+z) for some values of y and z. For example, version x of the Drill client is deployed in the field. It must connect to the oldest Drill cluster available to that client. (That is it must connect to servers up to y versions old.) During an upgrade, the server may be upgraded before the client. Thus, the client must also work with servers up to z versions newer than the client. If we wish to tackle rolling upgrades, then y and z can both be 1 for server-to-server APIs. A version x server will talk with (x-1) servers when the cluster upgrades to x, and will talk to (x+1) servers when the cluster is upgraded to version (x+1). h4. Current State Drill currently provides some ad-hoc version compatibility: * Slow change. Drill's APIs have not changed much since Drill 1.0, thereby avoiding the issue. * Protobuf support. Drill uses Protobuf for message bodies, leveraging that format's ability to absorb the additional or deprecation of individual fields. * API version number. The API holds a version number, though the code to use it is rather ad-hoc. The above has allowed clever coding to handle some version changes, but each is a one-off, ad-hoc collision. The recent security work is an example that, with enough effort, ad-hoc solutions can be found. The above cannot handle: * Change in the message order * Change in the "pbody/dbody" structure of each message. * Change in the structure of serialized value vectors. As a result, the current structure prevents any change to Drill's core mechanism, value vectors, as there is no way or clients and servers to negotiate the vector wire format. For example, Drill cannot adopt Arrow because a pre-Arrow client would not understand "dbody" message parts encoded in Arrow format and visa-versa. h4. API Version The core of the proposal is to introduce an API version. This is a simple integer which is incremented each time that a breaking change is made to the API. (If the change can be absorbed by the Protobuf mechanism, then it is not a breaking change.) Note that the API version *is not* the same as the product version. Two different Drill versions may have the same API version if nothing changed in the API. h4. Version Negotiation Given a set of well-defined protocol versions, we can next define the version negotiation protocol between client and server: * The client connects and sends a "hello" message that identifies the range of API versions that it supports, with the newest version being the version of the client itself. * The server receives the message and computes the version of the session as the newest client version the the server supports. * The server returns this version to the client which switches to the selected API version. (The server returns an error, and disconnects, if there is no common version.) * The server and client use only messages valid for the given API version. This may mean converting data from one representation to another. The above is pretty standard. h4. Backward Compatibility Implementation Consider a server that must work with its own version (version c) and, say, two older versions (a and b). In most cases, changes across versions are minor. Perhaps version b introduced a better error reporting format (akin to SQLWARN and SQLERROR codes). Version c may have change the layout of a particular vector (or introduce a new vector type.) How does the server handle these when talking with
[jira] [Created] (DRILL-5957) Wire protocol versioning, version negotiation
Paul Rogers created DRILL-5957: -- Summary: Wire protocol versioning, version negotiation Key: DRILL-5957 URL: https://issues.apache.org/jira/browse/DRILL-5957 Project: Apache Drill Issue Type: Improvement Affects Versions: 1.11.0 Reporter: Paul Rogers Drill has very limited support for evolving its wire protocol. As Drill becomes more widely deployed, this limitation will constrain the project's ability to rapidly evolve the wire protocol based on user experience to improve simplicitly, performance or minimize resource use. Proposed is a standard mechanism to version the API and negotiate the API version between client and server at connect time. The focus here is between Drill clients (JDBC, ODBC) and the Drill server. The same mechanism can also be used between servers to support rolling upgrades. This proposal is an outline; it is not a detailed design. The purpose here is to drive understanding of the problem. Once we have that, we can focus on the implementation details. h4. Problem Statement The problem we wish to address here concerns both the _syntax_ and _semantics_ of API messages. Syntax concerns: * The set of messages and their sequence * The format of bytes on the wire * The format of message packets Semantics concerns: * The meaning of each field. * The layout of non-message data (vectors, in Drill.) We wish to introduce a system whereby both syntax and semantics can be evolved in a controlled, known manner such that: * A client of version x can connect to, and interoperate with, a server in a range of versions (x-y, x+z) for some values of y and z. For example, version x of the Drill client is deployed in the field. It must connect to the oldest Drill cluster available to that client. (That is it must connect to servers up to y versions old.) During an upgrade, the server may be upgraded before the client. Thus, the client must also work with servers up to z versions newer than the client. If we wish to tackle rolling upgrades, then y and z can both be 1 for server-to-server APIs. A version x server will talk with (x-1) servers when the cluster upgrades to x, and will talk to (x+1) servers when the cluster is upgraded to version (x+1). h4. Current State Drill currently provides some ad-hoc version compatibility: * Slow change. Drill's APIs have not changed much since Drill 1.0, thereby avoiding the issue. * Protobuf support. Drill uses Protobuf for message bodies, leveraging that format's ability to absorb the additional or deprecation of individual fields. * API version number. The API holds a version number, though the code to use it is rather ad-hoc. The above has allowed clever coding to handle some version changes, but each is a one-off, ad-hoc collision. The recent security work is an example that, with enough effort, ad-hoc solutions can be found. The above cannot handle: * Change in the message order * Change in the "pbody/dbody" structure of each message. * Change in the structure of serialized value vectors. As a result, the current structure prevents any change to Drill's core mechanism, value vectors, as there is no way or clients and servers to negotiate the vector wire format. For example, Drill cannot adopt Arrow because a pre-Arrow client would not understand "dbody" message parts encoded in Arrow format and visa-versa. h4. API Version The core of the proposal is to introduce an API version. This is a simple integer which is incremented each time that a breaking change is made to the API. (If the change can be absorbed by the Protobuf mechanism, then it is not a breaking change.) Note that the API version *is not* the same as the product version. Two different Drill versions may have the same API version if nothing changed in the API. h4. Version Negotiation Given a set of well-defined protocol versions, we can next define the version negotiation protocol between client and server: * The client connects and sends a "hello" message that identifies the range of API versions that it supports, with the newest version being the version of the client itself. * The server receives the message and computes the version of the session as the newest client version the the server supports. * The server returns this version to the client which switches to the selected API version. (The server returns an error, and disconnects, if there is no common version.) * The server and client use only messages valid for the given API version. This may mean converting data from one representation to another. The above is pretty standard. h4. Backward Compatibility Implementation Consider a server that must work with its own version (version c) and, say, two older versions (a and b). In most cases, changes across versions are minor. Perhaps version b introduced a better error reporting format (akin to SQLWARN and
[jira] [Updated] (DRILL-5956) Add storage plugin for Druid
[ https://issues.apache.org/jira/browse/DRILL-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaqi Liu updated DRILL-5956: - Component/s: Storage - Other > Add storage plugin for Druid > > > Key: DRILL-5956 > URL: https://issues.apache.org/jira/browse/DRILL-5956 > Project: Apache Drill > Issue Type: Wish > Components: Storage - Other >Reporter: Jiaqi Liu > > As more and more companies are using Druid for mission-critical industrial > products, Drill could gain much more popularity with Druid as one of its > supported storage plugin so that uses could easily bind Druid cluster to > running Drill instance -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5956) Add storage plugin for Druid
Jiaqi Liu created DRILL-5956: Summary: Add storage plugin for Druid Key: DRILL-5956 URL: https://issues.apache.org/jira/browse/DRILL-5956 Project: Apache Drill Issue Type: Wish Reporter: Jiaqi Liu As more and more companies are using Druid for mission-critical industrial products, Drill could gain much more popularity with Druid as one of its supported storage plugin so that uses could easily bind Druid cluster to running Drill instance -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5955) Revisit Union Vectors
[
https://issues.apache.org/jira/browse/DRILL-5955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5955:
---
Description:
Drill supports a “Union Vector” type that allows a single column to hold values
of multiple types. Conceptually, each column value is a (type, value) pair. For
example, row 0 might be an Int, row 1 a Varchar and row 2 a NULL value.
The name refers to a C “union” in which the same bit of memory is used to
represent one of a set of defined types.
Drill implements the union vector a bit like a map: as a collection of typed
vectors. Each value is keyed by type. The result is that a union vector is more
like a C “struct” than a C “union”: every vector takes space, but only one of
the vectors is used for each row. For the example above, the union vector
contains an Int vector, a Varchar vector and a type vector. For each row,
either the Int or the Varchar is used. For NULL values, neither vector is used.
h4. Memory Footprint Concerns
The current representation, despite its name, makes very inefficient use of
memory because it requires the sum of the storage for each included type. (That
is, if we store 1000 rows, we need 1000 slots for integers, another 1000 for
Varchar and yet another 1000 for the type vector.)
Drill poorly supports the union type. One operator that does support it is the
sort. If the union type is enabled, and the sort sees a schema change, the sort
will create a new union vector that combines the two types. The result is a
sudden, unplanned increase in memory usage. Since the sort can buffer many
hundreds of batches, this unplanned memory increase can cause the sort to run
out of memory.
h4. Muddy Semantics
The union vector is closely tied with the List vector: a list vector is,
essentially, an array of unions. (See DRILL-). The list type is used to
model JSON in which a list can hold anything: another list, an object or
scalars. For this reason, the union vector also can hold any type. And, indeed,
it can hold a union of any of these types: a Map and an Int, or a List and a
Map.
Drill is a relational, SQL-based tool. Work is required to bring non-relational
structures into Drill. As discussed below, a union of scalars can be made to
work. But, a union of structured types (lists, arrays or Maps) makes no sense.
h4. High Complexity
The union vector, as implemented is quite complex. It contains member variables
for every other vector type (except, strangely, the decimal types.) Access to
typed members is by type-specific methods, meaning that the client code must
include a separate call for every type, resulting in very complex client code.
The complexity allowed the union type to be made to work, but causes this one
type to consume a disproportionate amount of the vector and client code.
h4. Proposed Revision to Structure: The Variant Type
Given the above, we can now present the proposed changes. First let us
recognize that a union vector need not hold structured types; there are other
solutions as discussed in DRILL-. This leaves the union vector to hold just
scalars.
h4. Proposed Revision to Storage
This in turn lets us adopt the [Variant
type|https://en.wikipedia.org/wiki/Variant_type] originally introduced in
Visual Basic. Variant “is a tagged union that can be used to represent any
other data type”. The Variant type was designed to be compact by building on
the idea of a tagged union in C.
{code}
struct {
int tag; // type
union {
int intValue;
long longValue;
…
}
}
{code}
When implemented as a vector, the format could consume just a single
variable-width vector with each entry of the form: {{\[type value]}}. The
vector is simply a sequence of these (type, value) pairs.
The type is a single-byte that encodes the MinorType that describes the value.
That is, the type byte is like the existing type vector, but stored in the same
location as the data. The data is simply the serialized format of data. (Four
bytes for an Int, 8 bytes for a Float8 and so on.)
Variable-width types require an extra field: the type field: {{\[type length
value]}}. For example, a Varchar would be encoded as {{\[Varchar 27 byte0-26]}}.
A writer uses the type to drive the serialization. A reader uses the type to
drive deserialization.
Note that the type field must include a special marker for nulls. Today, the
union type uses 0 to indicate a null value. (Note that, in a union and variant,
a null value is not a null of some type, both the type and value are null.)
That form should be used in the variant representation as well. But, note that
the 0 value in the MajorType enum is not Null but is instead Late. This is an
unpleasant messiness that the union (and variant )encoding must handle.
An offset vector provides the location of each field, as is done with
variable-length vectors today.
The result is huge compaction
[jira] [Commented] (DRILL-4092) Support for INTERSECT
[
https://issues.apache.org/jira/browse/DRILL-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248982#comment-16248982
]
Khurram Faraaz commented on DRILL-4092:
---
[~prasadns14]
{noformat}
Here are some links to INTERSECT set operator in existing DBMSs
(1)
https://www.ibm.com/support/knowledgecenter/en/SSGU8G_12.1.0/com.ibm.sqls.doc/ids_sqs_1834.htm
(2) https://docs.oracle.com/cd/B19306_01/server.102/b14200/queries004.htm
(3) https://www.postgresql.org/docs/9.6/static/sql-select.html
{noformat}
> Support for INTERSECT
> --
>
> Key: DRILL-4092
> URL: https://issues.apache.org/jira/browse/DRILL-4092
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Victoria Markman
>Assignee: Prasad Nagaraj Subramanya
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5955) Revisit Union Vectors
Paul Rogers created DRILL-5955:
--
Summary: Revisit Union Vectors
Key: DRILL-5955
URL: https://issues.apache.org/jira/browse/DRILL-5955
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.11.0
Reporter: Paul Rogers
Drill supports a “Union Vector” type that allows a single column to hold values
of multiple types. Conceptually, each column value is a (type, value) pair. For
example, row 0 might be an Int, row 1 a Varchar and row 2 a NULL value.
The name refers to a C “union” in which the same bit of memory is used to
represent one of a set of defined types.
Drill implements the union vector a bit like a map: as a collection of typed
vectors. Each value is keyed by type. The result is that a union vector is more
like a C “struct” than a C “union”: every vector takes space, but only one of
the vectors is used for each row. For the example above, the union vector
contains an Int vector, a Varchar vector and a type vector. For each row,
either the Int or the Varchar is used. For NULL values, neither vector is used.
h4. Memory Footprint Concerns
The current representation, despite its name, makes very inefficient use of
memory because it requires the sum of the storage for each included type. (That
is, if we store 1000 rows, we need 1000 slots for integers, another 1000 for
Varchar and yet another 1000 for the type vector.)
Drill poorly supports the union type. One operator that does support it is the
sort. If the union type is enabled, and the sort sees a schema change, the sort
will create a new union vector that combines the two types. The result is a
sudden, unplanned increase in memory usage. Since the sort can buffer many
hundreds of batches, this unplanned memory increase can cause the sort to run
out of memory.
h4. Muddy Semantics
The union vector is closely tied with the List vector: a list vector is,
essentially, an array of unions. (See DRILL-). The list type is used to
model JSON in which a list can hold anything: another list, an object or
scalars. For this reason, the union vector also can hold any type. And, indeed,
it can hold a union of any of these types: a Map and an Int, or a List and a
Map.
Drill is a relational, SQL-based tool. Work is required to bring non-relational
structures into Drill. As discussed below, a union of scalars can be made to
work. But, a union of structured types (lists, arrays or Maps) makes no sense.
h4. High Complexity
The union vector, as implemented is quite complex. It contains member variables
for every other vector type (except, strangely, the decimal types.) Access to
typed members is by type-specific methods, meaning that the client code must
include a separate call for every type, resulting in very complex client code.
The complexity allowed the union type to be made to work, but causes this one
type to consume a disproportionate amount of the vector and client code.
h4. Proposed Revision to Structure: The Variant Type
Given the above, we can now present the proposed changes. First let us
recognize that a union vector need not hold structured types; there are other
solutions as discussed in DRILL-. This leaves the union vector to hold just
scalars.
h4. Proposed Revision to Storage
This in turn lets us adopt the [Variant
type|https://en.wikipedia.org/wiki/Variant_type] originally introduced in
Visual Basic. Variant “is a tagged union that can be used to represent any
other data type”. The Variant type was designed to be compact by building on
the idea of a tagged union in C.
{code}
struct {
int tag; // type
union {
int intValue;
long longValue;
…
}
}
{code}
When implemented as a vector, the format could consume just a single
variable-width vector with each entry of the form: {{\[type value]}}. The
vector is simply a sequence of these (type, value) pairs.
The type is a single-byte that encodes the MinorType that describes the value.
That is, the type byte is like the existing type vector, but stored in the same
location as the data. The data is simply the serialized format of data. (Four
bytes for an Int, 8 bytes for a Float8 and so on.)
Variable-width types require an extra field: the type field: {{\[type length
value]}}. For example, a Varchar would be encoded as {{\[Varchar 27 byte0-26]}}.
A writer uses the type to drive the serialization. A reader uses the type to
drive deserialization.
Note that the type field must include a special marker for nulls. Today, the
union type uses 0 to indicate a null value. (Note that, in a union and variant,
a null value is not a null of some type, both the type and value are null.)
That form should be used in the variant representation as well. But, note that
the 0 value in the MajorType enum is not Null but is instead Late. This is an
unpleasant messiness that the union (and variant )encoding must
