Re: JSON reader enhancement

Hi Paul, I have added some comments to DRILL-5974. Perhaps better to discuss in the JIRA for future referencing. -Aman On Sat, Nov 18, 2017 at 7:33 PM, Paul Rogers wrote: > Hi Arina, > > The proposal is to represent 2D arrays as a string (using the original, > unparsed

[GitHub] drill pull request #1042: DRILL-5261: Expose REST endpoint in zookeeper

GitHub user xhochy opened a pull request: https://github.com/apache/drill/pull/1042 DRILL-5261: Expose REST endpoint in zookeeper You can merge this pull request into a Git repository by running: $ git pull https://github.com/xhochy/drill DRILL-5261 Alternatively you can

[GitHub] drill issue #1042: DRILL-5261: Expose REST endpoint in zookeeper

Github user xhochy commented on the issue: https://github.com/apache/drill/pull/1042 This still needs the protobufs to be re-compiled for C++, I would do this once someone signals me that I have taken the correct approach. ---

Re: JSON reader enhancement

I don't see the value with this suggestion. It isn't going to make things much better since the user will be totally stunned when the structure doesn't come through as an array. A bigger issue is the fact that elements of maps aren't marked correctly as missing. That means that if I have these

[GitHub] drill issue #1030: DRILL-5941: Skip header / footer improvements for Hive st...

Github user ppadma commented on the issue: https://github.com/apache/drill/pull/1030 @arina-ielchiieva I am concerned about performance impact by grouping all splits in a single reader (essentially, not parallelizing at all). Wondering if it is possible to do this way: During

[jira] [Created] (DRILL-5977) predicate pushdown support kafkaMsgOffset

B Anil Kumar created DRILL-5977: --- Summary: predicate pushdown support kafkaMsgOffset Key: DRILL-5977 URL: https://issues.apache.org/jira/browse/DRILL-5977 Project: Apache Drill Issue Type:

Re: JSON reader enhancement

I can't speak to all use cases, but the variant map is very important when dealing with JSON data that has changed schema over time. In order to determine which version my data is, I need to know which fields are present in a map. Last time I looked, all that happened is that all films for set to

[jira] [Created] (DRILL-5976) Kafka MessageReader config optimization

B Anil Kumar created DRILL-5976: --- Summary: Kafka MessageReader config optimization Key: DRILL-5976 URL: https://issues.apache.org/jira/browse/DRILL-5976 Project: Apache Drill Issue Type:

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151890058 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/decoders/MessageReader.java --- @@ -0,0 +1,45 @@ +/* + *

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151890101 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaSubScan.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151888697 --- Diff: contrib/storage-kafka/README.md --- @@ -0,0 +1,230 @@ +# Drill Kafka Plugin + +Drill kafka storage plugin allows you to perform

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151890205 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaRecordReader.java --- @@ -0,0 +1,178 @@ +/** + * Licensed

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151890267 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaStoragePlugin.java --- @@ -0,0 +1,100 @@ +/* + * Licensed

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151890077 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaSubScan.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151890085 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaSubScan.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151890087 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaSubScan.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151890052 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/decoders/MessageReaderFactory.java --- @@ -0,0 +1,56 @@ +/* + *

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151890096 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaSubScan.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151890105 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaSubScan.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the

[GitHub] drill pull request #1027: DRILL-4779 : Kafka storage plugin

Github user akumarb2010 commented on a diff in the pull request: https://github.com/apache/drill/pull/1027#discussion_r151889218 --- Diff: contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/decoders/MessageReaderFactory.java --- @@ -0,0 +1,56 @@ +/* + *

Re: JSON reader enhancement

Hi Ted, Thanks for the suggestions. To handle nested lists correctly, we need Drill’s List data type, which uses Drill’s Union data type. (The List type is really mostly just a repeated Union, and so needs union support.) But, the union type disabled by default. The case I was trying to