[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[
https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468284#comment-16468284
]
ASF GitHub Bot commented on DRILL-6348:
---
vrozov commented on a change in pull request #1237: DRILL-6348: Fixed code so
that Unordered Receiver reports its memory …
URL: https://github.com/apache/drill/pull/1237#discussion_r186924024
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractDataCollector.java
##
@@ -116,4 +120,16 @@ public void close() throws Exception {
AutoCloseables.close(buffers);
}
+ /** {@inheritDoc} */
+ @Override
+ public BufferAllocator getAllocator() {
+return this.ownerAllocator;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void setAllocator(BufferAllocator allocator) {
+this.ownerAllocator = allocator;
Review comment:
Can passed allocator be `null`? If yes, `ownerAllocator` should not probably
be overwritten. If no, please add an `assert` or better `Preconditions` check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unordered Receiver does not report its memory usage
> ---
>
> Key: DRILL-6348
> URL: https://issues.apache.org/jira/browse/DRILL-6348
> Project: Apache Drill
> Issue Type: Task
> Components: Execution - Flow
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Drill Profile functionality doesn't show any memory usage for the
> Unordered Receiver operator. This is problematic when analyzing OOM
> conditions since we cannot account for all of a query memory usage. This Jira
> is to fix memory reporting for the Unordered Receiver operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5913) DrillReduceAggregatesRule mixed the same functions of the same inputRef which have different dataTypes
[
https://issues.apache.org/jira/browse/DRILL-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468197#comment-16468197
]
ASF GitHub Bot commented on DRILL-5913:
---
weijietong commented on issue #1016: DRILL-5913:solve the mixed processing of
same functions with same inputRefs but di…
URL: https://github.com/apache/drill/pull/1016#issuecomment-387589324
@vvysotskyi do you make sure that two same name aggs with the same input ref
but different data types should never happen here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DrillReduceAggregatesRule mixed the same functions of the same inputRef which
> have different dataTypes
> ---
>
> Key: DRILL-5913
> URL: https://issues.apache.org/jira/browse/DRILL-5913
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>Priority: Major
>
> sample query:
> {code:java}
> select stddev_samp(cast(employee_id as int)) as col1, sum(cast(employee_id as
> int)) as col2 from cp.`employee.json`
> {code}
> error info:
> {code:java}
> org.apache.drill.exec.rpc.RpcException:
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> AssertionError: Type mismatch:
> rel rowtype:
> RecordType(INTEGER $f0, INTEGER $f1, BIGINT NOT NULL $f2, INTEGER $f3) NOT
> NULL
> equivRel rowtype:
> RecordType(INTEGER $f0, INTEGER $f1, BIGINT NOT NULL $f2, BIGINT $f3) NOT NULL
> [Error Id: f5114e62-a57b-46b1-afe8-ae652f390896 on localhost:31010]
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: Internal error: Error while applying rule
> DrillReduceAggregatesRule, args
> [rel#29:LogicalAggregate.NONE.ANY([]).[](input=rel#28:Subset#3.NONE.ANY([]).[],group={},agg#0=SUM($1),agg#1=SUM($0),agg#2=COUNT($0),agg#3=$SUM0($0))]
> org.apache.drill.exec.work.foreman.Foreman.run():294
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745
> Caused By (java.lang.AssertionError) Internal error: Error while applying
> rule DrillReduceAggregatesRule, args
> [rel#29:LogicalAggregate.NONE.ANY([]).[](input=rel#28:Subset#3.NONE.ANY([]).[],group={},agg#0=SUM($1),agg#1=SUM($0),agg#2=COUNT($0),agg#3=$SUM0($0))]
> org.apache.calcite.util.Util.newInternal():792
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():811
> {code}
> The reason is that stddev_samp(cast(employee_id as int)) will be reduced as
> sum($0) ,sum($1) ,count($0) while the sum(cast(employee_id as int)) will be
> reduced as sum0($0) by the DrillReduceAggregatesRule's first time matching.
> The second time's matching will reduce stddev_samp's sum($0) to sum0($0) too
> . But this sum0($0) 's data type is different from the first time's sum0($0)
> : one is integer ,the other is bigint . But Calcite's addAggCall method treat
> them as the same by ignoring their data type. This leads to the bigint
> sum0($0) be replaced by the integer sum0($0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468180#comment-16468180 ] ASF GitHub Bot commented on DRILL-6348: --- sachouche commented on issue #1237: DRILL-6348: Fixed code so that Unordered Receiver reports its memory … URL: https://github.com/apache/drill/pull/1237#issuecomment-387586401 @vrozov, made the changes but note that I had to change the DataCollector IF visibility to public; as previously indicated, this IF was used only within the exec.work.batch package. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5913) DrillReduceAggregatesRule mixed the same functions of the same inputRef which have different dataTypes
[
https://issues.apache.org/jira/browse/DRILL-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468168#comment-16468168
]
ASF GitHub Bot commented on DRILL-5913:
---
weijietong commented on issue #1016: DRILL-5913:solve the mixed processing of
same functions with same inputRefs but di…
URL: https://github.com/apache/drill/pull/1016#issuecomment-387584681
I have noticed the return type inference codes. The return type is right. To
current implementation, if there are two agg calls with the same inputs but
truly with different data types,it will definitely go to choose wrong agg call
for reuse and cause potential errors.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DrillReduceAggregatesRule mixed the same functions of the same inputRef which
> have different dataTypes
> ---
>
> Key: DRILL-5913
> URL: https://issues.apache.org/jira/browse/DRILL-5913
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>Priority: Major
>
> sample query:
> {code:java}
> select stddev_samp(cast(employee_id as int)) as col1, sum(cast(employee_id as
> int)) as col2 from cp.`employee.json`
> {code}
> error info:
> {code:java}
> org.apache.drill.exec.rpc.RpcException:
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> AssertionError: Type mismatch:
> rel rowtype:
> RecordType(INTEGER $f0, INTEGER $f1, BIGINT NOT NULL $f2, INTEGER $f3) NOT
> NULL
> equivRel rowtype:
> RecordType(INTEGER $f0, INTEGER $f1, BIGINT NOT NULL $f2, BIGINT $f3) NOT NULL
> [Error Id: f5114e62-a57b-46b1-afe8-ae652f390896 on localhost:31010]
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: Internal error: Error while applying rule
> DrillReduceAggregatesRule, args
> [rel#29:LogicalAggregate.NONE.ANY([]).[](input=rel#28:Subset#3.NONE.ANY([]).[],group={},agg#0=SUM($1),agg#1=SUM($0),agg#2=COUNT($0),agg#3=$SUM0($0))]
> org.apache.drill.exec.work.foreman.Foreman.run():294
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745
> Caused By (java.lang.AssertionError) Internal error: Error while applying
> rule DrillReduceAggregatesRule, args
> [rel#29:LogicalAggregate.NONE.ANY([]).[](input=rel#28:Subset#3.NONE.ANY([]).[],group={},agg#0=SUM($1),agg#1=SUM($0),agg#2=COUNT($0),agg#3=$SUM0($0))]
> org.apache.calcite.util.Util.newInternal():792
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():811
> {code}
> The reason is that stddev_samp(cast(employee_id as int)) will be reduced as
> sum($0) ,sum($1) ,count($0) while the sum(cast(employee_id as int)) will be
> reduced as sum0($0) by the DrillReduceAggregatesRule's first time matching.
> The second time's matching will reduce stddev_samp's sum($0) to sum0($0) too
> . But this sum0($0) 's data type is different from the first time's sum0($0)
> : one is integer ,the other is bigint . But Calcite's addAggCall method treat
> them as the same by ignoring their data type. This leads to the bigint
> sum0($0) be replaced by the integer sum0($0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6386) Disallow Unused Imports In Checkstyle
[ https://issues.apache.org/jira/browse/DRILL-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468141#comment-16468141 ] ASF GitHub Bot commented on DRILL-6386: --- vrozov commented on a change in pull request #1252: DRILL-6386: Disallowed unused imports and removed them. URL: https://github.com/apache/drill/pull/1252#discussion_r186901891 ## File path: src/main/resources/checkstyle-config.xml ## @@ -27,7 +27,11 @@ - + + + Review comment: There is already 'AvoidStarImport' rule. Please change it to use defaults. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Disallow Unused Imports In Checkstyle > - > > Key: DRILL-6386 > URL: https://issues.apache.org/jira/browse/DRILL-6386 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6386) Disallow Unused Imports In Checkstyle
[ https://issues.apache.org/jira/browse/DRILL-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468108#comment-16468108 ] ASF GitHub Bot commented on DRILL-6386: --- ilooner commented on issue #1252: DRILL-6386: Disallowed unused imports and removed them. URL: https://github.com/apache/drill/pull/1252#issuecomment-387574479 @vrozov Disallowed start imports @arina-ielchiieva Address comments questions Please let me know if you have anymore comments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Disallow Unused Imports In Checkstyle > - > > Key: DRILL-6386 > URL: https://issues.apache.org/jira/browse/DRILL-6386 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6386) Disallow Unused Imports In Checkstyle
[ https://issues.apache.org/jira/browse/DRILL-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468109#comment-16468109 ] ASF GitHub Bot commented on DRILL-6386: --- ilooner commented on issue #1252: DRILL-6386: Disallowed unused imports and removed them. URL: https://github.com/apache/drill/pull/1252#issuecomment-387574479 @vrozov Disallowed start imports @arina-ielchiieva Addressed comments and questions Please let me know if you have anymore comments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Disallow Unused Imports In Checkstyle > - > > Key: DRILL-6386 > URL: https://issues.apache.org/jira/browse/DRILL-6386 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5696) Change default compiler strategy
[ https://issues.apache.org/jira/browse/DRILL-5696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468088#comment-16468088 ] Kunal Khatua commented on DRILL-5696: - [~weijie] I'm not able to repro the issue with the latest Drill code. Do you still see the issue? > Change default compiler strategy > > > Key: DRILL-5696 > URL: https://issues.apache.org/jira/browse/DRILL-5696 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Codegen >Affects Versions: 1.9.0, 1.10.0, 1.11.0 >Reporter: weijie.tong >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > at our production ,when we have more than 20 agg expression, the compile > time is high using the default janino. but when changed to jdk compiler,we > gain fewer compile time than the janino one. Our product jdk version is 1.8. > So the default one should be JDK , if user's jdk version is upper than 1.7. > We should add another check condition to the ClassCompilerSelector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6393) Radians should take an argument (x)
Robert Hou created DRILL-6393:
-
Summary: Radians should take an argument (x)
Key: DRILL-6393
URL: https://issues.apache.org/jira/browse/DRILL-6393
Project: Apache Drill
Issue Type: Bug
Components: Documentation
Affects Versions: 1.13.0
Reporter: Robert Hou
Assignee: Bridget Bevens
Fix For: 1.14.0
The radians function is missing an argument on this webpage:
https://drill.apache.org/docs/math-and-trig/
The table has this information:
{noformat}
RADIANS FLOAT8 Converts x degress to radians.
{nformat}
It should be:
{noformat}
RADIANS(x) FLOAT8 Converts x degrees to radians.
{noformat}
Also, degress is mis-spelled. It should be degrees.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6393) Radians should take an argument (x)
[
https://issues.apache.org/jira/browse/DRILL-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Hou updated DRILL-6393:
--
Description:
The radians function is missing an argument on this webpage:
https://drill.apache.org/docs/math-and-trig/
The table has this information:
{noformat}
RADIANS FLOAT8 Converts x degress to radians.
{noformat}
It should be:
{noformat}
RADIANS(x) FLOAT8 Converts x degrees to radians.
{noformat}
Also, degress is mis-spelled. It should be degrees.
was:
The radians function is missing an argument on this webpage:
https://drill.apache.org/docs/math-and-trig/
The table has this information:
{noformat}
RADIANS FLOAT8 Converts x degress to radians.
{nformat}
It should be:
{noformat}
RADIANS(x) FLOAT8 Converts x degrees to radians.
{noformat}
Also, degress is mis-spelled. It should be degrees.
> Radians should take an argument (x)
> ---
>
> Key: DRILL-6393
> URL: https://issues.apache.org/jira/browse/DRILL-6393
> Project: Apache Drill
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.13.0
>Reporter: Robert Hou
>Assignee: Bridget Bevens
>Priority: Major
> Fix For: 1.14.0
>
>
> The radians function is missing an argument on this webpage:
>https://drill.apache.org/docs/math-and-trig/
> The table has this information:
> {noformat}
> RADIANS FLOAT8 Converts x degress to radians.
> {noformat}
> It should be:
> {noformat}
> RADIANS(x)FLOAT8 Converts x degrees to radians.
> {noformat}
> Also, degress is mis-spelled. It should be degrees.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6392) System option exec.max_hash_table_size is completely unused. We should remove it.
Timothy Farkas created DRILL-6392: - Summary: System option exec.max_hash_table_size is completely unused. We should remove it. Key: DRILL-6392 URL: https://issues.apache.org/jira/browse/DRILL-6392 Project: Apache Drill Issue Type: Improvement Reporter: Timothy Farkas -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468023#comment-16468023
]
Paul Rogers commented on DRILL-6390:
This is, unfortunately, a known limitation of Drill's JSON support. If your
fields were at the top level, you could work around this by using a CAST. But,
the trick does not work in nested structures (without also re-projecting the
column to the top level.) There is another ticket open with the gory details.
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {code:java}
> SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {code}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {code:java}
> {
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
> "application":
> {
> "id": "some_id",
> "name": "Some App",
> "object": "application"
> },
> "application_fee": null,
> "amount": 5
> }
> ],
> "has_more": false,
> "object": "list",
> "url": "/v1/charges"
> }
> {code}
> I ran some further tests on the query and found out that the query only fails
> at object "application_fee", which is null in both arrays of object "data".
> Which might explain why I never ran into this error in the other datasets (no
> object arrays there are null).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[ https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467923#comment-16467923 ] Sorabh Hamirwasia commented on DRILL-6255: -- Earlier solution to submit control message doesn't look complete since it was taking care of handling Fragment initialization and status update message from remote fragment only. Other message like UnpauseFragment, CancelFragment, etc was not taken care of. With the current solution all the control channel messages will be taken care to submit it locally rather than over a network channel. A new LocalControlConnectionManager is created for this which is accessed inside ControlTunnel. When a tunnel is created for a Drillbit Endpoint such that it's the local Drillbit then internally it just bypasses the connection creation process and instead submits the request and return the response locally. > Drillbit while sending control message to itself creates a connection instead > of submitting locally > --- > > Key: DRILL-6255 > URL: https://issues.apache.org/jira/browse/DRILL-6255 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.12.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.14.0 > > > With the new shutdown feature introduced in 1.12, there is a state introduced > in DrillbitEndpoint. Due to this the equality check happening > [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256] > will result in false and hence the fragments supposed to be scheduled on > Foreman will be treated as remote fragments and a connection will be created > to schedule it. The equality check is false because localEndpoint state is > STARTUP whereas state in assigned Drillbit is ONLINE. > I guess now we should update the equality check to verify just for address > and control port to be same between assigned and local Drillbit endpoint. A > test can be added for this based on _sys.memory_ table since that guarantees > scheduling minor fragments on each Drillbit node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[ https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6255: - Reviewer: (was: Arina Ielchiieva) > Drillbit while sending control message to itself creates a connection instead > of submitting locally > --- > > Key: DRILL-6255 > URL: https://issues.apache.org/jira/browse/DRILL-6255 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.12.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.14.0 > > > With the new shutdown feature introduced in 1.12, there is a state introduced > in DrillbitEndpoint. Due to this the equality check happening > [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256] > will result in false and hence the fragments supposed to be scheduled on > Foreman will be treated as remote fragments and a connection will be created > to schedule it. The equality check is false because localEndpoint state is > STARTUP whereas state in assigned Drillbit is ONLINE. > I guess now we should update the equality check to verify just for address > and control port to be same between assigned and local Drillbit endpoint. A > test can be added for this based on _sys.memory_ table since that guarantees > scheduling minor fragments on each Drillbit node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[ https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6255: - Reviewer: Arina Ielchiieva > Drillbit while sending control message to itself creates a connection instead > of submitting locally > --- > > Key: DRILL-6255 > URL: https://issues.apache.org/jira/browse/DRILL-6255 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.12.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.14.0 > > > With the new shutdown feature introduced in 1.12, there is a state introduced > in DrillbitEndpoint. Due to this the equality check happening > [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256] > will result in false and hence the fragments supposed to be scheduled on > Foreman will be treated as remote fragments and a connection will be created > to schedule it. The equality check is false because localEndpoint state is > STARTUP whereas state in assigned Drillbit is ONLINE. > I guess now we should update the equality check to verify just for address > and control port to be same between assigned and local Drillbit endpoint. A > test can be added for this based on _sys.memory_ table since that guarantees > scheduling minor fragments on each Drillbit node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[ https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467914#comment-16467914 ] ASF GitHub Bot commented on DRILL-6255: --- sohami commented on issue #1253: DRILL-6255: Drillbit while sending control message to itself creates … URL: https://github.com/apache/drill/pull/1253#issuecomment-387526941 @arina-ielchiieva / @parthchandra - Please help to review this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drillbit while sending control message to itself creates a connection instead > of submitting locally > --- > > Key: DRILL-6255 > URL: https://issues.apache.org/jira/browse/DRILL-6255 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.12.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.14.0 > > > With the new shutdown feature introduced in 1.12, there is a state introduced > in DrillbitEndpoint. Due to this the equality check happening > [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256] > will result in false and hence the fragments supposed to be scheduled on > Foreman will be treated as remote fragments and a connection will be created > to schedule it. The equality check is false because localEndpoint state is > STARTUP whereas state in assigned Drillbit is ONLINE. > I guess now we should update the equality check to verify just for address > and control port to be same between assigned and local Drillbit endpoint. A > test can be added for this based on _sys.memory_ table since that guarantees > scheduling minor fragments on each Drillbit node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[ https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467913#comment-16467913 ] ASF GitHub Bot commented on DRILL-6255: --- sohami commented on issue #1253: DRILL-6255: Drillbit while sending control message to itself creates … URL: https://github.com/apache/drill/pull/1253#issuecomment-387526941 @arina-ielchiieva - Please help to review this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drillbit while sending control message to itself creates a connection instead > of submitting locally > --- > > Key: DRILL-6255 > URL: https://issues.apache.org/jira/browse/DRILL-6255 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.12.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.14.0 > > > With the new shutdown feature introduced in 1.12, there is a state introduced > in DrillbitEndpoint. Due to this the equality check happening > [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256] > will result in false and hence the fragments supposed to be scheduled on > Foreman will be treated as remote fragments and a connection will be created > to schedule it. The equality check is false because localEndpoint state is > STARTUP whereas state in assigned Drillbit is ONLINE. > I guess now we should update the equality check to verify just for address > and control port to be same between assigned and local Drillbit endpoint. A > test can be added for this based on _sys.memory_ table since that guarantees > scheduling minor fragments on each Drillbit node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[ https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467912#comment-16467912 ] ASF GitHub Bot commented on DRILL-6255: --- sohami opened a new pull request #1253: DRILL-6255: Drillbit while sending control message to itself creates … URL: https://github.com/apache/drill/pull/1253 …a connection instead of submitting locally This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drillbit while sending control message to itself creates a connection instead > of submitting locally > --- > > Key: DRILL-6255 > URL: https://issues.apache.org/jira/browse/DRILL-6255 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.12.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.14.0 > > > With the new shutdown feature introduced in 1.12, there is a state introduced > in DrillbitEndpoint. Due to this the equality check happening > [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256] > will result in false and hence the fragments supposed to be scheduled on > Foreman will be treated as remote fragments and a connection will be created > to schedule it. The equality check is false because localEndpoint state is > STARTUP whereas state in assigned Drillbit is ONLINE. > I guess now we should update the equality check to verify just for address > and control port to be same between assigned and local Drillbit endpoint. A > test can be added for this based on _sys.memory_ table since that guarantees > scheduling minor fragments on each Drillbit node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-2674) Add user authenticator interface and PAM based implementation
[ https://issues.apache.org/jira/browse/DRILL-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467874#comment-16467874 ] Ravi Venugopal commented on DRILL-2674: --- any idea on how to do this on Windows ? > Add user authenticator interface and PAM based implementation > - > > Key: DRILL-2674 > URL: https://issues.apache.org/jira/browse/DRILL-2674 > Project: Apache Drill > Issue Type: Sub-task > Components: Client - JDBC >Reporter: Venki Korukanti >Assignee: Venki Korukanti >Priority: Major > Fix For: 0.9.0 > > Attachments: DRILL-2674-2.patch > > > Currently Drill doesn't authenticate users connecting through JDBC/ODBC > clients. > This JIRA is to define an interface of userName/password based authenticator > interface and provide a Pluggable Authentication Module (PAM) based > implementation of the user authenticator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6391) Drill filter pushdown for RDBMS storage
Ravi Venugopal created DRILL-6391: - Summary: Drill filter pushdown for RDBMS storage Key: DRILL-6391 URL: https://issues.apache.org/jira/browse/DRILL-6391 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Environment: Oracle 12c Apache Drill 1.13 Windows R12 Reporter: Ravi Venugopal While running SQL from Drill on Oracle RDBMS plugin, Drill uses its own QP than using a Filter pushdown to get the Oracle optimizer's output. Effect, Drill runs at 60 seconds vs Oracle at 15 seconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[
https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467821#comment-16467821
]
ASF GitHub Bot commented on DRILL-6348:
---
vrozov commented on a change in pull request #1237: DRILL-6348: Fixed code so
that Unordered Receiver reports its memory …
URL: https://github.com/apache/drill/pull/1237#discussion_r186831642
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/IncomingBuffers.java
##
@@ -129,6 +134,10 @@ public int getRemainingRequired() {
return collectorMap.get(senderMajorFragmentId).getBuffers();
}
+ public void setOprAllocator(int senderMajorFragmentId, BufferAllocator
oprAllocator) {
Review comment:
It can be done using annotations (I do not suggest to introduce such
annotations now).
I don't see a need to hide DataCollector as long as HashMap is not exposed
directly or concrete type of a class that implements DataCollector is exposed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unordered Receiver does not report its memory usage
> ---
>
> Key: DRILL-6348
> URL: https://issues.apache.org/jira/browse/DRILL-6348
> Project: Apache Drill
> Issue Type: Task
> Components: Execution - Flow
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Drill Profile functionality doesn't show any memory usage for the
> Unordered Receiver operator. This is problematic when analyzing OOM
> conditions since we cannot account for all of a query memory usage. This Jira
> is to fix memory reporting for the Unordered Receiver operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467817#comment-16467817 ] ASF GitHub Bot commented on DRILL-6348: --- vrozov commented on a change in pull request #1237: DRILL-6348: Fixed code so that Unordered Receiver reports its memory … URL: https://github.com/apache/drill/pull/1237#discussion_r186829797 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/DataCollector.java ## @@ -28,4 +29,16 @@ public RawBatchBuffer[] getBuffers(); public int getTotalIncomingFragments(); public void close() throws Exception; + /** + * Allows an operator (e.g., receiver) to attach its buffer allocator to this Data Collector so that + * incoming batches are owned by the operator instead of the parent (fragment). Note + * that this operation is optional; the fragment allocator is used in case no operator invokes this method. + * + * @param oprAllocator operator buffer allocator + */ + void setOprAllocator(BufferAllocator oprAllocator); Review comment: IMO, get/setAllocator is good enough. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[ https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467811#comment-16467811 ] ASF GitHub Bot commented on DRILL-5270: --- ilooner commented on issue #1250: DRILL-5270: Improve loading of profiles listing in the WebUI URL: https://github.com/apache/drill/pull/1250#issuecomment-387502051 @kkhatua I'm still not sure why you want to override the definition of oldest? Why is the default LRU eviction policy not sufficient? If you need an ordered list of keys for the cache you can accomplish this with the Guava cache by adding a key to a TreeSet when the Loader is called, and removing a key from a TreeSet when the Removal Listener is called. My main concern is that implementing our own cache creates complexity and opens up the possibility for bugs. Whereas a pre-existing cache is already debugged and tested for us. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve loading of profiles listing in the WebUI > > > Key: DRILL-5270 > URL: https://issues.apache.org/jira/browse/DRILL-5270 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > Currently, as the number of profiles increase, we reload the same list of > profiles from the FS. > An ideal improvement would be to detect if there are any new profiles and > only reload from the disk then. Otherwise, a cached list is sufficient. > For a directory of 280K profiles, the load time is close to 6 seconds on a 32 > core server. With the caching, we can get it down to as much as a few > milliseconds. > To render the cache as invalid, we inspect the last modified time of the > directory to confirm whether a reload is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6386) Disallow Unused Imports In Checkstyle
[ https://issues.apache.org/jira/browse/DRILL-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467781#comment-16467781 ] ASF GitHub Bot commented on DRILL-6386: --- ilooner commented on issue #1252: DRILL-6386: Disallowed unused imports and removed them. URL: https://github.com/apache/drill/pull/1252#issuecomment-387492658 @arina-ielchiieva Addressing your questions: 1. This doesn't cover import order. From what I vaguely remember with Apache Apex, import order was a highly controversial topic. I want to avoid that discussion for now and get everybody comfortable with adding more checkstyle checks bit by bit. Once people are more comfortable with stricter checkstyle checks I am hoping a discussion about adding import order checks can reach a consensus quicker. 2. I will refactor into two commits as you suggested. 3. I wasn't planning on adding any documentation about this now since the checkstyle errors are pretty descriptive. However, I was planning on writing a Checkstyle.md which will explain how to import a checkstyle.xml into your IDE so that you can see checkstyle warnings within your IDE as well as autoformat your code according to Drill's checkstyle config. Chandni figured out all these things for Apache Apex, but I don't think it's documented anywhere. So I'll sync up with her and get it written down as part of a separate Jira. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Disallow Unused Imports In Checkstyle > - > > Key: DRILL-6386 > URL: https://issues.apache.org/jira/browse/DRILL-6386 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467759#comment-16467759 ] ASF GitHub Bot commented on DRILL-6348: --- sachouche commented on a change in pull request #1237: DRILL-6348: Fixed code so that Unordered Receiver reports its memory … URL: https://github.com/apache/drill/pull/1237#discussion_r186811609 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/DataCollector.java ## @@ -28,4 +29,16 @@ public RawBatchBuffer[] getBuffers(); public int getTotalIncomingFragments(); public void close() throws Exception; + /** + * Allows an operator (e.g., receiver) to attach its buffer allocator to this Data Collector so that + * incoming batches are owned by the operator instead of the parent (fragment). Note + * that this operation is optional; the fragment allocator is used in case no operator invokes this method. + * + * @param oprAllocator operator buffer allocator + */ + void setOprAllocator(BufferAllocator oprAllocator); Review comment: Agreed. What about get/setOwningAllocator? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[
https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467758#comment-16467758
]
ASF GitHub Bot commented on DRILL-6348:
---
sachouche commented on a change in pull request #1237: DRILL-6348: Fixed code
so that Unordered Receiver reports its memory …
URL: https://github.com/apache/drill/pull/1237#discussion_r186814021
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/IncomingBuffers.java
##
@@ -129,6 +134,10 @@ public int getRemainingRequired() {
return collectorMap.get(senderMajorFragmentId).getBuffers();
}
+ public void setOprAllocator(int senderMajorFragmentId, BufferAllocator
oprAllocator) {
Review comment:
I don't mind doing so if you think Collector is a top-level interface. I
wish there is a better way to indicate which class is an internal framework
class vs a public one safe for all operators to access. If you look closely at
the code, the DataCollector interface was created to allow multiple
implementations; the Drill code exposed the RawFragmentBatchProvider as the
main IF for operators to consume incoming data.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unordered Receiver does not report its memory usage
> ---
>
> Key: DRILL-6348
> URL: https://issues.apache.org/jira/browse/DRILL-6348
> Project: Apache Drill
> Issue Type: Task
> Components: Execution - Flow
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Drill Profile functionality doesn't show any memory usage for the
> Unordered Receiver operator. This is problematic when analyzing OOM
> conditions since we cannot account for all of a query memory usage. This Jira
> is to fix memory reporting for the Unordered Receiver operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467742#comment-16467742
]
ASF GitHub Bot commented on DRILL-6321:
---
vrozov commented on a change in pull request #1224: DRILL-6321: Customize
Drill's conformance. Allow support to APPLY key…
URL: https://github.com/apache/drill/pull/1224#discussion_r186810609
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillConformance.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import org.apache.calcite.sql.validate.SqlConformanceEnum;
+import org.apache.calcite.sql.validate.SqlDelegatingConformance;
+
+/**
+ * Drill's SQL conformance is SqlConformanceEnum.DEFAULT except for method
isApplyAllowed().
+ * Since Drill is going to allow OUTER APPLY and CROSS APPLY to allow each row
from left child of Join
+ * to join with output of right side (sub-query or table function that will be
invoked for each row).
+ * Refer to DRILL-5999 for more information.
+ */
+public class DrillConformance extends SqlDelegatingConformance {
Review comment:
Premature optimization for the future is the evil of the software
development. It is possible to override any behavior using the anonymous class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lateral Join: Planning changes - enable submitting physical plan
>
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
> Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Implement changes to enable submitting a physical plan containing lateral and
> unnest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[ https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467737#comment-16467737 ] ASF GitHub Bot commented on DRILL-5270: --- kkhatua commented on issue #1250: DRILL-5270: Improve loading of profiles listing in the WebUI URL: https://github.com/apache/drill/pull/1250#issuecomment-387483281 I did consider using the Gauva Cache initially, but I could not figure out how to specify the eviction policy based on the profile name. Guava provides a mechanism to limit the cache size and evict the oldest entry, but I wanted to override the mechanism that defines 'oldest'. Lastly, the TreeSet allows us to access the elements in a sorted order, which seemed missing in Guava. Do you think it makes the code cleaner if I were to extract the mechanism into a separate implementation of this 'cache' ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve loading of profiles listing in the WebUI > > > Key: DRILL-5270 > URL: https://issues.apache.org/jira/browse/DRILL-5270 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > Currently, as the number of profiles increase, we reload the same list of > profiles from the FS. > An ideal improvement would be to detect if there are any new profiles and > only reload from the disk then. Otherwise, a cached list is sufficient. > For a directory of 280K profiles, the load time is close to 6 seconds on a 32 > core server. With the caching, we can get it down to as much as a few > milliseconds. > To render the cache as invalid, we inspect the last modified time of the > directory to confirm whether a reload is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[ https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467733#comment-16467733 ] ASF GitHub Bot commented on DRILL-5270: --- kkhatua commented on issue #1250: DRILL-5270: Improve loading of profiles listing in the WebUI URL: https://github.com/apache/drill/pull/1250#issuecomment-387483281 I did consider using the Gauva Cache initially, but I could not figure out how to specify the eviction policy based on the profile name. Guava provides a mechanism to limit the cache size and evict the oldest entry, but I wanted to override the mechanism that defines 'oldest'. Lastly, the TreeSet allows us to access the elements in a sorted order, which seemed missing in Guava. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve loading of profiles listing in the WebUI > > > Key: DRILL-5270 > URL: https://issues.apache.org/jira/browse/DRILL-5270 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > Currently, as the number of profiles increase, we reload the same list of > profiles from the FS. > An ideal improvement would be to detect if there are any new profiles and > only reload from the disk then. Otherwise, a cached list is sufficient. > For a directory of 280K profiles, the load time is close to 6 seconds on a 32 > core server. With the caching, we can get it down to as much as a few > milliseconds. > To render the cache as invalid, we inspect the last modified time of the > directory to confirm whether a reload is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467723#comment-16467723 ] ASF GitHub Bot commented on DRILL-6348: --- vrozov commented on a change in pull request #1237: DRILL-6348: Fixed code so that Unordered Receiver reports its memory … URL: https://github.com/apache/drill/pull/1237#discussion_r186807111 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/IncomingBuffers.java ## @@ -103,8 +104,12 @@ public boolean batchArrived(final IncomingDataBatch incomingBatch) throws Fragme Arrays.toString(collectorMap.values().toArray(; } + // Use the Data Collector's buffer allocator if set, otherwise the fragment's one + BufferAllocator ownerAllocator = +collector.getOprAllocator() != null ? collector.getOprAllocator() : context.getAllocator(); Review comment: Consider moving this logic to `AbstractDataCollector.getAllocator()` or `AbstractDataCollector.setAllocator()`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467719#comment-16467719 ] ASF GitHub Bot commented on DRILL-6348: --- vrozov commented on a change in pull request #1237: DRILL-6348: Fixed code so that Unordered Receiver reports its memory … URL: https://github.com/apache/drill/pull/1237#discussion_r186804993 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/DataCollector.java ## @@ -28,4 +29,16 @@ public RawBatchBuffer[] getBuffers(); public int getTotalIncomingFragments(); public void close() throws Exception; + /** + * Allows an operator (e.g., receiver) to attach its buffer allocator to this Data Collector so that + * incoming batches are owned by the operator instead of the parent (fragment). Note + * that this operation is optional; the fragment allocator is used in case no operator invokes this method. + * + * @param oprAllocator operator buffer allocator + */ + void setOprAllocator(BufferAllocator oprAllocator); Review comment: I am not sure that `Opr` qualification is necessary. There is no assumption that it is an operator. It can be any `BufferAllocator`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[
https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467703#comment-16467703
]
ASF GitHub Bot commented on DRILL-6348:
---
vrozov commented on a change in pull request #1237: DRILL-6348: Fixed code so
that Unordered Receiver reports its memory …
URL: https://github.com/apache/drill/pull/1237#discussion_r186801073
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/IncomingBuffers.java
##
@@ -129,6 +134,10 @@ public int getRemainingRequired() {
return collectorMap.get(senderMajorFragmentId).getBuffers();
}
+ public void setOprAllocator(int senderMajorFragmentId, BufferAllocator
oprAllocator) {
Review comment:
`DataCollector` is a top-level interface.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unordered Receiver does not report its memory usage
> ---
>
> Key: DRILL-6348
> URL: https://issues.apache.org/jira/browse/DRILL-6348
> Project: Apache Drill
> Issue Type: Task
> Components: Execution - Flow
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Drill Profile functionality doesn't show any memory usage for the
> Unordered Receiver operator. This is problematic when analyzing OOM
> conditions since we cannot account for all of a query memory usage. This Jira
> is to fix memory reporting for the Unordered Receiver operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467701#comment-16467701 ] ASF GitHub Bot commented on DRILL-6348: --- sachouche commented on issue #1237: DRILL-6348: Fixed code so that Unordered Receiver reports its memory … URL: https://github.com/apache/drill/pull/1237#issuecomment-387474411 Implemented the changes suggested by @vrozov and fixed a memory leak warning message as the receive operators were closed before the buffers. Now closing the buffers before closing the operators. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6386) Disallow Unused Imports In Checkstyle
[ https://issues.apache.org/jira/browse/DRILL-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467697#comment-16467697 ] ASF GitHub Bot commented on DRILL-6386: --- vrozov commented on issue #1252: DRILL-6386: Disallowed unused imports and removed them. URL: https://github.com/apache/drill/pull/1252#issuecomment-387473728 In addition to disabling unused imports, it will be good to disable wildcard imports as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Disallow Unused Imports In Checkstyle > - > > Key: DRILL-6386 > URL: https://issues.apache.org/jira/browse/DRILL-6386 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467624#comment-16467624 ] ASF GitHub Bot commented on DRILL-6348: --- sachouche commented on a change in pull request #1237: DRILL-6348: Fixed code so that Unordered Receiver reports its memory … URL: https://github.com/apache/drill/pull/1237#discussion_r186782477 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/MergingReceiverCreator.java ## @@ -44,6 +44,11 @@ public MergingRecordBatch getBatch(ExecutorFragmentContext context, assert bufHolder != null : "IncomingBuffers must be defined for any place a receiver is declared."; RawBatchBuffer[] buffers = bufHolder.getBuffers(receiver.getOppositeMajorFragmentId()); -return new MergingRecordBatch(context, receiver, buffers); +MergingRecordBatch mergeReceiver = new MergingRecordBatch(context, receiver, buffers); + +// Register this operator's buffer allocator so that incoming buffers are owned by this allocator +bufHolder.setOprAllocator(receiver.getOppositeMajorFragmentId(), mergeReceiver.getOprAllocator()); Review comment: sure. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467625#comment-16467625 ] ASF GitHub Bot commented on DRILL-6348: --- sachouche commented on a change in pull request #1237: DRILL-6348: Fixed code so that Unordered Receiver reports its memory … URL: https://github.com/apache/drill/pull/1237#discussion_r186782554 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverCreator.java ## @@ -40,6 +40,11 @@ public UnorderedReceiverBatch getBatch(ExecutorFragmentContext context, Unordere RawBatchBuffer[] buffers = bufHolder.getBuffers(receiver.getOppositeMajorFragmentId()); assert buffers.length == 1; RawBatchBuffer buffer = buffers[0]; -return new UnorderedReceiverBatch(context, buffer, receiver); +UnorderedReceiverBatch receiverBatch = new UnorderedReceiverBatch(context, buffer, receiver); Review comment: sure. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[
https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467619#comment-16467619
]
ASF GitHub Bot commented on DRILL-6348:
---
sachouche commented on a change in pull request #1237: DRILL-6348: Fixed code
so that Unordered Receiver reports its memory …
URL: https://github.com/apache/drill/pull/1237#discussion_r186782290
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/IncomingBuffers.java
##
@@ -129,6 +134,10 @@ public int getRemainingRequired() {
return collectorMap.get(senderMajorFragmentId).getBuffers();
}
+ public void setOprAllocator(int senderMajorFragmentId, BufferAllocator
oprAllocator) {
Review comment:
I feel the Collector is an internal class (also used for synchronization).
Why exposed it if this is not necessary?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unordered Receiver does not report its memory usage
> ---
>
> Key: DRILL-6348
> URL: https://issues.apache.org/jira/browse/DRILL-6348
> Project: Apache Drill
> Issue Type: Task
> Components: Execution - Flow
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Drill Profile functionality doesn't show any memory usage for the
> Unordered Receiver operator. This is problematic when analyzing OOM
> conditions since we cannot account for all of a query memory usage. This Jira
> is to fix memory reporting for the Unordered Receiver operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5913) DrillReduceAggregatesRule mixed the same functions of the same inputRef which have different dataTypes
[
https://issues.apache.org/jira/browse/DRILL-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467536#comment-16467536
]
ASF GitHub Bot commented on DRILL-5913:
---
vvysotskyi commented on issue #1016: DRILL-5913:solve the mixed processing of
same functions with same inputRefs but di…
URL: https://github.com/apache/drill/pull/1016#issuecomment-387434799
@weijietong, the root cause of this error may have another reason. Drill has
its own rules to determine the return type for aggregate functions, and in the
most cases, it differs from Calcite rules. I suppose in some places were used
rules for Calcite, but should be used rules for Drill.
Regarding the creating new calls for the same agg call with the same inputs,
I suppose it will be ineffective to create the new calls only because return
types differ. The problem is that return type was chosen incorrectly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DrillReduceAggregatesRule mixed the same functions of the same inputRef which
> have different dataTypes
> ---
>
> Key: DRILL-5913
> URL: https://issues.apache.org/jira/browse/DRILL-5913
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>Priority: Major
>
> sample query:
> {code:java}
> select stddev_samp(cast(employee_id as int)) as col1, sum(cast(employee_id as
> int)) as col2 from cp.`employee.json`
> {code}
> error info:
> {code:java}
> org.apache.drill.exec.rpc.RpcException:
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> AssertionError: Type mismatch:
> rel rowtype:
> RecordType(INTEGER $f0, INTEGER $f1, BIGINT NOT NULL $f2, INTEGER $f3) NOT
> NULL
> equivRel rowtype:
> RecordType(INTEGER $f0, INTEGER $f1, BIGINT NOT NULL $f2, BIGINT $f3) NOT NULL
> [Error Id: f5114e62-a57b-46b1-afe8-ae652f390896 on localhost:31010]
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: Internal error: Error while applying rule
> DrillReduceAggregatesRule, args
> [rel#29:LogicalAggregate.NONE.ANY([]).[](input=rel#28:Subset#3.NONE.ANY([]).[],group={},agg#0=SUM($1),agg#1=SUM($0),agg#2=COUNT($0),agg#3=$SUM0($0))]
> org.apache.drill.exec.work.foreman.Foreman.run():294
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745
> Caused By (java.lang.AssertionError) Internal error: Error while applying
> rule DrillReduceAggregatesRule, args
> [rel#29:LogicalAggregate.NONE.ANY([]).[](input=rel#28:Subset#3.NONE.ANY([]).[],group={},agg#0=SUM($1),agg#1=SUM($0),agg#2=COUNT($0),agg#3=$SUM0($0))]
> org.apache.calcite.util.Util.newInternal():792
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():811
> {code}
> The reason is that stddev_samp(cast(employee_id as int)) will be reduced as
> sum($0) ,sum($1) ,count($0) while the sum(cast(employee_id as int)) will be
> reduced as sum0($0) by the DrillReduceAggregatesRule's first time matching.
> The second time's matching will reduce stddev_samp's sum($0) to sum0($0) too
> . But this sum0($0) 's data type is different from the first time's sum0($0)
> : one is integer ,the other is bigint . But Calcite's addAggCall method treat
> them as the same by ignoring their data type. This leads to the bigint
> sum0($0) be replaced by the integer sum0($0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6386) Disallow Unused Imports In Checkstyle
[ https://issues.apache.org/jira/browse/DRILL-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467473#comment-16467473 ] ASF GitHub Bot commented on DRILL-6386: --- arina-ielchiieva commented on issue #1252: DRILL-6386: Disallowed unused imports and removed them. URL: https://github.com/apache/drill/pull/1252#issuecomment-387421645 1. Will this cover imports order? 2. Should we have two commits: one by drill dev that fixes unused imports, another by Tim that adds unused commits check? 3. Will this information be added in PR-1251 or here to specific .md document? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Disallow Unused Imports In Checkstyle > - > > Key: DRILL-6386 > URL: https://issues.apache.org/jira/browse/DRILL-6386 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6249: Labels: ready-to-commit (was: ) > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5913) DrillReduceAggregatesRule mixed the same functions of the same inputRef which have different dataTypes
[
https://issues.apache.org/jira/browse/DRILL-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467360#comment-16467360
]
ASF GitHub Bot commented on DRILL-5913:
---
weijietong commented on issue #1016: DRILL-5913:solve the mixed processing of
same functions with same inputRefs but di…
URL: https://github.com/apache/drill/pull/1016#issuecomment-387387738
@vvysotskyi I have tested the jira issue sql on the current master and it
passed .But another new test case:
```
@Test
public void testDRILL5913_t() throws Exception {
test("select n_nationkey, stddev((case when ( bigint_col ) >0 then 1
else 0 end)) * 1.0 as col1, avg((case when ( bigint_col) >0 then 1 else 0
end)) * 1.0 as col2 from "
+ "( select n_name,n_nationkey, sum( n_regionkey) as bigint_col
from cp.`tpch/nation.parquet` group by n_name,n_nationkey ) t group by
n_nationkey");
}
```
will throw another Exception at Drill version 1.13 with Calcite version 1.15
but passed at current master. The exception message is:
```
Caused by: java.lang.AssertionError: Type mismatch:
rel rowtype:
RecordType(ANY n_nationkey, BIGINT $f1, BIGINT $f2, BIGINT NOT NULL $f3,
BIGINT $f4) NOT NULL
equivRel rowtype:
RecordType(ANY n_nationkey, BIGINT $f1, BIGINT $f2, BIGINT NOT NULL $f3,
BIGINT NOT NULL $f4) NOT NULL
```
All of the main reason is that DrillReduceAggregationRule.reduceAgg invoked
RexBuilder.addAggCall method whose parameter aggCallMapping acts as a AggCall
cache. The aggCallMapping cache only care about the call name not the data
type. The current master code of Calcite does nothing about this part since I
find this bug. I don't think I can exhaustive all the test cases to prove our
current master implementation right. But it seems security to have my tuned
part of codes (validating AggCall cache with data type) to the master to
prevent any future possible issues.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DrillReduceAggregatesRule mixed the same functions of the same inputRef which
> have different dataTypes
> ---
>
> Key: DRILL-5913
> URL: https://issues.apache.org/jira/browse/DRILL-5913
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>Priority: Major
>
> sample query:
> {code:java}
> select stddev_samp(cast(employee_id as int)) as col1, sum(cast(employee_id as
> int)) as col2 from cp.`employee.json`
> {code}
> error info:
> {code:java}
> org.apache.drill.exec.rpc.RpcException:
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> AssertionError: Type mismatch:
> rel rowtype:
> RecordType(INTEGER $f0, INTEGER $f1, BIGINT NOT NULL $f2, INTEGER $f3) NOT
> NULL
> equivRel rowtype:
> RecordType(INTEGER $f0, INTEGER $f1, BIGINT NOT NULL $f2, BIGINT $f3) NOT NULL
> [Error Id: f5114e62-a57b-46b1-afe8-ae652f390896 on localhost:31010]
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: Internal error: Error while applying rule
> DrillReduceAggregatesRule, args
> [rel#29:LogicalAggregate.NONE.ANY([]).[](input=rel#28:Subset#3.NONE.ANY([]).[],group={},agg#0=SUM($1),agg#1=SUM($0),agg#2=COUNT($0),agg#3=$SUM0($0))]
> org.apache.drill.exec.work.foreman.Foreman.run():294
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745
> Caused By (java.lang.AssertionError) Internal error: Error while applying
> rule DrillReduceAggregatesRule, args
> [rel#29:LogicalAggregate.NONE.ANY([]).[](input=rel#28:Subset#3.NONE.ANY([]).[],group={},agg#0=SUM($1),agg#1=SUM($0),agg#2=COUNT($0),agg#3=$SUM0($0))]
> org.apache.calcite.util.Util.newInternal():792
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():811
> {code}
> The reason is that stddev_samp(cast(employee_id as int)) will be reduced as
> sum($0) ,sum($1) ,count($0) while the sum(cast(employee_id as int)) will be
> reduced as sum0($0) by the DrillReduceAggregatesRule's first time matching.
> The second time's matching will reduce stddev_samp's sum($0) to sum0($0) too
> . But this sum0($0) 's data type is different from the first time's sum0($0)
> : one is integer ,the other is bigint . But Calcite's addAggCall
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467190#comment-16467190 ] ASF GitHub Bot commented on DRILL-6249: --- vvysotskyi commented on issue #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#issuecomment-387354275 @ilooner, thanks for adding testing documentation! Changes in `GeneratedCode.md` looks good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467187#comment-16467187 ] ASF GitHub Bot commented on DRILL-6249: --- arina-ielchiieva commented on issue #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#issuecomment-387353189 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6272) Remove binary jars files from source distribution
[ https://issues.apache.org/jira/browse/DRILL-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6272: Labels: ready-to-commit (was: ) > Remove binary jars files from source distribution > - > > Key: DRILL-6272 > URL: https://issues.apache.org/jira/browse/DRILL-6272 > Project: Apache Drill > Issue Type: Task >Reporter: Vlad Rozov >Assignee: Arina Ielchiieva >Priority: Critical > Labels: ready-to-commit > Fix For: 1.14.0 > > > Per [~vrozov] the source distribution contains binary jar files under > exec/java-exec/src/test/resources/jars -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Description:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{code:java}
SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{code}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{code:java}
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
{code}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
was:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{code:java}
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
{code}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {code:java}
> SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {code}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
>
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Description:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
was:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {quote}SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {quote}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
> "application":
> {
> "id": "some_id",
> "name": "Some App",
> "object": "application"
> },
> "application_fee": null,
> "amount": 5
> }
> ],
> "has_more": false,
> "object": "list",
> "url": "/v1/charges"
> }
> I ran some further tests on the query and found out that the query only fails
> at object "application_fee", which is null in both arrays of object "data".
> Which might explain why I never ran
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Description:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
bq. "data":
bq. [
bq. {
bq. "application": null,
bq. "application_fee": null,
bq. "amount": 1
bq. },
bq. {
bq. "application":
bq. {
bq. "id": "some_id",
bq. "name": "Some App",
bq. "object": "application"
bq. },
bq. "application_fee": null,
bq. "amount": 5
bq. }
bq. ],
bq. "has_more": false,
bq. "object": "list",
bq. "url": "/v1/charges"
bq. }
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
was:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {quote}SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {quote}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> bq. "data":
> bq. [
> bq. {
> bq. "application": null,
> bq. "application_fee": null,
> bq. "amount": 1
> bq. },
> bq. {
> bq. "application":
> bq. {
> bq. "id": "some_id",
> bq. "name": "Some App",
> bq. "object": "application"
> bq. },
> bq. "application_fee": null,
> bq. "amount": 5
> bq. }
> bq. ],
> bq. "has_more": false,
> bq. "object": "list",
> bq. "url": "/v1/charges"
> bq. }
> I ran some further tests on the query and found out that the query only fails
> at object
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Description:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{code:java}
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
{code}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
was:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
bq. "data":
bq. [
bq. {
bq. "application": null,
bq. "application_fee": null,
bq. "amount": 1
bq. },
bq. {
bq. "application":
bq. {
bq. "id": "some_id",
bq. "name": "Some App",
bq. "object": "application"
bq. },
bq. "application_fee": null,
bq. "amount": 5
bq. }
bq. ],
bq. "has_more": false,
bq. "object": "list",
bq. "url": "/v1/charges"
bq. }
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {quote}SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {quote}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {code:java}
> {
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
>
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Description:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
was:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {quote}SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {quote}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
> "application":
> {
> "id": "some_id",
> "name": "Some App",
>
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Component/s: Storage - JSON
Execution - Data Types
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {quote}SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {quote}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
> "application":
> {
> "id": "some_id",
> "name": "Some App",
> "object": "application"
> },
> "application_fee": null,
> "amount": 5
> }
> ],
> "has_more": false,
> "object": "list",
> "url": "/v1/charges"
> }
> I ran some further tests on the query and found out that the query only fails
> at object "application_fee", which is null in both arrays of object "data".
> Which might explain why I never ran into this error in the other datasets (no
> object arrays there are null).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Description:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
was:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {quote}SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {quote}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {
> "data":
> [
> {
> "application": null,
>
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Description:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
was:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{quote}{
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
{quote}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {quote}SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {quote}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {
> "data":
> [
> {
> "application": null,
>
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Description:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{quote}{
{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
{quote}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
was:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{quote}{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
{quote}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {quote}SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {quote}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {quote}{
> {
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
> "application":
> {
> "id": "some_id",
> "name": "Some App",
>
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Description:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{quote}{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
{quote}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
was:
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{quote}{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
{quote}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {quote}SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {quote}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {quote}{
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
> "application":
> {
> "id": "some_id",
> "name": "Some App",
> "object": "application"
> },
> "application_fee": null,
> "amount": 5
> }
> ],
> "has_more": false,
> "object": "list",
> "url": "/v1/charges"
> }
> {quote}
> I
[jira] [Updated] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trung Ta updated DRILL-6390:
Affects Version/s: 1.11.0
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {quote}SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {quote}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {quote}{
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
> "application":
> {
> "id": "some_id",
> "name": "Some App",
> "object": "application"
> },
> "application_fee": null,
> "amount": 5
> }
> ],
> "has_more": false,
> "object": "list",
> "url": "/v1/charges"
> }
> {quote}
> I ran some further tests on the query and found out that the query only fails
> at object "application_fee", which is null in both arrays of object "data".
> Which might explain why I never ran into this error in the other datasets (no
> object arrays there are null).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
Trung Ta created DRILL-6390:
---
Summary: UnsupportedOperationException: Unable to get holder type
for minor type [LATE] and mode [OPTIONAL]
Key: DRILL-6390
URL: https://issues.apache.org/jira/browse/DRILL-6390
Project: Apache Drill
Issue Type: Bug
Reporter: Trung Ta
I tried this query on some of our JSON files that are stored in a S3 Bucket, in
which I tried to convert JSON objects (application, application_fee) into
varchar:
{quote}SELECT
CONCAT(
'"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
, ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
) as JSONs
FROM (
SELECT
flatten(js.data) AS data
FROM
s3_bucket.`` AS js
) _stg
{quote}
On some of the datasets the query failed and I got this error message
{quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnsupportedOperationException: Unable to get holder type for minor type [LATE]
and mode [OPTIONAL] Fragment 0:0 [Error Id: 5e74e125-de22-46fa-8d31-233d0fc40140
{quote}
The dataset, on which the query failed looks like following:
{quote}{
"data":
[
{
"application": null,
"application_fee": null,
"amount": 1
},
{
"application":
{
"id": "some_id",
"name": "Some App",
"object": "application"
},
"application_fee": null,
"amount": 5
}
],
"has_more": false,
"object": "list",
"url": "/v1/charges"
}
{quote}
I ran some further tests on the query and found out that the query only fails
at object "application_fee", which is null in both arrays of object "data".
Which might explain why I never ran into this error in the other datasets (no
object arrays there are null).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4920) Connect to json file on web (http/https)
[ https://issues.apache.org/jira/browse/DRILL-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467098#comment-16467098 ] Alin Jurj commented on DRILL-4920: -- i am asking myself the same questions, does it work ? > Connect to json file on web (http/https) > > > Key: DRILL-4920 > URL: https://issues.apache.org/jira/browse/DRILL-4920 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - JSON >Affects Versions: 1.8.0 >Reporter: Michael Rans >Priority: Major > > I have not been able to set up Drill to connect to a JSON file at url: > https://data.humdata.org/api/3/action/current_package_list_with_resources?limit=1 > I can connect to files locally. > It is not clear to me from the documentation whether or not this feature > exists. If it doesn't, it should. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466997#comment-16466997
]
ASF GitHub Bot commented on DRILL-6321:
---
chunhui-shi commented on a change in pull request #1224: DRILL-6321: Customize
Drill's conformance. Allow support to APPLY key…
URL: https://github.com/apache/drill/pull/1224#discussion_r186636779
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillConformance.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import org.apache.calcite.sql.validate.SqlConformanceEnum;
+import org.apache.calcite.sql.validate.SqlDelegatingConformance;
+
+/**
+ * Drill's SQL conformance is SqlConformanceEnum.DEFAULT except for method
isApplyAllowed().
+ * Since Drill is going to allow OUTER APPLY and CROSS APPLY to allow each row
from left child of Join
+ * to join with output of right side (sub-query or table function that will be
invoked for each row).
+ * Refer to DRILL-5999 for more information.
+ */
+public class DrillConformance extends SqlDelegatingConformance {
Review comment:
There are multiple SQL extensions that could be introduced in the future as
we have discussed some of them and I believe there could be more. To name a
few, STREAM, implicit table key word, Although Calcite defines several ENUMs,
we chose to inherit DelegatingConformance to customize our own flavor of SQL.
As the class defining what SQL standard/extension(conformance) Drill is
providing. I think it is significant and should be standalone for reference
purpose.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lateral Join: Planning changes - enable submitting physical plan
>
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
> Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Implement changes to enable submitting a physical plan containing lateral and
> unnest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
