[
https://issues.apache.org/jira/browse/DRILL-6547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312925#comment-17312925
]
ASF GitHub Bot commented on DRILL-6547:
---------------------------------------
paul-rogers commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-811694253
@oleg-zinovev, sorry for the delay. For some reason, Github does not notify
me of new comments, even though I'm a "watcher." Odd.
Thanks also for the explanation and generated code. I agree with your
analysis. If we ever do a "revised edition" of the Drill book, we should
explain this detail.
I wonder: is this the only remaining function that has this problem? When I
quickly glanced through the UDFs, it seemed some used the pattern like the one
you are using here, others use the "wrong" pattern. Should we fix all of them
to avoid having to fix them one by one?
Also, the reason that I said that one of the VARCHAR fields would probably
be null is because of the name of the function:
``` java
public static class ConcatRightNullInput implements DrillSimpleFunc {
```
We seem to also have:
``` java
public static class ConcatLeftNullInput implements DrillSimpleFunc {
```
Which, by the way, has the same bug you are fixing here.
There is also:
``` java
public static class ConcatBothNullInput implements DrillSimpleFunc {
```
For which, presumably, both inputs are NULL (zero length), but we still
allocate a buffer, and that code has the bug you are fixing. Of course, if both
inputs are NULL, then there length is undefined (NULL says to ignore the value;
we cannot assume that the string value is the empty string or that the
start/end offsets are valid.)
Finally, we also have:
``` java
public static class ConcatOperator implements DrillSimpleFunc {
```
Which does use both inputs, and which already has the fix you propose here.
This is what I meant: we have multiple variations of the same function, some
are fixed, some are still broken. The variations claim that one or both sides
are NULL, but we are still working with the (possibly invalid) VARCHARs in
those cases, when we should not.
Now, because things are not complex enough, we have another issue. If either
of the arguments of `CONCAT(left, right)` are the SQL NULL value, then the
output should be... the non-null argument? Or, NULL? At least according to [SQL
Server](https://docs.microsoft.com/en-us/sql/t-sql/functions/concat-transact-sql?view=sql-server-ver15),
the rule is that a NULL value is treated as an empty string. We should thus
modify the three `Null` variations to do that: use an empty string, *not* the
value of the null argument. That is, for `ConcatRightNullInput`, return the
left input. For `ConcatLeftNullInput`, return the right input. For
`ConcatBothNullInput`, return an empty (zero-length) string.
I hope this clarifies things a bit!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> IllegalStateException: Tried to remove unmanaged buffer.
> --------------------------------------------------------
>
> Key: DRILL-6547
> URL: https://issues.apache.org/jira/browse/DRILL-6547
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 1.14.0
> Reporter: Robert Hou
> Assignee: Pritesh Maker
> Priority: Major
>
> This is the query:
> select * from (
> select Index, concat(BinaryValue, 'aaa') NewVarcharValue from (select * from
> dfs.`/drill/testdata/batch_memory/alltypes_large_1MB.parquet`)) d where
> d.Index = 1;
> This is the plan:
> {noformat}
> | 00-00 Screen
> 00-01 Project(Index=[$0], NewVarcharValue=[$1])
> 00-02 SelectionVectorRemover
> 00-03 Filter(condition=[=($0, 1)])
> 00-04 Project(Index=[$0], NewVarcharValue=[CONCAT($1, 'aaa')])
> 00-05 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath
> [path=maprfs:///drill/testdata/batch_memory/alltypes_large_1MB.parquet]],
> selectionRoot=maprfs:/drill/testdata/batch_memory/alltypes_large_1MB.parquet,
> numFiles=1, numRowGroups=1, usedMetadataFile=false, columns=[`Index`,
> `BinaryValue`]]])
> {noformat}
> Here is the stack trace from drillbit.log:
> {noformat}
> 2018-06-27 13:55:03,291 [24cc0659-30b7-b290-7fae-ecb1c1f15c05:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
> Tried to remove unmanaged buffer.
> Fragment 0:0
> [Error Id: bc1f2f72-c31b-4b9a-964f-96dec9e0f388 on qa-node186.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Tried to remove unmanaged buffer.
> Fragment 0:0
> [Error Id: bc1f2f72-c31b-4b9a-964f-96dec9e0f388 on qa-node186.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.IllegalStateException: Tried to remove unmanaged buffer.
> at
> org.apache.drill.exec.ops.BufferManagerImpl.replace(BufferManagerImpl.java:50)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at io.netty.buffer.DrillBuf.reallocIfNeeded(DrillBuf.java:97)
> ~[drill-memory-base-1.14.0-SNAPSHOT.jar:4.0.48.Final]
> at
> org.apache.drill.exec.test.generated.ProjectorGen4046.doEval(ProjectorTemplate.java:77)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.ProjectorGen4046.projectRecords(ProjectorTemplate.java:67)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:236)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:117)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:147)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:147)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:294)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:281)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[na:1.8.0_161]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_161]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
> ~[hadoop-common-2.7.0-mapr-1707.jar:na]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:281)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> ... 4 common frames omitted
> {noformat}
> The table can be found in
> /home/MAPRTECH/qa/rhou/drill4479/alltypes_large_1MB.parquet
> This is the commit id:
> aa127b70b1e46f7f4aa19881f25eda583627830a
> alltypes_large10.q
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
