Hello Kou,
Thank you for your further infomation, it's very helpful!
As you advised, we dived into our code and found that on any condition that
a CAST keyword passed to Arrow, it would throw out exception as below.
Attached is our source code, and our input is a SQL clause "SELECT
CAST(a.ID AS INTEGER) FROM MDS01."mds_work_team" a".
Would you help us a little out of our difficulty?
Thanks a lot in advance!
## Exception
Stack: [0x00007f59ed16b000,0x00007f59ed26c000], sp=0x00007f59ed2694b8, free
space=1017k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted,
Vv=VM code, C=native code)
V [libjvm.so+0x578764]
AccessInternal::PostRuntimeDispatch<G1BarrierSet::AccessBarrier<1097844ul,
G1BarrierSet>, (AccessInternal::BarrierType)2,
1097844ul>::oop_access_barrier(void*)+0x4
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j
org.apache.arrow.gandiva.evaluator.JniWrapper.buildProjector(Ljava/lang/Object;[B[BIJ)J+0
j
org.apache.arrow.gandiva.evaluator.Projector.make(Lorg/apache/arrow/vector/types/pojo/Schema;Ljava/util/List;Lorg/apache/arrow/gandiva/ipc/GandivaTypes$SelectionVectorType;JLorg/apache/arrow/gandiva/evaluator/JavaSecondaryCacheInterface;)Lorg/apache/arrow/gandiva/evaluator/Projector;+85
j
org.apache.arrow.gandiva.evaluator.Projector.make(Lorg/apache/arrow/vector/types/pojo/Schema;Ljava/util/List;Lorg/apache/arrow/gandiva/ipc/GandivaTypes$SelectionVectorType;J)Lorg/apache/arrow/gandiva/evaluator/Projector;+5
j
org.apache.arrow.gandiva.evaluator.Projector.make(Lorg/apache/arrow/vector/types/pojo/Schema;Ljava/util/List;Lorg/apache/arrow/gandiva/evaluator/ConfigurationBuilder$ConfigOptions;)Lorg/apache/arrow/gandiva/evaluator/Projector;+9
j com.bigknow.sabot.op.llvm.NativeProjector.build()V+104
j
com.bigknow.sabot.op.llvm.NativeProjectorBuilder.build(Lorg/apache/arrow/vector/types/pojo/Schema;Lcom/bigknow/sabot/exec/context/OperatorStats;Lcom/bigknow/sabot/op/llvm/GandivaSecondaryCacheWithStats;)Lcom/bigknow/sabot/op/llvm/NativeProjectEvaluator;+194
j
com.bigknow.exec.expr.SplitStageExecutor.setupFinish(Lcom/bigknow/exec/record/VectorContainer;Lcom/google/common/base/Stopwatch;Lcom/google/common/base/Stopwatch;Lcom/bigknow/sabot/op/llvm/GandivaSecondaryCacheWithStats;)V+28
j
com.bigknow.exec.expr.SplitStageExecutor.setupProjector(Lcom/bigknow/exec/record/VectorContainer;Lcom/google/common/base/Stopwatch;Lcom/google/common/base/Stopwatch;Lcom/bigknow/sabot/op/llvm/GandivaSecondaryCacheWithStats;)V+98
j
com.bigknow.exec.expr.ExpressionSplitter.projectorSetup(Lcom/bigknow/exec/record/VectorContainer;Lcom/google/common/base/Stopwatch;Lcom/google/common/base/Stopwatch;Lcom/bigknow/sabot/op/llvm/GandivaSecondaryCacheWithStats;)V+40
j
com.bigknow.exec.expr.ExpressionSplitter.setupProjector(Lcom/bigknow/exec/record/VectorContainer;Lcom/google/common/base/Stopwatch;Lcom/google/common/base/Stopwatch;Lcom/bigknow/sabot/op/llvm/GandivaSecondaryCacheWithStats;)Lcom/bigknow/exec/record/VectorContainer;+14
j
com.bigknow.exec.expr.ExpressionSplitter.setupProjector(Lcom/bigknow/exec/record/VectorContainer;Lcom/google/common/base/Stopwatch;Lcom/google/common/base/Stopwatch;)Lcom/bigknow/exec/record/VectorContainer;+5
j
com.bigknow.exec.store.CoercionReader.setupProjector(Lcom/bigknow/exec/record/VectorContainer;)V+101
j
com.bigknow.exec.store.CoercionReader.newSchema(Lcom/bigknow/exec/record/VectorContainer;Lcom/bigknow/sabot/op/scan/OutputMutator;)V+88
j
com.bigknow.exec.store.CoercionReader.setup(Lcom/bigknow/sabot/op/scan/OutputMutator;)V+31
j
com.bigknow.sabot.op.scan.ScanOperator.setupReaderAsCorrectUser(Lcom/bigknow/exec/store/RecordReader;)V+11
j
com.bigknow.sabot.op.scan.ScanOperator.setupReader(Lcom/bigknow/exec/store/RecordReader;)V+35
j
com.bigknow.sabot.op.scan.ScanOperator.setup()Lcom/bigknow/exec/record/VectorAccessible;+50
j
com.bigknow.sabot.driver.SmartOp$SmartProducer.setup()Lcom/bigknow/exec/record/VectorAccessible;+22
j
com.bigknow.sabot.driver.Pipe$SetupVisitor.visitProducer(Lcom/bigknow/sabot/op/spi/ProducerOperator;Lcom/bigknow/exec/record/VectorAccessible;)Lcom/bigknow/exec/record/VectorAccessible;+13
j
com.bigknow.sabot.driver.Pipe$SetupVisitor.visitProducer(Lcom/bigknow/sabot/op/spi/ProducerOperator;Ljava/lang/Object;)Ljava/lang/Object;+6
j
com.bigknow.sabot.driver.SmartOp$SmartProducer.accept(Lcom/bigknow/sabot/op/spi/Operator$OperatorVisitor;Ljava/lang/Object;)Ljava/lang/Object;+3
j
com.bigknow.sabot.driver.StraightPipe.setup()Lcom/bigknow/exec/record/VectorAccessible;+19
j
com.bigknow.sabot.driver.WyePipe.setup()Lcom/bigknow/exec/record/VectorAccessible;+34
j
com.bigknow.sabot.driver.StraightPipe.setup()Lcom/bigknow/exec/record/VectorAccessible;+34
j
com.bigknow.sabot.driver.StraightPipe.setup()Lcom/bigknow/exec/record/VectorAccessible;+34
j
com.bigknow.sabot.driver.StraightPipe.setup()Lcom/bigknow/exec/record/VectorAccessible;+34
j
com.bigknow.sabot.driver.StraightPipe.setup()Lcom/bigknow/exec/record/VectorAccessible;+34
j
com.bigknow.sabot.driver.StraightPipe.setup()Lcom/bigknow/exec/record/VectorAccessible;+34
j
com.bigknow.sabot.driver.StraightPipe.setup()Lcom/bigknow/exec/record/VectorAccessible;+34
j
com.bigknow.sabot.driver.StraightPipe.setup()Lcom/bigknow/exec/record/VectorAccessible;+34
j com.bigknow.sabot.driver.Pipeline.setup()V+4
j com.bigknow.sabot.exec.fragment.FragmentExecutor.setupExecution()V+252
j com.bigknow.sabot.exec.fragment.FragmentExecutor.run()V+392
j
com.bigknow.sabot.exec.fragment.FragmentExecutor.access$1700(Lcom/bigknow/sabot/exec/fragment/FragmentExecutor;)V+1
j com.bigknow.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run()V+4
j com.bigknow.sabot.task.AsyncTaskWrapper.run()V+74
j com.bigknow.sabot.task.slicing.SlicingThread.mainExecutionLoop()V+367
j com.bigknow.sabot.task.slicing.SlicingThread.run()V+4
v ~StubRoutines::call_stub
Stack slot to memory mapping:
stack at sp + 0 slots: 0x00007f5a347ebb69: <offset 0x00000000008dfb69> in
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so at 0x00007f5a33f0c000
stack at sp + 1 slots: 0x0 is NULL
stack at sp + 2 slots: 0x00007f59ed2698c0 is pointing into the stack for
thread: 0x00007f59e875a000
stack at sp + 3 slots: 0x00007f59e875a348 points into unknown readable memory:
0x00007f5a35147b40 | 40 7b 14 35 5a 7f 00 00
stack at sp + 4 slots: 0x00007f59e2f8c3f7:
Java_org_apache_arrow_gandiva_evaluator_JniWrapper_buildProjector+0x0000000000000057
in /tmp/libgandiva_jni.soa81db5ba-fcd3-4494-b24b-3da3a3a99162 at
0x00007f59e2b1b000
stack at sp + 5 slots: 0x00007f59ed269500 is pointing into the stack for
thread: 0x00007f59e875a000
stack at sp + 6 slots: 0x00007f5a344ec003: <offset 0x00000000005e0003> in
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so at 0x00007f5a33f0c000
stack at sp + 7 slots: 0x00007f59ed2697b0 is pointing into the stack for
thread: 0x00007f59e875a000
## GDB Debug info
$ gdb /usr/bin/java core
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f2576f32859 in __GI_abort () at abort.c:79
#2 0x00007f2575eca435 in os::abort(bool, void*, void const*) [clone .cold] ()
from /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#3 0x00007f2576ae2e5d in VMError::report_and_die(int, char const*, char
const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*,
int, unsigned long) () from
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#4 0x00007f2576ae39cf in VMError::report_and_die(Thread*, unsigned int,
unsigned char*, void*, void*, char const*, ...) () from
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#5 0x00007f2576ae3a02 in VMError::report_and_die(Thread*, unsigned int,
unsigned char*, void*, void*) () from
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#6 0x00007f2576838166 in JVM_handle_linux_signal () from
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#7 0x00007f257682ac8c in signalHandler(int, siginfo_t*, void*) () from
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#8 <signal handler called>
#9 0x00007f25761bf764 in
AccessInternal::PostRuntimeDispatch<G1BarrierSet::AccessBarrier<1097844ul,
G1BarrierSet>, (AccessInternal::BarrierType)2,
1097844ul>::oop_access_barrier(void*) () from
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#10 0x00007f2576526b69 in jni_GetArrayLength () from
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#11 0x00007f2533306586 in JNIEnv_::GetArrayLength (this=0x7f25737a7b48,
array=0x0) at /usr/lib/jvm/java-11-openjdk-amd64/include/jni.h:1629
#12 0x00007f25332feebf in
Java_org_apache_arrow_gandiva_evaluator_JniWrapper_buildProjector
(env=0x7f25737a7b48, obj=0x7f25280899a8, schema_arr=0x0,
exprs_arr=0x7f2528089998, selection_vector_type=671652240, configuration_id=0)
at
/home/ubuntu/work/arrow/arrow-maint-9.0.0/cpp/src/gandiva/jni/jni_common.cc:611
#13 0x00007f25587eaa30 in ?? ()
#14 0x0000000000000001 in ?? ()
#15 0x0000000000000000 in ?? ()
## IDEA call stack
make:238, Projector (org.apache.arrow.gandiva.evaluator)
make:190, Projector (org.apache.arrow.gandiva.evaluator)
make:95, Projector (org.apache.arrow.gandiva.evaluator)
build:87, NativeProjector (com.bigknow.sabot.op.llvm)
build:84, NativeProjectorBuilder (com.bigknow.sabot.op.llvm)
setupFinish:185, SplitStageExecutor (com.bigknow.exec.expr)
setupProjector:221, SplitStageExecutor (com.bigknow.exec.expr)
projectorSetup:351, ExpressionSplitter (com.bigknow.exec.expr)
setupProjector:562, ExpressionSplitter (com.bigknow.exec.expr)
setupProjector:555, ExpressionSplitter (com.bigknow.exec.expr)
setupProjector:159, CoercionReader (com.bigknow.exec.store)
newSchema:136, CoercionReader (com.bigknow.exec.store)
setup:119, CoercionReader (com.bigknow.exec.store)
setupReaderAsCorrectUser:349, ScanOperator (com.bigknow.sabot.op.scan)
setupReader:340, ScanOperator (com.bigknow.sabot.op.scan)
setup:304, ScanOperator (com.bigknow.sabot.op.scan)
setup:595, SmartOp$SmartProducer (com.bigknow.sabot.driver)
visitProducer:80, Pipe$SetupVisitor (com.bigknow.sabot.driver)
visitProducer:64, Pipe$SetupVisitor (com.bigknow.sabot.driver)
accept:565, SmartOp$SmartProducer (com.bigknow.sabot.driver)
setup:102, StraightPipe (com.bigknow.sabot.driver)
setup:102, StraightPipe (com.bigknow.sabot.driver)
setup:102, StraightPipe (com.bigknow.sabot.driver)
setup:102, StraightPipe (com.bigknow.sabot.driver)
setup:102, StraightPipe (com.bigknow.sabot.driver)
setup:102, StraightPipe (com.bigknow.sabot.driver)
setup:102, StraightPipe (com.bigknow.sabot.driver)
setup:71, Pipeline (com.bigknow.sabot.driver)
setupExecution:619, FragmentExecutor (com.bigknow.sabot.exec.fragment)
run:441, FragmentExecutor (com.bigknow.sabot.exec.fragment)
access$1700:107, FragmentExecutor (com.bigknow.sabot.exec.fragment)
run:999, FragmentExecutor$AsyncTaskImpl (com.bigknow.sabot.exec.fragment)
run:122, AsyncTaskWrapper (com.bigknow.sabot.task)
mainExecutionLoop:249, SlicingThread (com.bigknow.sabot.task.slicing)
run:171, SlicingThread (com.bigknow.sabot.task.slicing)
## Values → Projector.make
schema: Schema<expr$0: Decimal(38, 6, 128)>
exprs: size = 1
## Projector.make → JniWrapper.buildProjector
schemaBuf.toByteArray():
[10, 18, 10, 6, 101, 120, 112, 114, 36, 48, 18, 6, 8, 22, 24, 38, 32, 6, 24, 1]
builder.build().toByteArray():
[18, 77, 10, 59, 18, 57, 10, 7, 99, 97, 115, 116, 73, 78, 84, 18, 42, 18, 40,
10, 10, 99, 97, 115, 116, 70, 76, 79, 65, 84, 56, 18, 22, 10, 20, 10, 18, 10,
6, 101, 120, 112, 114, 36, 48, 18, 6, 8, 22, 24, 38, 32, 6, 24, 1, 26, 2, 8,
12, 26, 2, 8, 7, 18, 14, 10, 6, 69, 88, 80, 82, 36, 48, 18, 2, 8, 7, 24, 1]
## Errors we found in JniWrapper.buildProjector calling libgandiva_jni.so at
Java_org_apache_arrow_gandiva_evaluator_JniWrapper_buildProjector
schema_arr in Java layer schemaBuf.toByteArray() in NOT Null，but here is 0x0
Java_org_apache_arrow_gandiva_evaluator_JniWrapper_buildProjector
(env=0x7f25737a7b48, obj=0x7f25280899a8, schema_arr=0x0,
exprs_arr=0x7f2528089998, selection_vector_type=671652240, configuration_id=0)
## Error location in Java source
at
arrow-gandiva-9.0.0-20221123064031-c39b8a6253-bigknow.jar!/org/apache/arrow/gandiva/evaluator/Projector.class
Projector.make method
wrapper.buildProjector(secondaryCache, schemaBuf.toByteArray(),
builder.build().toByteArray(), selectionVectorType.getNumber(),
configurationId);
```
public static Projector make(Schema schema, List<ExpressionTree> exprs,
GandivaTypes.SelectionVectorType selectionVectorType, long configurationId,
JavaSecondaryCacheInterface secondaryCache) throws GandivaException {
GandivaTypes.ExpressionList.Builder builder =
ExpressionList.newBuilder();
Iterator var7 = exprs.iterator();
while(var7.hasNext()) {
ExpressionTree expr = (ExpressionTree)var7.next();
builder.addExprs(expr.toProtobuf());
}
GandivaTypes.Schema schemaBuf =
ArrowTypeHelper.arrowSchemaToProtobuf(schema);
JniWrapper wrapper = JniLoader.getInstance().getWrapper();
long moduleId = wrapper.buildProjector(secondaryCache,
schemaBuf.toByteArray(), builder.build().toByteArray(),
selectionVectorType.getNumber(), configurationId);
logger.debug("Created module for the projector with id {}", moduleId);
return new Projector(wrapper, moduleId, schema, exprs.size());
}
Projector.java
Description: Projector.java
