[jira] [Comment Edited] (CALCITE-6393) Byte code of SqlFunctions is invalid
[
https://issues.apache.org/jira/browse/CALCITE-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842488#comment-17842488
]
Sergey Nuyanzin edited comment on CALCITE-6393 at 4/30/24 7:13 PM:
---
Thanks for the info [~MasseGuillaume]
btw, do you happen to have bytecode check test which I assume you usually use
and which could be integrated into Calcite gradle as well?
I'm asking since probably I found the root cause for all the issues here.
It seems {{Checkerframework}} generates different bytecode for cases where
annotations are used in second nested parameterized type.
e.g. here {{@Nullable}} is used for first nested type
{code:java}
private static final ThreadLocal<@Nullable Map>
THREAD_SEQUENCES =
ThreadLocal.withInitial(HashMap::new);
{code}
everything is ok
However with second nested level like here
{code:java}
private static final Function1>
ARRAY_CARTESIAN_PRODUCT =
lists -> {
final List> enumerators = new
ArrayList<>();
for (Object list : lists) {
enumerators.add(Linq4j.enumerator((List) list));
}
final Enumerator> product =
Linq4j.product(enumerators);
return new AbstractEnumerable<@Nullable Object[]>() {
@Override public Enumerator<@Nullable Object[]> enumerator() {
return Linq4j.transform(product, List::toArray);
}
};
};
{code}
there is an issue.
It looks like this const {{ARRAY_CARTESIAN_PRODUCT}} is not used anywhere and
private. So I've just removed it and all the errors are disappeared.
So it seems the {{assert}} was just a marker for accumulated error however not
the root cause.
At the same side it would be great to have test integrated into build system to
have it always automatically checked
was (Author: sergey nuyanzin):
Thanks for the info [~MasseGuillaume]
btw, do you happen to have bytecode check test which I assume you usually use
and which could be integrated into Calcite gradle as well?
I'm asking since probably I found the root cause for all the issues here.
It seems {{Checkerframework}} generates different bytecode for cases where
annotations are used in second nested parameterized type.
e.g. here {{@Nullable}} is used for first nested type
{code:java}
private static final ThreadLocal<@Nullable Map>
THREAD_SEQUENCES =
ThreadLocal.withInitial(HashMap::new);
{code}
everything is ok
However with second nested level like here
{code:java}
private static final Function1>
ARRAY_CARTESIAN_PRODUCT =
lists -> {
final List> enumerators = new
ArrayList<>();
for (Object list : lists) {
enumerators.add(Linq4j.enumerator((List) list));
}
final Enumerator> product =
Linq4j.product(enumerators);
return new AbstractEnumerable<@Nullable Object[]>() {
@Override public Enumerator<@Nullable Object[]> enumerator() {
return Linq4j.transform(product, List::toArray);
}
};
};
{code}
there is an issue.
It looks like this const {{ARRAY_CARTESIAN_PRODUCT}} is not used anywhere and
private. So I've just removed it and all the errors are disappeared.
at the same side it would be great to have test integrated into build system to
have it always automatically checked
> Byte code of SqlFunctions is invalid
>
>
> Key: CALCITE-6393
> URL: https://issues.apache.org/jira/browse/CALCITE-6393
> Project: Calcite
> Issue Type: Bug
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> The issue is a result of testing of Apache Calcite 1.37.0 rc 4 in this thread
> [1]
> There is test project andprocedure provided by [~MasseGuillaume] [2] (see
> also original thread where this was first discussed [3])
> it shows that since Calcite 1.36.0 it starts failing as
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: Index 65536 out of bounds for
> length 297
> at org.objectweb.asm.ClassReader.readLabel(ClassReader.java:2695)
> at org.objectweb.asm.ClassReader.createLabel(ClassReader.java:2711)
> at
> org.objectweb.asm.ClassReader.readTypeAnnotations(ClassReader.java:2777)
> at org.objectweb.asm.ClassReader.readCode(ClassReader.java:1929)
> at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515)
> at org.objectweb.asm.ClassReader.accept(ClassReader.java:745)
> {noformat}
> Also since Calcite 1.27.0 it starts failing as
> {noformat}
> java.lang.IllegalArgumentException: Invalid end label (must be visited
> first)
> at
> org.objectweb.asm.util.CheckMethodAdapter.checkLabel(CheckMethodAdapter.java:1453)
> at
> org.objectweb.asm.util.CheckMethodAdapter.visitLocalVariableAnnotation(CheckMethodAdapter.java:996)
> at
>
[jira] [Comment Edited] (CALCITE-6393) Byte code of SqlFunctions is invalid
[
https://issues.apache.org/jira/browse/CALCITE-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842475#comment-17842475
]
Guillaume Massé edited comment on CALCITE-6393 at 4/30/24 6:24 PM:
---
We hava a fork of Calcite so we can build on jdk11 and get around the issue
where SqlFunctions bytecode is invalid.
(we run self-hosted Github actions)
AMI image: ubuntu/images/hvm-ssd/ubuntu-lunar-23.04-amd64-server-20240117
{code:java}
- name: 'Set up JDK 11'
uses: actions/setup-java@v2
with:
java-version: 11
distribution: 'zulu' {code}
{code:java}
Downloading Java 11.0.22+7 (Zulu) from
https://cdn.azul.com/zulu/bin/zulu11.70.15-ca-jdk11.0.22-linux_x64.tar.gz ...
{code}
I'm not sure what's the root cause of all this, but it looks like bytecode
generation is fragile and we should add a check after we run the jar task to
make sure that every classfile is valid bytecode with asm.
was (Author: masseguillaume):
We hava a fork of Calcite so we can build on jdk11 and get around the issue
where SqlFunctions bytecode is invalid.
- name: 'Set up JDK 11'
uses: actions/setup-java@v2
with:
java-version: 11
distribution: 'zulu'
Downloading Java 11.0.22+7 (Zulu) from
https://cdn.azul.com/zulu/bin/zulu11.70.15-ca-jdk11.0.22-linux_x64.tar.gz ...
I'm not sure what's the root cause of all this, but it looks like bytecode
generation is fragile and we should add a check after we run the jar task to
make sure that every classfile is valid bytecode with asm.
> Byte code of SqlFunctions is invalid
>
>
> Key: CALCITE-6393
> URL: https://issues.apache.org/jira/browse/CALCITE-6393
> Project: Calcite
> Issue Type: Bug
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> The issue is a result of testing of Apache Calcite 1.37.0 rc 4 in this thread
> [1]
> There is test project andprocedure provided by [~MasseGuillaume] [2] (see
> also original thread where this was first discussed [3])
> it shows that since Calcite 1.36.0 it starts failing as
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: Index 65536 out of bounds for
> length 297
> at org.objectweb.asm.ClassReader.readLabel(ClassReader.java:2695)
> at org.objectweb.asm.ClassReader.createLabel(ClassReader.java:2711)
> at
> org.objectweb.asm.ClassReader.readTypeAnnotations(ClassReader.java:2777)
> at org.objectweb.asm.ClassReader.readCode(ClassReader.java:1929)
> at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515)
> at org.objectweb.asm.ClassReader.accept(ClassReader.java:745)
> {noformat}
> Also since Calcite 1.27.0 it starts failing as
> {noformat}
> java.lang.IllegalArgumentException: Invalid end label (must be visited
> first)
> at
> org.objectweb.asm.util.CheckMethodAdapter.checkLabel(CheckMethodAdapter.java:1453)
> at
> org.objectweb.asm.util.CheckMethodAdapter.visitLocalVariableAnnotation(CheckMethodAdapter.java:996)
> at
> org.objectweb.asm.MethodVisitor.visitLocalVariableAnnotation(MethodVisitor.java:757)
> at
> org.objectweb.asm.commons.MethodRemapper.visitLocalVariableAnnotation(MethodRemapper.java:257)
> at org.objectweb.asm.ClassReader.readCode(ClassReader.java:2614)
> at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515)
> {noformat}
> [1] https://lists.apache.org/thread/n6cs1l86mt6fc5q8pcxr97czs3p6w65f
> [2] https://github.com/MasseGuillaume/asm-remapper-bug
> [3] https://lists.apache.org/thread/o736wz4qnr4l285bj5gv073cy0qll9t0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-6393) Byte code of SqlFunctions is invalid
[
https://issues.apache.org/jira/browse/CALCITE-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842393#comment-17842393
]
Sergey Nuyanzin edited comment on CALCITE-6393 at 4/30/24 1:56 PM:
---
{quote}
What other exception do you get with jdk11+?
{quote}
if I build it with jdk11 (in my case 11.0.20.1 (Eclipse Adoptium 11.0.20.1+1))
then i get this instead of {{ArrayIndexOutOfBoundsException}}
{noformat}
RemapperTest > ccalcite37WithCheck() FAILED
java.lang.IllegalArgumentException: Invalid type reference sort 0x10
at
org.objectweb.asm.util.CheckMethodAdapter.visitTypeAnnotation(CheckMethodAdapter.java:524)
at
org.objectweb.asm.MethodVisitor.visitTypeAnnotation(MethodVisitor.java:181)
at
org.objectweb.asm.commons.MethodRemapper.visitTypeAnnotation(MethodRemapper.java:94)
at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1453)
at org.objectweb.asm.ClassReader.accept(ClassReader.java:745)
at org.objectweb.asm.ClassReader.accept(ClassReader.java:425)
at remapper.bug.RemapperTest.runTest(RemapperTest.java:63)
at remapper.bug.RemapperTest.ccalcite37WithCheck(RemapperTest.java:49)
{noformat}
UPD: with jdk17 (in my case 17.0.10 (Eclipse Adoptium 17.0.10+7))
the exception is going away
was (Author: sergey nuyanzin):
{quote}
What other exception do you get with jdk11+?
{quote}
if I build it with jdk11 (in my case 11.0.20.1 (Eclipse Adoptium 11.0.20.1+1))
then i get this instead of {{ArrayIndexOutOfBoundsException}}
{noformat}
RemapperTest > ccalcite37WithCheck() FAILED
java.lang.IllegalArgumentException: Invalid type reference sort 0x10
at
org.objectweb.asm.util.CheckMethodAdapter.visitTypeAnnotation(CheckMethodAdapter.java:524)
at
org.objectweb.asm.MethodVisitor.visitTypeAnnotation(MethodVisitor.java:181)
at
org.objectweb.asm.commons.MethodRemapper.visitTypeAnnotation(MethodRemapper.java:94)
at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1453)
at org.objectweb.asm.ClassReader.accept(ClassReader.java:745)
at org.objectweb.asm.ClassReader.accept(ClassReader.java:425)
at remapper.bug.RemapperTest.runTest(RemapperTest.java:63)
at remapper.bug.RemapperTest.ccalcite37WithCheck(RemapperTest.java:49)
{noformat}
> Byte code of SqlFunctions is invalid
>
>
> Key: CALCITE-6393
> URL: https://issues.apache.org/jira/browse/CALCITE-6393
> Project: Calcite
> Issue Type: Bug
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> The issue is a result of testing of Apache Calcite 1.37.0 rc 4 in this thread
> [1]
> There is test project andprocedure provided by [~MasseGuillaume] [2] (see
> also original thread where this was first discussed [3])
> it shows that since Calcite 1.36.0 it starts failing as
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: Index 65536 out of bounds for
> length 297
> at org.objectweb.asm.ClassReader.readLabel(ClassReader.java:2695)
> at org.objectweb.asm.ClassReader.createLabel(ClassReader.java:2711)
> at
> org.objectweb.asm.ClassReader.readTypeAnnotations(ClassReader.java:2777)
> at org.objectweb.asm.ClassReader.readCode(ClassReader.java:1929)
> at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515)
> at org.objectweb.asm.ClassReader.accept(ClassReader.java:745)
> {noformat}
> Also since Calcite 1.27.0 it starts failing as
> {noformat}
> java.lang.IllegalArgumentException: Invalid end label (must be visited
> first)
> at
> org.objectweb.asm.util.CheckMethodAdapter.checkLabel(CheckMethodAdapter.java:1453)
> at
> org.objectweb.asm.util.CheckMethodAdapter.visitLocalVariableAnnotation(CheckMethodAdapter.java:996)
> at
> org.objectweb.asm.MethodVisitor.visitLocalVariableAnnotation(MethodVisitor.java:757)
> at
> org.objectweb.asm.commons.MethodRemapper.visitLocalVariableAnnotation(MethodRemapper.java:257)
> at org.objectweb.asm.ClassReader.readCode(ClassReader.java:2614)
> at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515)
> {noformat}
> [1] https://lists.apache.org/thread/n6cs1l86mt6fc5q8pcxr97czs3p6w65f
> [2] https://github.com/MasseGuillaume/asm-remapper-bug
> [3] https://lists.apache.org/thread/o736wz4qnr4l285bj5gv073cy0qll9t0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
