[
https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538626#comment-17538626
]
Jonathan Swenson edited comment on CALCITE-2040 at 5/18/22 5:46 PM:
--------------------------------------------------------------------
For reference the linker error I get on an M1 mac when running the arrow tests
is:
{code:java}
FAILURE 2.3sec, org.apache.calcite.adapter.arrow.ArrowAdapterTest >
testArrowProjectFieldsWithFloatFilter()
java.lang.UnsatisfiedLinkError:
/private/var/folders/fj/63_6n5dx10n4b5x7jtdj6tvh0000gn/T/libgandiva_jni.dylib804580c2-6fe4-4294-bdbb-c0c7d9e582a8:
dlopen(/private/var/folders/fj/63_6n5dx10n4b5x7jtdj6tvh0000gn/T/libgandiva_jni.dylib804580c2-6fe4-4294-bdbb-c0c7d9e582a8,
0x0001): tried:
'/private/var/folders/fj/63_6n5dx10n4b5x7jtdj6tvh0000gn/T/libgandiva_jni.dylib804580c2-6fe4-4294-bdbb-c0c7d9e582a8'
(mach-o file, but is an incompatible architecture (have 'x86_64', need
'arm64e'))
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
at java.lang.Runtime.load0(Runtime.java:811)
at java.lang.System.load(System.java:1088)
at
org.apache.arrow.gandiva.evaluator.JniLoader.loadGandivaLibraryFromJar(JniLoader.java:74)
at
org.apache.arrow.gandiva.evaluator.JniLoader.setupInstance(JniLoader.java:63)
at
org.apache.arrow.gandiva.evaluator.JniLoader.getInstance(JniLoader.java:53)
at
org.apache.arrow.gandiva.evaluator.JniLoader.getDefaultConfiguration(JniLoader.java:144)
at org.apache.arrow.gandiva.evaluator.Filter.make(Filter.java:67) {code}
Filed https://issues.apache.org/jira/browse/ARROW-16608 to track this issue.
was (Author: jswenson):
For reference the linker error I get on an M1 mac when running the arrow tests
is:
{code:java}
FAILURE 2.3sec, org.apache.calcite.adapter.arrow.ArrowAdapterTest >
testArrowProjectFieldsWithFloatFilter()
java.lang.UnsatisfiedLinkError:
/private/var/folders/fj/63_6n5dx10n4b5x7jtdj6tvh0000gn/T/libgandiva_jni.dylib804580c2-6fe4-4294-bdbb-c0c7d9e582a8:
dlopen(/private/var/folders/fj/63_6n5dx10n4b5x7jtdj6tvh0000gn/T/libgandiva_jni.dylib804580c2-6fe4-4294-bdbb-c0c7d9e582a8,
0x0001): tried:
'/private/var/folders/fj/63_6n5dx10n4b5x7jtdj6tvh0000gn/T/libgandiva_jni.dylib804580c2-6fe4-4294-bdbb-c0c7d9e582a8'
(mach-o file, but is an incompatible architecture (have 'x86_64', need
'arm64e'))
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
at java.lang.Runtime.load0(Runtime.java:811)
at java.lang.System.load(System.java:1088)
at
org.apache.arrow.gandiva.evaluator.JniLoader.loadGandivaLibraryFromJar(JniLoader.java:74)
at
org.apache.arrow.gandiva.evaluator.JniLoader.setupInstance(JniLoader.java:63)
at
org.apache.arrow.gandiva.evaluator.JniLoader.getInstance(JniLoader.java:53)
at
org.apache.arrow.gandiva.evaluator.JniLoader.getDefaultConfiguration(JniLoader.java:144)
at org.apache.arrow.gandiva.evaluator.Filter.make(Filter.java:67) {code}
> Create adapter for Apache Arrow
> -------------------------------
>
> Key: CALCITE-2040
> URL: https://issues.apache.org/jira/browse/CALCITE-2040
> Project: Calcite
> Issue Type: Bug
> Reporter: Julian Hyde
> Assignee: Julian Hyde
> Priority: Major
> Labels: pull-request-available
> Attachments: arrow_data.py
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would
> allow people to execute SQL statements, via JDBC or ODBC, on data stored in
> Arrow in-memory format.
> Since Arrow is an in-memory format, it is not as straightforward as reading,
> say, CSV files using the file adapter: an Arrow data set does not have a URL.
> (Unless we use Arrow's
> [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/]
> format, or use an in-memory file system such as Alluxio.) So we would need
> to devise a way of addressing Arrow data sets.
> Also, since Arrow is an extremely efficient format for processing data, it
> would also be good to have Arrow as a calling convention. That is,
> implementations of relational operators such as Filter, Project, Aggregate in
> addition to just TableScan.
> Lastly, when we have an Arrow convention, if we build adapters for file
> formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in
> CALCITE-2025) it would make a lot of sense to translate those formats
> directly into Arrow (applying simple projects and filters first if
> applicable). Those adapters would belong as a "contrib" module in the Arrow
> project better than in Calcite.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
