[jira] [Resolved] (DRILL-7854) When writing to S3 "WriteOperationHelper.newUploadPartRequest" throws "NoSuchMethodError"
[
https://issues.apache.org/jira/browse/DRILL-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon resolved DRILL-7854.
---
Resolution: Fixed
Should have been resolved when Guava version was updated to 30.0
> When writing to S3 "WriteOperationHelper.newUploadPartRequest" throws
> "NoSuchMethodError"
> -
>
> Key: DRILL-7854
> URL: https://issues.apache.org/jira/browse/DRILL-7854
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Affects Versions: 1.18.0
> Environment: Standalone drill running in a docker environment on
> x86_64 (centos7, alpine 3, debian buster)
>
> Drill 1.18.0 was downloaded from an official mirror
>Reporter: Joshua Pedrick
>Priority: Major
> Fix For: 1.19.0
>
>
> When writing to S3(hosted on local Minio cluster) I am getting a
> "NoSuchMethod" error. It appears to be related to the guava version included
> with hadoop-3.2.1.
>
> {code:java}
> 2021-02-01 21:39:17,753 [1fe78bd6-0525-196e-b3d2-25c294a604e2:frag:1:13]
> ERROR o.a.d.e.p.i.p.ProjectRecordBatch -
> ProjectRecordBatch[projector=Projector[vector2=null,
> selectionVectorMode=NONE], hasRemainder=false, remainderIndex=0,
> recordCount=0,
> container=org.apache.drill.exec.record.VectorContainer@3efe0be4[recordCount =
> 27884, schemaChanged = false, schema = BatchSchema [, ...]]2021-02-01
> 21:39:17,754 [1fe78bd6-0525-196e-b3d2-25c294a604e2:frag:1:13] ERROR
> o.a.d.e.physical.impl.BaseRootExec - Batch dump completed.2021-02-01
> 21:39:17,754 [1fe78bd6-0525-196e-b3d2-25c294a604e2:frag:1:13] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1fe78bd6-0525-196e-b3d2-25c294a604e2:1:13: State change requested
> CANCELLATION_REQUESTED --> FAILED2021-02-01 21:39:25,199
> [1fe78bd6-0525-196e-b3d2-25c294a604e2:frag:1:13] ERROR
> o.a.d.exec.server.BootStrapContext -
> org.apache.drill.exec.work.WorkManager$WorkerBee$1.run() leaked an
> exception.java.lang.NoSuchMethodError:
> com/google/common/base/Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;J)V
> (loaded from by sun.misc.Launcher$AppClassLoader@3156fd60) called
> from class org.apache.hadoop.fs.s3a.WriteOperationHelper (loaded from
> file:/opt/drill/jars/3rdparty/hadoop-aws-3.2.1.jar by
> sun.misc.Launcher$AppClassLoader@3156fd60).at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.newUploadPartRequest(WriteOperationHelper.java:397)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.uploadBlockAsync(S3ABlockOutputStream.java:584)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.access$000(S3ABlockOutputStream.java:521)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.uploadCurrentBlock(S3ABlockOutputStream.java:314)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.write(S3ABlockOutputStream.java:292)
>at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)at
> java.io.FilterOutputStream.write(FilterOutputStream.java:97) at
> org.apache.parquet.hadoop.util.HadoopPositionOutputStream.write(HadoopPositionOutputStream.java:45)
> at
> org.apache.parquet.bytes.CapacityByteArrayOutputStream.writeToOutput(CapacityByteArrayOutputStream.java:234)
> at
> org.apache.parquet.bytes.CapacityByteArrayOutputStream.writeTo(CapacityByteArrayOutputStream.java:247)
>at
> org.apache.parquet.bytes.BytesInput$CapacityBAOSBytesInput.writeAllTo(BytesInput.java:421)
>at
> org.apache.parquet.hadoop.ParquetFileWriter.writeColumnChunk(ParquetFileWriter.java:620)
> at
> org.apache.parquet.hadoop.ParquetColumnChunkPageWriteStore$ColumnChunkPageWriter.writeToFileWriter(ParquetColumnChunkPageWriteStore.java:268)
> at
> org.apache.parquet.hadoop.ParquetColumnChunkPageWriteStore.flushToFileWriter(ParquetColumnChunkPageWriteStore.java:89)
>at
> org.apache.drill.exec.store.parquet.ParquetRecordWriter.flushParquetFileWriter(ParquetRecordWriter.java:737)
> at
> org.apache.drill.exec.store.parquet.ParquetRecordWriter.flush(ParquetRecordWriter.java:435)
> at
> org.apache.drill.exec.store.parquet.ParquetRecordWriter.cleanup(ParquetRecordWriter.java:703)
> at
> org.apache.drill.exec.physical.impl.WriterRecordBatch.closeWriter(WriterRecordBatch.java:203)
> at
> org.apache.drill.exec.physical.impl.WriterRecordBatch.close(WriterRecordBatch.java:221)
> at
> org.apache.drill.common.DeferredException.suppressingClose(DeferredException.java:159)
>at
> org.apache.drill.exec.physical.impl.BaseRootExec.close(BaseRootExec.java:169)
>at
>
[jira] [Commented] (DRILL-7551) Improve Error Reporting
[ https://issues.apache.org/jira/browse/DRILL-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360604#comment-17360604 ] Laurent Goujon commented on DRILL-7551: --- Can this issue considered as resolved? > Improve Error Reporting > --- > > Key: DRILL-7551 > URL: https://issues.apache.org/jira/browse/DRILL-7551 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.17.0 >Reporter: Charles Givre >Priority: Major > Fix For: 1.19.0 > > > This Jira is to serve as a master Jira issue to improve the usability of > error messages. Instead of dumping stack traces, the overall goal is to give > the user something that can actually explain: > # What went wrong > # How to fix > Work that relates to this, should be created as subtasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7914) Apache Drill 1.19.0 Release Activities
[ https://issues.apache.org/jira/browse/DRILL-7914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon resolved DRILL-7914. --- Resolution: Fixed > Apache Drill 1.19.0 Release Activities > -- > > Key: DRILL-7914 > URL: https://issues.apache.org/jira/browse/DRILL-7914 > Project: Apache Drill > Issue Type: Task >Reporter: Laurent Goujon >Assignee: Laurent Goujon >Priority: Major > Fix For: 1.19.0 > > > Source - https://github.com/apache/drill/blob/master/docs/dev/Release.md > 1.19.0 Dashboard: > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336127 > Kanban Board - > https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=185 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7909) GitHub Actions maven Could not transfer artifact from/to central
[
https://issues.apache.org/jira/browse/DRILL-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon updated DRILL-7909:
--
Fix Version/s: (was: 1.19.0)
> GitHub Actions maven Could not transfer artifact from/to central
> -
>
> Key: DRILL-7909
> URL: https://issues.apache.org/jira/browse/DRILL-7909
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build Test
>Affects Versions: 1.18.0
>Reporter: Vitalii Diravka
>Priority: Minor
>
>
> GitHub actions intermittently fails with the following error:
> {code:java}
> Progress (1): 2.5 MB
> 97050
> 97051Downloaded from apache-releases-repo:
> https://repository.apache.org/content/repositories/releases/org/apache/xmlbeans/xmlbeans/4.0.0/xmlbeans-4.0.0.jar
> (2.5 MB at 1.8 MB/s)
> 97052[INFO]
>
> 97053[INFO] Reactor Summary for Drill : 1.19.0-SNAPSHOT:
> 97054[INFO]
> 97055[INFO] Drill : SUCCESS
> [01:25 min]
> 97056[INFO] Drill : Tools : SUCCESS [
> 0.434 s]
> 97057[INFO] Drill : Tools : Freemarker codegen . SUCCESS [
> 45.025 s]
> 97058[INFO] Drill : Protocol ... SUCCESS [
> 15.609 s]
> 97059[INFO] Drill : Common . SUCCESS [
> 22.267 s]
> 97060[INFO] Drill : Logical Plan ... SUCCESS [
> 25.050 s]
> 97061[INFO] Drill : Exec : . SUCCESS [
> 0.350 s]
> 97062[INFO] Drill : Exec : Memory : SUCCESS [
> 0.303 s]
> 97063[INFO] Drill : Exec : Memory : Base ... SUCCESS [
> 9.417 s]
> 97064[INFO] Drill : Exec : RPC . SUCCESS [
> 59.834 s]
> 97065[INFO] Drill : Exec : Vectors . SUCCESS
> [01:34 min]
> 97066[INFO] Drill : Contrib : .. SUCCESS [
> 0.239 s]
> 97067[INFO] Drill : Contrib : Data : ... SUCCESS [
> 0.237 s]
> 97068[INFO] Drill : Contrib : Data : TPCH Sample ... SUCCESS [
> 1.944 s]
> 97069[INFO] Drill : Metastore : SUCCESS [
> 0.250 s]
> 97070[INFO] Drill : Metastore : API SUCCESS [
> 9.793 s]
> 97071[INFO] Drill : Metastore : Iceberg SUCCESS [
> 45.843 s]
> 97072[INFO] Drill : Exec : Java Execution Engine ... SUCCESS
> [38:18 min]
> 97073[INFO] Drill : Exec : JDBC Driver using dependencies .. SUCCESS
> [01:51 min]
> 97074[INFO] Drill : Exec : JDBC JAR with all dependencies .. SUCCESS [
> 40.196 s]
> 97075[INFO] Drill : On-YARN SUCCESS [
> 18.222 s]
> 97076[INFO] Drill : Metastore : RDBMS .. SUCCESS [
> 26.788 s]
> 97077[INFO] Drill : Contrib : Storage : Kudu ... SUCCESS [
> 17.223 s]
> 97078[INFO] Drill : Contrib : Format : XML . SUCCESS [
> 16.269 s]
> 97079[INFO] Drill : Contrib : Storage : HTTP ... SUCCESS [
> 24.527 s]
> 97080[INFO] Drill : Contrib : Storage : OpenTSDB ... SUCCESS [
> 28.559 s]
> 97081[INFO] Drill : Contrib : Storage : MongoDB SUCCESS [
> 29.931 s]
> 97082[INFO] Drill : Contrib : Storage : HBase .. SUCCESS
> [02:28 min]
> 97083[INFO] Drill : Contrib : Storage : JDBC ... SUCCESS [
> 39.633 s]
> 97084[INFO] Drill : Contrib : Storage : Hive : . SUCCESS [
> 0.407 s]
> 97085[INFO] Drill : Contrib : Storage : Hive : Exec Shaded . SUCCESS
> [01:10 min]
> 97086[INFO] Drill : Contrib : Storage : Hive : Core SUCCESS
> [02:06 min]
> 97087[INFO] Drill : Contrib : Storage : Kafka .. SUCCESS
> [04:17 min]
> 97088[INFO] Drill : Contrib : Storage : Cassandra .. SUCCESS
> [01:17 min]
> 97089[INFO] Drill : Contrib : Storage : ElasticSearch .. SUCCESS
> [01:37 min]
> 97090[INFO] Drill : Contrib : Storage : Splunk . SUCCESS
> [02:51 min]
> 97091[INFO] Drill : Contrib : UDFs . SUCCESS [
> 53.366 s]
> 97092[INFO] Drill : Contrib : Format : Syslog .. SUCCESS [
> 16.563 s]
> 97093[INFO] Drill : Contrib : Format : Httpd/Nginx Access Log .. SUCCESS [
> 42.437 s]
> 97094[INFO] Drill : Contrib : Format : HDF5 SUCCESS [
> 22.199 s]
> 97095[INFO] Drill : Contrib : Format : SPSS SUCCESS [
> 15.653 s]
> 97096[INFO] Drill : Contrib : Format : LTSV SUCCESS [
> 13.222 s]
> 97097[INFO] Drill : Contrib : Format :
[jira] [Updated] (DRILL-7907) Replace ParquetFileWriter in Drill with parquet version
[ https://issues.apache.org/jira/browse/DRILL-7907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7907: -- Fix Version/s: (was: 1.19.0) > Replace ParquetFileWriter in Drill with parquet version > --- > > Key: DRILL-7907 > URL: https://issues.apache.org/jira/browse/DRILL-7907 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.18.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Minor > > This is an internal ticket to track PARQUET-2026 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7896) Drill Parquet UUID function
[ https://issues.apache.org/jira/browse/DRILL-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7896: -- Fix Version/s: (was: 1.19.0) > Drill Parquet UUID function > --- > > Key: DRILL-7896 > URL: https://issues.apache.org/jira/browse/DRILL-7896 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill, Storage - Parquet >Affects Versions: 1.18.0 >Reporter: Vitalii Diravka >Priority: Minor > Labels: UUID, parquet > > DRILL-7825 allows to read Parquet UUID datatype. To adjust it to human > readable format it is need to choose or add an appropriate UDF function > (could be implemented via java.util.UUID). For the cases, where UUID datatype > is known from metadata it necessary to create implicit reader with same > semantic as in the above UDF. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7880) Fix LGTM Warnings
[ https://issues.apache.org/jira/browse/DRILL-7880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7880: -- Fix Version/s: (was: 1.19.0) > Fix LGTM Warnings > - > > Key: DRILL-7880 > URL: https://issues.apache.org/jira/browse/DRILL-7880 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Evan Wong >Priority: Minor > > Fix LGTM Warnings > > https://lgtm.com/projects/g/apache/drill/alerts/?mode=list=warning -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7223) Make the timeout in TimedCallable a configurable boot time parameter
[ https://issues.apache.org/jira/browse/DRILL-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7223: -- Fix Version/s: (was: 1.19.0) > Make the timeout in TimedCallable a configurable boot time parameter > > > Key: DRILL-7223 > URL: https://issues.apache.org/jira/browse/DRILL-7223 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.16.0 >Reporter: Aman Sinha >Assignee: Boaz Ben-Zvi >Priority: Minor > > The > [TimedCallable.TIMEOUT_PER_RUNNABLE_IN_MSECS|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/TimedCallable.java#L52] > is currently an internal Drill constant defined as 15 secs. This has been > there from day 1 of the introduction. Drill's TimedCallable implements the > Java concurrency's Callable interface to create timed threads. It is used by > the REFRESH METADATA command which creates multiple threads on the Foreman > node to gather Parquet metadata to build the metadata cache. > Depending on the load on the system or for very large scale number of parquet > files (millions) it is possible to exceed this timeout. While the exact root > cause of exceeding the timeout is being investigated, it makes sense to make > this timeout a configurable parameter to aid with large scale testing. This > JIRA is to make this a configurable bootstrapping option in the > drill-override. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7879) Fix LGTM Errors
[ https://issues.apache.org/jira/browse/DRILL-7879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7879: -- Fix Version/s: (was: 1.19.0) > Fix LGTM Errors > --- > > Key: DRILL-7879 > URL: https://issues.apache.org/jira/browse/DRILL-7879 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.18.0 >Reporter: Evan Wong >Priority: Minor > > Fix LGTM Errors > https://lgtm.com/projects/g/apache/drill/alerts/?mode=list=error -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7922) Add the profile import function
[ https://issues.apache.org/jira/browse/DRILL-7922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7922: -- Fix Version/s: (was: 1.19.0) > Add the profile import function > --- > > Key: DRILL-7922 > URL: https://issues.apache.org/jira/browse/DRILL-7922 > Project: Apache Drill > Issue Type: New Feature >Reporter: Cong Luo >Assignee: wtf >Priority: Major > > We can easy to read the profile metrics on the web console, but hard to > review the profile of other cluster (specified the users. although they can > provide the profile json, but the file is not easy to readable). > Since we can export the profile json via the console, So, We can also add the > import function to quickly review the new cluster metrics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7939) Kafka Key Avro Schema can not cast right
[ https://issues.apache.org/jira/browse/DRILL-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7939: -- Fix Version/s: (was: 1.19.0) > Kafka Key Avro Schema can not cast right > - > > Key: DRILL-7939 > URL: https://issues.apache.org/jira/browse/DRILL-7939 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Kafka >Affects Versions: 1.19.0 >Reporter: cdmikechen >Priority: Major > Attachments: kafka-avro-key.jpg > > > Using drill 1.19.0-snapshot to query kafka avro message, If kafka key data is > avro, drill can not return data right. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7916) Support new plugin installation on the running system
[ https://issues.apache.org/jira/browse/DRILL-7916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7916: -- Fix Version/s: (was: 1.19.0) > Support new plugin installation on the running system > - > > Key: DRILL-7916 > URL: https://issues.apache.org/jira/browse/DRILL-7916 > Project: Apache Drill > Issue Type: New Feature >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > Drill does not support the new plugin installation on the running system : > # Boot the Drill. > # Load plugins to the persistent storage : `pluginStore`. > ## Upgrade the plugin if the override file exist > (storage-plugins-override.conf). (Done) > ## Check and add new plugin with the new release. (To-do) > ## If 1 and 2 are not true, then initial all the plugins via loading > bootstrap configuration. (Done) > # End the Boot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7895) Use Parquet bloom filter
[ https://issues.apache.org/jira/browse/DRILL-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7895: -- Fix Version/s: (was: 1.19.0) > Use Parquet bloom filter > > > Key: DRILL-7895 > URL: https://issues.apache.org/jira/browse/DRILL-7895 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.18.0 >Reporter: Vitalii Diravka >Priority: Major > Labels: bloom-filter, parquet > > The Bloom Filter can be useful in filtering entire row groups > See details in PARQUET umbrella ticket: PARQUET-41 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7869) CSV files can't mix line breaks \x0d Vs. \x0d\x0a
[ https://issues.apache.org/jira/browse/DRILL-7869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7869: -- Fix Version/s: (was: 1.19.0) > CSV files can't mix line breaks \x0d Vs. \x0d\x0a > - > > Key: DRILL-7869 > URL: https://issues.apache.org/jira/browse/DRILL-7869 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Text CSV >Affects Versions: 1.19.0 >Reporter: Curtis Lambert >Priority: Major > > Querying CSV files with \x0d new line delimiters results in "DATA_READ ERROR: > Column exceeds maximum length of 1024" with the default configuration. > The \x0d new line isn't used to break lines resulting in the entire file > being read in as a single record. This is configurable as "delimiter" in the > format but if you have mixed csv files with different line endings it's > problematic. If I have files with both \x0d and \x0d\x0a new lines (\r\n) and > need to be able to read both without having to change the configuration > between queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7621) Refactor ExecConstants and PlannerSettings constant classes
[ https://issues.apache.org/jira/browse/DRILL-7621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7621: -- Fix Version/s: (was: 1.19.0) > Refactor ExecConstants and PlannerSettings constant classes > --- > > Key: DRILL-7621 > URL: https://issues.apache.org/jira/browse/DRILL-7621 > Project: Apache Drill > Issue Type: Task >Reporter: Igor Guzenko >Assignee: Igor Guzenko >Priority: Major > > According to [the > discussion|http://mail-archives.apache.org/mod_mbox/drill-dev/202003.mbox/%3CBCB4CFC2-8BC5-43C6-8BD4-956F66F6D0D3%40gmail.com%3E], > it makes sense to split the classes into multiple constant interfaces and > get rid of validator constants. Then the validator instances won't be used > for getting option values and the general approach will be getting type > specific option value by string key from config instance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7671) Fix builds for cdh and hdp profiles
[ https://issues.apache.org/jira/browse/DRILL-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7671: -- Fix Version/s: (was: 1.19.0) > Fix builds for cdh and hdp profiles > --- > > Key: DRILL-7671 > URL: https://issues.apache.org/jira/browse/DRILL-7671 > Project: Apache Drill > Issue Type: Task >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > > cdh and hdp profiles use too obsolete versions of Hadoop and other libraries. > So when attempting to build the project with these profiles, the build fails > with compilation errors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7366) Improve Null Handling for UDFs with Complex Output
[
https://issues.apache.org/jira/browse/DRILL-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon updated DRILL-7366:
--
Fix Version/s: (was: 1.19.0)
> Improve Null Handling for UDFs with Complex Output
> --
>
> Key: DRILL-7366
> URL: https://issues.apache.org/jira/browse/DRILL-7366
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Priority: Major
>
> If there is a UDF which has a complex field (Map or List) as output, Drill
> does not allow the UDF to have nullable input and it creates additional
> complexity when writing these kinds of UDFs.
> I therefore would like to propose that two options be added to the
> FunctionTemplate for null handling: {{EMPTY_LIST_IF_NULL}}, and
> {{EMPTY_MAP_IF_NULL}} which, would simplify UDF creation. I'm envisioning
> that if either of these options were selected, and the UDF receives any null
> value as input, the UDF will return either an empty map or list.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (DRILL-7526) Assertion Error when only type is used with schema in table function
[
https://issues.apache.org/jira/browse/DRILL-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon updated DRILL-7526:
--
Fix Version/s: (was: 1.19.0)
> Assertion Error when only type is used with schema in table function
>
>
> Key: DRILL-7526
> URL: https://issues.apache.org/jira/browse/DRILL-7526
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Arina Ielchiieva
>Assignee: Vova Vysotskyi
>Priority: Major
>
> {{org.apache.drill.TestSchemaWithTableFunction}}
> {noformat}
> @Test
> public void testWithTypeAndSchema() {
> String query = "select Year from
> table(dfs.`store/text/data/cars.csvh`(type=> 'text', " +
> "schema=>'inline=(`Year` int)')) where Make = 'Ford'";
> queryBuilder().sql(query).print();
> }
> {noformat}
> {noformat}
> Caused by: java.lang.AssertionError: BOOLEAN
> at
> org.apache.calcite.sql.type.SqlTypeExplicitPrecedenceList.compareTypePrecedence(SqlTypeExplicitPrecedenceList.java:140)
> at org.apache.calcite.sql.SqlUtil.bestMatch(SqlUtil.java:687)
> at
> org.apache.calcite.sql.SqlUtil.filterRoutinesByTypePrecedence(SqlUtil.java:656)
> at
> org.apache.calcite.sql.SqlUtil.lookupSubjectRoutines(SqlUtil.java:515)
> at org.apache.calcite.sql.SqlUtil.lookupRoutine(SqlUtil.java:435)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:240)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:218)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5640)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5627)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1692)
> at
> org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl(ProcedureNamespace.java:53)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1009)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:969)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3129)
> at
> org.apache.drill.exec.planner.sql.conversion.DrillValidator.validateFrom(DrillValidator.java:63)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3111)
> at
> org.apache.drill.exec.planner.sql.conversion.DrillValidator.validateFrom(DrillValidator.java:63)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3383)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1009)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:969)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:944)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:651)
> at
> org.apache.drill.exec.planner.sql.conversion.SqlConverter.validate(SqlConverter.java:189)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:648)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:196)
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:170)
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:283)
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:163)
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:128)
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:93)
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:590)
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:275)
> ... 1 more
> {noformat}
> Note: when other format options are used or schema is used alone, everything
> works fine.
> See test examples:
>
[jira] [Updated] (DRILL-7192) Drill limits rows when autoLimit is disabled
[
https://issues.apache.org/jira/browse/DRILL-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon updated DRILL-7192:
--
Fix Version/s: (was: 1.19.0)
> Drill limits rows when autoLimit is disabled
>
>
> Key: DRILL-7192
> URL: https://issues.apache.org/jira/browse/DRILL-7192
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Vova Vysotskyi
>Assignee: Vova Vysotskyi
>Priority: Major
>
> In DRILL-7048 was implemented autoLimit for JDBC and rest clients.
> *Steps to reproduce the issue:*
> 1. Check that autoLimit was disabled, if not, disable it and restart Drill.
> 2. Submit any query, and verify that rows count is correct, for example,
> {code:sql}
> SELECT * FROM cp.`employee.json`;
> {code}
> returns 1,155 rows
> 3. Enable autoLimit for sqlLine sqlLine client:
> {code:sql}
> !set rowLimit 10
> {code}
> 4. Submit the same query and verify that the result has 10 rows.
> 5. Disable autoLimit:
> {code:sql}
> !set rowLimit 0
> {code}
> 6. Submit the same query, but for this time, *it returns 10 rows instead of
> 1,155*.
> Correct rows count is returned only after creating a new connection.
> The same issue is also observed for SQuirreL SQL client, but for example, for
> Postgres, it works correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (DRILL-7133) Duplicate Corrupt PCAP Functionality in PCAP-NG Plugin
[
https://issues.apache.org/jira/browse/DRILL-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon updated DRILL-7133:
--
Fix Version/s: (was: 1.19.0)
> Duplicate Corrupt PCAP Functionality in PCAP-NG Plugin
> --
>
> Key: DRILL-7133
> URL: https://issues.apache.org/jira/browse/DRILL-7133
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
>
> There was a JIRA (https://issues.apache.org/jira/browse/DRILL-7032) which
> resulted in some improvements to the PCAP format plugin which converted the
> TCP flags to boolean format and also added a {{is_corrupt}} boolean field.
> This field allows users to look for packets that are corrupt.
> Unfortunately, this functionality is not duplicated in the PCAP-NG format
> plugin, so this JIRA proposes to do that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (DRILL-7284) reusing the hashCodes computed at exchange nodes
[ https://issues.apache.org/jira/browse/DRILL-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7284: -- Fix Version/s: (was: 1.19.0) > reusing the hashCodes computed at exchange nodes > > > Key: DRILL-7284 > URL: https://issues.apache.org/jira/browse/DRILL-7284 > Project: Apache Drill > Issue Type: New Feature >Reporter: Weijie Tong >Assignee: Weijie Tong >Priority: Major > > To HashJoin or HashAggregate, we will shuffle the input data according to > hashCodes of join conditions or group by keys at the exchange nodes. This > computing of the hash codes will be redo at the HashJoin or HashAggregate > nodes. We could send the computed hashCodes of exchange nodes to the upper > nodes. So the HashJoin or HashAggregate nodes will not need to do the hash > computing again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7938) Convert JDBC Storage Plugin to EVF
[ https://issues.apache.org/jira/browse/DRILL-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7938: -- Fix Version/s: (was: 1.19.0) > Convert JDBC Storage Plugin to EVF > -- > > Key: DRILL-7938 > URL: https://issues.apache.org/jira/browse/DRILL-7938 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - JDBC >Affects Versions: 1.18.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Minor > > The JDBC plugin currently uses the older vector objects. As part of a > massive code cleanup, this PR updates the plugin to use the newer EVF > framework. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7906) Replace ParquetColumnChunkPageWriter with original Parquet class
[ https://issues.apache.org/jira/browse/DRILL-7906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7906: -- Fix Version/s: (was: 1.19.0) > Replace ParquetColumnChunkPageWriter with original Parquet class > > > Key: DRILL-7906 > URL: https://issues.apache.org/jira/browse/DRILL-7906 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.18.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Minor > > This is Drill internal ticket to handle PARQUET-1006 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7921) Support the Linux ARM64 based system
[ https://issues.apache.org/jira/browse/DRILL-7921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon resolved DRILL-7921. --- Resolution: Fixed > Support the Linux ARM64 based system > > > Key: DRILL-7921 > URL: https://issues.apache.org/jira/browse/DRILL-7921 > Project: Apache Drill > Issue Type: New Feature >Reporter: Cong Luo >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Fix For: 1.19.0 > > > This is an umbrella Jira for support to running on Linux ARM64 based system. > # Build must be passed. > # All the JUnit test must be passed. > # Running stable on the Linux aarch64 machines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7871) StoragePluginStore instances for different users
[ https://issues.apache.org/jira/browse/DRILL-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7871: -- Fix Version/s: (was: 1.19.0) > StoragePluginStore instances for different users > > > Key: DRILL-7871 > URL: https://issues.apache.org/jira/browse/DRILL-7871 > Project: Apache Drill > Issue Type: New Feature > Components: Security >Affects Versions: 1.18.0 >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > > Different users should have their own storage plugin configs to have access > to own storages only. The feature can be based on Drill User Impersonation > model -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7554) Convert LTSV Format Plugin to EVF
[ https://issues.apache.org/jira/browse/DRILL-7554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7554: -- Fix Version/s: (was: 1.19.0) > Convert LTSV Format Plugin to EVF > - > > Key: DRILL-7554 > URL: https://issues.apache.org/jira/browse/DRILL-7554 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Text CSV >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7531) Convert format plugins to EVF
[ https://issues.apache.org/jira/browse/DRILL-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7531: -- Fix Version/s: (was: 1.19.0) > Convert format plugins to EVF > - > > Key: DRILL-7531 > URL: https://issues.apache.org/jira/browse/DRILL-7531 > Project: Apache Drill > Issue Type: Improvement >Reporter: Arina Ielchiieva >Priority: Major > > This is umbrella Jira to track down process of converting format plugins to > EVF. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-4232) Support for EXCEPT set operator
[ https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-4232: -- Fix Version/s: (was: 1.19.0) > Support for EXCEPT set operator > --- > > Key: DRILL-4232 > URL: https://issues.apache.org/jira/browse/DRILL-4232 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning Optimization >Reporter: Victoria Markman >Assignee: Bohdan Kazydub >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7946) Bump HttpClient from 4.5.12 to 4.5.13 for CVE-2020-13956
[ https://issues.apache.org/jira/browse/DRILL-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon resolved DRILL-7946. --- Resolution: Fixed > Bump HttpClient from 4.5.12 to 4.5.13 for CVE-2020-13956 > > > Key: DRILL-7946 > URL: https://issues.apache.org/jira/browse/DRILL-7946 > Project: Apache Drill > Issue Type: Improvement >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > Fix For: 1.19.0 > > > Apache HttpClient versions prior to version 4.5.13 and 5.0.3 can misinterpret > malformed authority component in request URIs passed to the library as > java.net.URI object and pick the wrong target host for request execution. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7945) Unable to patch Guava classes
[
https://issues.apache.org/jira/browse/DRILL-7945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon resolved DRILL-7945.
---
Assignee: Laurent Goujon (was: Vitalii Diravka)
Resolution: Fixed
> Unable to patch Guava classes
> -
>
> Key: DRILL-7945
> URL: https://issues.apache.org/jira/browse/DRILL-7945
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Vova Vysotskyi
>Assignee: Laurent Goujon
>Priority: Blocker
> Fix For: 1.19.0
>
>
> When starting Drill in embedded mode, logs have the following line:
> {noformat}
> 2021-06-03 21:59:07,711 [main] WARN o.a.drill.common.util.GuavaPatcher -
> Unable to patch Guava classes: [source error]
> format(java.lang.String,java.lang.Object[]) not found in
> com.google.common.base.Preconditions
> {noformat}
> Most likely it is a regression after DRILL-7904.
> The importance of this issue is that all 3rd party libraries that initially
> relied on patched methods from {{Preconditions}} will stop working because of
> method not found errors during the runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7947) Unable to aggregate or filter nullable decimals in large record sets
[
https://issues.apache.org/jira/browse/DRILL-7947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357593#comment-17357593
]
Laurent Goujon commented on DRILL-7947:
---
The same query also fails on 1.18.0, although with a different error message:
{noformat}
Error: INTERNAL_ERROR ERROR: nullFragment: 0:0Please, refer to logs for more
information.[Error Id: a310dbf1-593e-4c4e-8fd0-1e3e00709077 on localhost:31010]
(state=,code=0){noformat}
logs show this:
{noformat}
[Error Id: a310dbf1-593e-4c4e-8fd0-1e3e00709077 on localhost:31010]
at
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:125)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException:
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:234)
at
org.apache.drill.exec.physical.impl.ScanBatch.internalNext(ScanBatch.java:234)
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:298)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:111)
at
[jira] [Commented] (DRILL-7948) Unable to query file with required fixed_len_byte_array decimal columns
[
https://issues.apache.org/jira/browse/DRILL-7948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357592#comment-17357592
]
Laurent Goujon commented on DRILL-7948:
---
I just verified that 1.18.0 release has the same issue
> Unable to query file with required fixed_len_byte_array decimal columns
> ---
>
> Key: DRILL-7948
> URL: https://issues.apache.org/jira/browse/DRILL-7948
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.19.0, 1.18.0
> Environment: Docker: master
> Sys.Version:
> |1.19.0-SNAPSHOT|82c03e44eac3863a10e599d8f2068fc58e986041|
>
>Reporter: Nick Stenroos-Dam
>Priority: Blocker
> Attachments: sqlline.log
>
>
> When attempting to query a file with required fixed_len_byte_array decimal
> columns, the exception: *INTERNAL_ERROR ERROR: Error in drill parquet reader*
> is thrown
> Full Exception:
>
> {noformat}
> {{org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR
> ERROR: Error in drill parquet reader (complex). Message: Failure in setting
> up reader Parquet Metadata: ParquetMetaData{FileMetaData{schema: message
> schema
> { required fixed_len_byte_array(16) Sale (DECIMAL(29,6)); required
> fixed_len_byte_array(16) Cost (DECIMAL(29,6)); }
> , metadata: {}}, blocks: [BlockMetaData{1, 260 [ColumnMetaData
> {SNAPPY [Sale] required fixed_len_byte_array(16) Sale (DECIMAL(29,6)) [RLE,
> PLAIN, PLAIN_DICTIONARY], 36}
> , ColumnMetaData
> {SNAPPY [Cost] required fixed_len_byte_array(16) Cost (DECIMAL(29,6)) [RLE,
> PLAIN, PLAIN_DICTIONARY], 292}
> ]}]} Fragment: 0:0 Please, refer to logs for more information. [Error Id:
> 772dfd77-1eaf-4cbc-9b7b-4d4a1e5c3080 on 67732da24351:31010]}}
> {noformat}
>
> Test Query:
> {code:sql}
> select * from data.`SingleRow_RequiredFixedLength_Decimal.parquet`{code}
> Test File:
> [https://projectbiaps-my.sharepoint.com/:u:/g/personal/nisd_project_bi/ET-9G1pyXmJGm6AaLv9BTMsBBInk-EuvFRWEeA-gAqXDwQ?e=wkAH7H]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (DRILL-7948) Unable to query file with required fixed_len_byte_array decimal columns
[
https://issues.apache.org/jira/browse/DRILL-7948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon updated DRILL-7948:
--
Affects Version/s: 1.18.0
> Unable to query file with required fixed_len_byte_array decimal columns
> ---
>
> Key: DRILL-7948
> URL: https://issues.apache.org/jira/browse/DRILL-7948
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.19.0, 1.18.0
> Environment: Docker: master
> Sys.Version:
> |1.19.0-SNAPSHOT|82c03e44eac3863a10e599d8f2068fc58e986041|
>
>Reporter: Nick Stenroos-Dam
>Priority: Blocker
> Attachments: sqlline.log
>
>
> When attempting to query a file with required fixed_len_byte_array decimal
> columns, the exception: *INTERNAL_ERROR ERROR: Error in drill parquet reader*
> is thrown
> Full Exception:
>
> {noformat}
> {{org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR
> ERROR: Error in drill parquet reader (complex). Message: Failure in setting
> up reader Parquet Metadata: ParquetMetaData{FileMetaData{schema: message
> schema
> { required fixed_len_byte_array(16) Sale (DECIMAL(29,6)); required
> fixed_len_byte_array(16) Cost (DECIMAL(29,6)); }
> , metadata: {}}, blocks: [BlockMetaData{1, 260 [ColumnMetaData
> {SNAPPY [Sale] required fixed_len_byte_array(16) Sale (DECIMAL(29,6)) [RLE,
> PLAIN, PLAIN_DICTIONARY], 36}
> , ColumnMetaData
> {SNAPPY [Cost] required fixed_len_byte_array(16) Cost (DECIMAL(29,6)) [RLE,
> PLAIN, PLAIN_DICTIONARY], 292}
> ]}]} Fragment: 0:0 Please, refer to logs for more information. [Error Id:
> 772dfd77-1eaf-4cbc-9b7b-4d4a1e5c3080 on 67732da24351:31010]}}
> {noformat}
>
> Test Query:
> {code:sql}
> select * from data.`SingleRow_RequiredFixedLength_Decimal.parquet`{code}
> Test File:
> [https://projectbiaps-my.sharepoint.com/:u:/g/personal/nisd_project_bi/ET-9G1pyXmJGm6AaLv9BTMsBBInk-EuvFRWEeA-gAqXDwQ?e=wkAH7H]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7945) Unable to patch Guava classes
[
https://issues.apache.org/jira/browse/DRILL-7945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356761#comment-17356761
]
Laurent Goujon commented on DRILL-7945:
---
It looks like this is not as serious as thought: Several methods were added in
the {{GuavaPatcher}} class because Apache Iceberg was using a more recent
version of Guava than Drill, but by moving to Guava 30, those methods are not
necessary anymore, and the reason the patch fails is because the formatting
method used in previous version of Guava (and used to patch the code) has been
removed since. But even if patching fails, because the methods used by Iceberg
are still present, there's no compatibility issues.
It seems anyway that we should add a test case to make sure patching doesn't
fail so that the next time Guava is updated, we can be alerted about checking
the state of this class.
> Unable to patch Guava classes
> -
>
> Key: DRILL-7945
> URL: https://issues.apache.org/jira/browse/DRILL-7945
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Vova Vysotskyi
>Assignee: Vitalii Diravka
>Priority: Blocker
> Fix For: 1.19.0
>
>
> When starting Drill in embedded mode, logs have the following line:
> {noformat}
> 2021-06-03 21:59:07,711 [main] WARN o.a.drill.common.util.GuavaPatcher -
> Unable to patch Guava classes: [source error]
> format(java.lang.String,java.lang.Object[]) not found in
> com.google.common.base.Preconditions
> {noformat}
> Most likely it is a regression after DRILL-7904.
> The importance of this issue is that all 3rd party libraries that initially
> relied on patched methods from {{Preconditions}} will stop working because of
> method not found errors during the runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-7795) Add option to native Drill Client to set TLS SNI property
[ https://issues.apache.org/jira/browse/DRILL-7795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon resolved DRILL-7795. --- Fix Version/s: 1.19.0 Resolution: Fixed > Add option to native Drill Client to set TLS SNI property > - > > Key: DRILL-7795 > URL: https://issues.apache.org/jira/browse/DRILL-7795 > Project: Apache Drill > Issue Type: Improvement > Components: Client - C++ >Reporter: James Duong >Priority: Major > Fix For: 1.19.0 > > > Add an option to the C++ DrillClient library that sets the TLS SNI extension > to let you set the Server Name Indication TLS property. If no option for this > value is specified, choose a default based on the target host. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7917) Update Kafka to the latest version
[ https://issues.apache.org/jira/browse/DRILL-7917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-7917: -- Fix Version/s: 1.19.0 > Update Kafka to the latest version > -- > > Key: DRILL-7917 > URL: https://issues.apache.org/jira/browse/DRILL-7917 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.19.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.19.0 > > > Update Kafka to the 2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-6547) IllegalStateException: Tried to remove unmanaged buffer.
[
https://issues.apache.org/jira/browse/DRILL-6547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon resolved DRILL-6547.
---
Fix Version/s: 1.19.0
Resolution: Fixed
> IllegalStateException: Tried to remove unmanaged buffer.
>
>
> Key: DRILL-6547
> URL: https://issues.apache.org/jira/browse/DRILL-6547
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.14.0
>Reporter: Robert Hou
>Assignee: Pritesh Maker
>Priority: Major
> Fix For: 1.19.0
>
>
> This is the query:
> select * from (
> select Index, concat(BinaryValue, 'aaa') NewVarcharValue from (select * from
> dfs.`/drill/testdata/batch_memory/alltypes_large_1MB.parquet`)) d where
> d.Index = 1;
> This is the plan:
> {noformat}
> | 00-00Screen
> 00-01 Project(Index=[$0], NewVarcharValue=[$1])
> 00-02SelectionVectorRemover
> 00-03 Filter(condition=[=($0, 1)])
> 00-04Project(Index=[$0], NewVarcharValue=[CONCAT($1, 'aaa')])
> 00-05 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath
> [path=maprfs:///drill/testdata/batch_memory/alltypes_large_1MB.parquet]],
> selectionRoot=maprfs:/drill/testdata/batch_memory/alltypes_large_1MB.parquet,
> numFiles=1, numRowGroups=1, usedMetadataFile=false, columns=[`Index`,
> `BinaryValue`]]])
> {noformat}
> Here is the stack trace from drillbit.log:
> {noformat}
> 2018-06-27 13:55:03,291 [24cc0659-30b7-b290-7fae-ecb1c1f15c05:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
> Tried to remove unmanaged buffer.
> Fragment 0:0
> [Error Id: bc1f2f72-c31b-4b9a-964f-96dec9e0f388 on qa-node186.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Tried to remove unmanaged buffer.
> Fragment 0:0
> [Error Id: bc1f2f72-c31b-4b9a-964f-96dec9e0f388 on qa-node186.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.IllegalStateException: Tried to remove unmanaged buffer.
> at
> org.apache.drill.exec.ops.BufferManagerImpl.replace(BufferManagerImpl.java:50)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at io.netty.buffer.DrillBuf.reallocIfNeeded(DrillBuf.java:97)
> ~[drill-memory-base-1.14.0-SNAPSHOT.jar:4.0.48.Final]
> at
> org.apache.drill.exec.test.generated.ProjectorGen4046.doEval(ProjectorTemplate.java:77)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.ProjectorGen4046.projectRecords(ProjectorTemplate.java:67)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:236)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:117)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:147)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
>
[jira] [Updated] (DRILL-5940) Avro with schema registry support for Kafka
[ https://issues.apache.org/jira/browse/DRILL-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-5940: -- Fix Version/s: 1.19.0 > Avro with schema registry support for Kafka > --- > > Key: DRILL-5940 > URL: https://issues.apache.org/jira/browse/DRILL-5940 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Avro, Storage - Kafka >Reporter: AnilKumar B >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.19.0 > > > Support Avro messages with Schema registry for Kafka storage plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (DRILL-7928) Add fourth parameter for split_part udf
[
https://issues.apache.org/jira/browse/DRILL-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon closed DRILL-7928.
-
Resolution: Fixed
> Add fourth parameter for split_part udf
> ---
>
> Key: DRILL-7928
> URL: https://issues.apache.org/jira/browse/DRILL-7928
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
>Reporter: wtf
>Assignee: wtf
>Priority: Minor
> Fix For: 1.19.0
>
>
> suport split_part('a,b,c,d', ',' , 1, 2) = 'a,b'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (DRILL-7928) Add fourth parameter for split_part udf
[
https://issues.apache.org/jira/browse/DRILL-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon updated DRILL-7928:
--
Fix Version/s: 1.19.0
> Add fourth parameter for split_part udf
> ---
>
> Key: DRILL-7928
> URL: https://issues.apache.org/jira/browse/DRILL-7928
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
>Reporter: wtf
>Assignee: wtf
>Priority: Minor
> Fix For: 1.19.0
>
>
> suport split_part('a,b,c,d', ',' , 1, 2) = 'a,b'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-7936) Remove Guava Files#createTempDir usage
[ https://issues.apache.org/jira/browse/DRILL-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon resolved DRILL-7936. --- Fix Version/s: 1.19.0 Reviewer: Charles Givre Resolution: Fixed > Remove Guava Files#createTempDir usage > -- > > Key: DRILL-7936 > URL: https://issues.apache.org/jira/browse/DRILL-7936 > Project: Apache Drill > Issue Type: Bug >Reporter: Laurent Goujon >Assignee: Laurent Goujon >Priority: Major > Fix For: 1.19.0 > > > According to CVE-2020-8908, method Files#createTempDir has some security > issues, and usage has been deprecated in Guava 30.0. > Usage of this function in Drill should be replaced with the native Java > version as recommended by the Guava team. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7162) Apache Drill uses 3rd Party with Highest CVEs
[ https://issues.apache.org/jira/browse/DRILL-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon resolved DRILL-7162. --- Fix Version/s: 1.19.0 Resolution: Fixed Apache Drill has been updated to the latest Jetty 9.4 version available. This should address all the CVEs on the list. > Apache Drill uses 3rd Party with Highest CVEs > -- > > Key: DRILL-7162 > URL: https://issues.apache.org/jira/browse/DRILL-7162 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.13.0, 1.14.0, 1.15.0 >Reporter: Ayush Sharma >Priority: Major > Fix For: 1.19.0 > > Attachments: Jars.xlsx > > > Apache Drill uses 3rd party libraries with almost 250+ CVEs. > Most of the CVEs are in the older version of Jetty (9.1.x) whereas the > current version of Jetty is 9.4.x > Also many of the other libraries are in EOF versions and the are not patched > even in the latest release. > This creates an issue of security when we use it in production. > We are able to replace many older version of libraries with the latest > versions with no CVEs , however many of them are not replaceable as it is and > would require some changes in the source code. > The jetty version is of the highest priority and needs migration to 9.4.x > version immediately. > > Please look into this issue at immediate priority as it compromises with the > security of the application utilizing Apache Drill. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7135) Upgrade to Jetty 9.4
[ https://issues.apache.org/jira/browse/DRILL-7135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon resolved DRILL-7135. --- Fix Version/s: 1.19.0 Reviewer: Paul Rogers Assignee: Laurent Goujon Resolution: Fixed > Upgrade to Jetty 9.4 > > > Key: DRILL-7135 > URL: https://issues.apache.org/jira/browse/DRILL-7135 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.15.0 >Reporter: Vitalii Diravka >Assignee: Laurent Goujon >Priority: Minor > Fix For: 1.19.0 > > > Initially DRILL-7051 updated Jetty to 9.4 version and DRILL-7081 updated > Jersey version to 2.28 version. These versions work fine for Drill with > Hadoop version below 3.0. > Starting from Hadoop 3.0 it uses > [org.eclipse.jetty|https://github.com/apache/hadoop/blob/branch-3.0/hadoop-project/pom.xml#L38] > 9.3 version. > That's why it conflicts with newer Jetty versions. > Drill can update Jetty and Jersey versions after resolution HADOOP-14930 and > HBASE-19256. > Or alternatively these libs can be shaded in Drill, but there is no real > reason to do it nowadays. > See details in > [#1681|https://github.com/apache/drill/pull/1681#discussion_r265904521] PR. > _Notes_: > * For Jersey update it is necessary to add > org.glassfish.jersey.inject:jersey-hk2 in Drill to solve all compilation > failures. > * See doc for Jetty update: > https://www.eclipse.org/jetty/documentation/9.4.x/upgrading-jetty.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7932) Update Hadoop version to 3.2.2.
[ https://issues.apache.org/jira/browse/DRILL-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon resolved DRILL-7932. --- Fix Version/s: 1.19.0 Reviewer: Cong Luo Resolution: Fixed Update done as part of the fix for DRILL-7135 > Update Hadoop version to 3.2.2. > --- > > Key: DRILL-7932 > URL: https://issues.apache.org/jira/browse/DRILL-7932 > Project: Apache Drill > Issue Type: Improvement >Reporter: Laurent Goujon >Assignee: Laurent Goujon >Priority: Major > Fix For: 1.19.0 > > > Update Hadoop version to the latest patch version for the 3.2.x branch. > This should address possible security vulnerability > [https://nvd.nist.gov/vuln/detail/CVE-2020-9492|CVE-2020-9492] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7936) Remove Guava Files#createTempDir usage
Laurent Goujon created DRILL-7936: - Summary: Remove Guava Files#createTempDir usage Key: DRILL-7936 URL: https://issues.apache.org/jira/browse/DRILL-7936 Project: Apache Drill Issue Type: Bug Reporter: Laurent Goujon Assignee: Laurent Goujon According to CVE-2020-8908, method Files#createTempDir has some security issues, and usage has been deprecated in Guava 30.0. Usage of this function in Drill should be replaced with the native Java version as recommended by the Guava team. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7932) Update Hadoop version to 3.2.2.
Laurent Goujon created DRILL-7932: - Summary: Update Hadoop version to 3.2.2. Key: DRILL-7932 URL: https://issues.apache.org/jira/browse/DRILL-7932 Project: Apache Drill Issue Type: Improvement Reporter: Laurent Goujon Assignee: Laurent Goujon Update Hadoop version to the latest patch version for the 3.2.x branch. This should address possible security vulnerability [https://nvd.nist.gov/vuln/detail/CVE-2020-9492|CVE-2020-9492] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7914) Apache Drill 1.19.0 Release Activities
Laurent Goujon created DRILL-7914: - Summary: Apache Drill 1.19.0 Release Activities Key: DRILL-7914 URL: https://issues.apache.org/jira/browse/DRILL-7914 Project: Apache Drill Issue Type: Task Reporter: Laurent Goujon Assignee: Laurent Goujon Source - https://github.com/apache/drill/blob/master/docs/dev/Release.md 1.19.0 Dashboard: https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336127 Kanban Board - https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=185 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7726) Boost requirement is incorrect
[ https://issues.apache.org/jira/browse/DRILL-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon resolved DRILL-7726. --- Fix Version/s: 1.18.0 Resolution: Fixed > Boost requirement is incorrect > -- > > Key: DRILL-7726 > URL: https://issues.apache.org/jira/browse/DRILL-7726 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Reporter: Laurent Goujon >Assignee: Laurent Goujon >Priority: Major > Fix For: 1.18.0 > > > Drill C++ connector documentation states that Boost 1.53 is required, but as > support for tlsv12 is present, Boost 1.54 is actually required. Trying to > compile with Boost 1.53 actually result in a compilation error for undefined > constant -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7714) Build DrillClient as a static archive
[ https://issues.apache.org/jira/browse/DRILL-7714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099160#comment-17099160 ] Laurent Goujon commented on DRILL-7714: --- I guess the issue is twofold: - drillClient library is opened by a program which has a different libssl loaded - there is some incompatibility between the openssl version so code compiled with version X cannot use version Y of the library? I guess it doesn't have to be a static library vs linking statically openssl to the dynamic library and making sure symbols are hidden so that there's no conflict? As for the patch, it looks like a static library is created which also contains all the objects present in the dynamic libraries the program is linked for. But I haven't seen this done for other oss projects, and usually static libraries only contains objects from the project itself. Is there a strong incentive of doing this vs letting the user of the static library doing it themselves? > Build DrillClient as a static archive > - > > Key: DRILL-7714 > URL: https://issues.apache.org/jira/browse/DRILL-7714 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.17.0 >Reporter: Sidharth Jha >Priority: Major > Attachments: 00177131.zip, stacktraces.txt > > > Hi, > > We encountered an issue recently which results due to the fact that the > application loads a different version of an openssl symbol (SSL_CTX_NEW from > 1.0.0) than what the boost archive (used by drill client) was being built > with (i.e., SSL_CTX_NEW 1.1.1). Attaching a sample stack trace, and my > standalone reproducer. Now there are two ways to fix this. > # Use a custom versioning script, and pass it to the linker to only put the > symbols in the .global section which need to be imported by other libraries > using drillClient. Needless to say, this would make sure that symbols such as > SSL_CTX_NEW doesn't show up in .dynsym, and cause issues with the runtime > loader. > # Build drill client as an archive to allow static linking to any > application, so that it can use a versioning script similar to what's > mentioned in the previous point to ensure that this ambiguity does not occur. > The latter is easier to maintain, from DrillClient's perspective. > Further details are in the attached archive. Please refer to the readmes > within. > > Thanks & Regards, > Sid -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7727) Address Protobuf C++ warnings
Laurent Goujon created DRILL-7727:
-
Summary: Address Protobuf C++ warnings
Key: DRILL-7727
URL: https://issues.apache.org/jira/browse/DRILL-7727
Project: Apache Drill
Issue Type: Test
Components: Client - C++
Reporter: Laurent Goujon
Assignee: Laurent Goujon
Compiling C++ client with recent versions of protobuf (3.4.0 or higher)
produces compilation warnings:
{noformat}
.../drill/contrib/native/client/src/clientlib/rpcMessage.cpp:208:32: warning:
'ByteSize' is deprecated: Please use ByteSizeLong() instead
[-Wdeprecated-declarations]
int header_length = header.ByteSize();
^
/usr/local/include/google/protobuf/message_lite.h:401:3: note: 'ByteSize' has
been explicitly marked deprecated here
PROTOBUF_DEPRECATED_MSG("Please use ByteSizeLong() instead")
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7726) Boost requirement is incorrect
Laurent Goujon created DRILL-7726: - Summary: Boost requirement is incorrect Key: DRILL-7726 URL: https://issues.apache.org/jira/browse/DRILL-7726 Project: Apache Drill Issue Type: Bug Components: Client - C++ Reporter: Laurent Goujon Assignee: Laurent Goujon Drill C++ connector documentation states that Boost 1.53 is required, but as support for tlsv12 is present, Boost 1.54 is actually required. Trying to compile with Boost 1.53 actually result in a compilation error for undefined constant -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-5582) [Threat Modeling] Drillbit may be spoofed by an attacker and this may lead to data being written to the attacker's target instead of Drillbit
[ https://issues.apache.org/jira/browse/DRILL-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225474#comment-16225474 ] Laurent Goujon commented on DRILL-5582: --- It seems this change is backward incompatible (newer JDBC/C++ connector cannot be used with an older server server). I'm not sure to follow the threat model either: since there's no server authentication (something TLS would solve), what prevents a MITM drillbit to just fake authentication? > [Threat Modeling] Drillbit may be spoofed by an attacker and this may lead to > data being written to the attacker's target instead of Drillbit > - > > Key: DRILL-5582 > URL: https://issues.apache.org/jira/browse/DRILL-5582 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Rob Wu >Assignee: Sorabh Hamirwasia >Priority: Minor > Labels: doc-impacting > Fix For: 1.12.0 > > > *Consider the scenario:* > Alice has a drillbit (my.drillbit.co) with plain and kerberos authentication > enabled containing important data. Bob, the attacker, attempts to spoof the > connection and redirect it to his own drillbit (fake.drillbit.co) with no > authentication setup. > When Alice is under attack and attempts to connect to her secure drillbit, > she is actually authenticating against Bob's drillbit. At this point, the > connection should have failed due to unmatched configuration. However, the > current implementation will return SUCCESS as long as the (spoofing) drillbit > has no authentication requirement set. > Currently, the drillbit <- to -> drill client connection accepts the lowest > authentication configuration set on the server. This leaves unsuspecting user > vulnerable to spoofing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5431) Support SSL
[
https://issues.apache.org/jira/browse/DRILL-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191789#comment-16191789
]
Laurent Goujon commented on DRILL-5431:
---
{quote}
Laurent Goujon I did put in the support to read from the Windows certificate
store for the C++ client, and support for the Mac Keychain and Windows
certificate store for the Java client which makes it a little more palatable
for organizations that want to have their CA at the OS level. It is not too
hard to add the hooks to make the libraries accept a trust store verifier, but
if we have the ability to read the system trust store then it may not be too
useful any more.
{quote}
I don't know for Java on Windows, but on Mac, it doesn't seem to be integrated
with Apple Keychain, and rely on its own truststore (located at
{{/Library/Java/JavaVirtualMachines/*/Contents/Home/jre/lib/security/cacerts}}).
OpenJDK packages on Centos/Redhat seems to be integrated with the system
keystore though.
> Support SSL
> ---
>
> Key: DRILL-5431
> URL: https://issues.apache.org/jira/browse/DRILL-5431
> Project: Apache Drill
> Issue Type: New Feature
> Components: Client - Java, Client - ODBC
>Reporter: Sudheesh Katkam
>Assignee: Parth Chandra
> Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports
> HTTPS for web traffic.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5431) Support SSL
[ https://issues.apache.org/jira/browse/DRILL-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166725#comment-16166725 ] Laurent Goujon commented on DRILL-5431: --- (It is also regrettable that OpenJDK/Oracle JDK has a different trust store than the OS one, or no way of integrating it) > Support SSL > --- > > Key: DRILL-5431 > URL: https://issues.apache.org/jira/browse/DRILL-5431 > Project: Apache Drill > Issue Type: New Feature > Components: Client - Java, Client - ODBC >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam > > Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports > HTTPS for web traffic. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5431) Support SSL
[ https://issues.apache.org/jira/browse/DRILL-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166721#comment-16166721 ] Laurent Goujon commented on DRILL-5431: --- I had a cursory look at the commits. One comment I have is about maybe some lack of flexibility regarding integration with external truststore for the client (and similar for hostname verification). For the JDBC driver, because of the way of driver is loaded, and connections are created, everything has to be done through connection properties :( But it doesn't have to be the case for the C++ and Java Drill clients. I believe that similar to http clients where you can provide trust stores and hostname verifier, these clients should have the same capabilities. It is probably more of the requirement for the C++ client than the Java one, as JRE comes up with a truststore similar to the browser ones, whereas OpenSSL library has maybe none on windows (or for organizations having their own CA integrated at the OS level, might require special care). Maybe [~robertw] has some opinion on this based on this experience on integrating drivers on various platforms? > Support SSL > --- > > Key: DRILL-5431 > URL: https://issues.apache.org/jira/browse/DRILL-5431 > Project: Apache Drill > Issue Type: New Feature > Components: Client - Java, Client - ODBC >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam > > Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports > HTTPS for web traffic. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5431) Support SSL
[ https://issues.apache.org/jira/browse/DRILL-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124141#comment-16124141 ] Laurent Goujon commented on DRILL-5431: --- Can we allow comments for the document? or shall we comment on the JIRA itself? > Support SSL > --- > > Key: DRILL-5431 > URL: https://issues.apache.org/jira/browse/DRILL-5431 > Project: Apache Drill > Issue Type: New Feature > Components: Client - Java, Client - ODBC >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam > > Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports > HTTPS for web traffic. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5659) C++ Client (master) behavior is unstable resulting incorrect result or exception in API calls
[
https://issues.apache.org/jira/browse/DRILL-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084243#comment-16084243
]
Laurent Goujon commented on DRILL-5659:
---
I think I reproduce the issue with the following query:
{code:sql}
SELECT
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
FROM (VALUES(1))
{code}
The JDBC driver doesn't seem to be affected
> C++ Client (master) behavior is unstable resulting incorrect result or
> exception in API calls
> -
>
> Key: DRILL-5659
> URL: https://issues.apache.org/jira/browse/DRILL-5659
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Rob Wu
> Fix For: 1.11.0
>
> Attachments: 1-10cppClient-1.10.0Drillbit-hive.log,
> 1-10cppClient-1.10.0Drillbit-metadata and catalog test.log,
> 1-10cppClient-1.9.0Drillbit-dfs.log, 1-10cppClient-1.9.0Drillbit-hive.log,
> 1-10cppClient-1.9.0Drillbit-metadata and catalog test.log,
> 1-11cppClient-1.10.0Drillbit-hive.log, 1-11cppClient-1.10.0Drillbit-metadata
> and catalog test.log, 1-11cppClient-1.9.0Drillbit-dfs.log,
> 1-11cppClient-1.9.0Drillbit-hive.log, 1-11cppClient-1.9.0Drillbit-metadata
> and catalog test.log
>
>
> I recently compiled the C++ client (on windows) from master and tested
> against a 1.9.0 drillbit. The client's behavior does not meet the stable
> release requirement.
> Some API functionalities are broken and should be investigated.
> Most noticeable is the getColumns(...) call. It will throw an exception with
> "Cannot decode getcolumns results" when the number of rows (column records)
> exceeds a certain number.
> I also noticed that: during query execution + data retrieval, if the table is
> large enough, the result coming back will start to become NULL or empty.
> I will see if I can generate some drillclient logs to put in the attachment.
> I will also compile and test on other platforms.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5668) C++ connector crash when query error message is too long
[
https://issues.apache.org/jira/browse/DRILL-5668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084207#comment-16084207
]
Laurent Goujon commented on DRILL-5668:
---
Test query to reproduce the crash:
{noformat}
SELECT 1 FROM
[jira] [Created] (DRILL-5668) C++ connector crash when query error message is too long
Laurent Goujon created DRILL-5668:
-
Summary: C++ connector crash when query error message is too long
Key: DRILL-5668
URL: https://issues.apache.org/jira/browse/DRILL-5668
Project: Apache Drill
Issue Type: Bug
Components: Client - C++
Affects Versions: 1.10.0
Reporter: Laurent Goujon
Assignee: Laurent Goujon
Fix For: 1.11.0
The C++ connector crashes if the query sent to the server is invalid and the
error message returned by the server is too long.
The cause of the crash is a call to a {{vsprintf}} function which takes a fixed
sized char buffer, but doesn't take a size argument.
https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/errmsgs.cpp#L80
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-3640) Drill JDBC driver support Statement.setQueryTimeout(int)
[
https://issues.apache.org/jira/browse/DRILL-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057749#comment-16057749
]
Laurent Goujon commented on DRILL-3640:
---
There are two PR linked to that JIRA, should both of them be applied?
Also, according to the JDBC spec, this methods applies to the {{execute*}}
methods and might apply to the ResultSet methods, but this is not required.
Also for executeBatch, it's up to the driver to decide if timeout is per query
or total, but that should be in the javadoc I guess.
> Drill JDBC driver support Statement.setQueryTimeout(int)
>
>
> Key: DRILL-3640
> URL: https://issues.apache.org/jira/browse/DRILL-3640
> Project: Apache Drill
> Issue Type: New Feature
> Components: Client - JDBC
>Affects Versions: 1.2.0
>Reporter: Chun Chang
>Assignee: Kunal Khatua
> Fix For: 1.11.0
>
>
> It would be nice if we have this implemented. Run away queries can be
> automatically canceled by setting the timeout.
> java.sql.SQLFeatureNotSupportedException: Setting network timeout is not
> supported.
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.setQueryTimeout(DrillStatementImpl.java:152)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-4335) Apache Drill should support network encryption
[ https://issues.apache.org/jira/browse/DRILL-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983616#comment-15983616 ] Laurent Goujon commented on DRILL-4335: --- unfortunately, not extensively. I'll do it this week! > Apache Drill should support network encryption > -- > > Key: DRILL-4335 > URL: https://issues.apache.org/jira/browse/DRILL-4335 > Project: Apache Drill > Issue Type: New Feature >Reporter: Keys Botzum >Assignee: Sorabh Hamirwasia > Labels: security > Attachments: ApacheDrillEncryptionUsingSASLDesign.pdf > > > This is clearly related to Drill-291 but wanted to make explicit that this > needs to include network level encryption and not just authentication. This > is particularly important for the client connection to Drill which will often > be sending passwords in the clear until there is encryption. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5402) ServerMeta should be updated during the session, if the last has been changed.
[
https://issues.apache.org/jira/browse/DRILL-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949393#comment-15949393
]
Laurent Goujon commented on DRILL-5402:
---
Like I mentioned in the pull request for DRILL-3510, I haven't seen this
feature implemented by other JDBC drivers, so this might get some sense
regarding priority.
Another thing to consider I believe is that the servermeta object is big
(contains all the function names for example), with these functions not
changing. So from a protocol point of view, instead of forcing the user to call
again {{GetServerMeta}}, user could request just the session, and the driver
might combine both the metadata and the session to return the final value? or
maybe extend GetServerMeta api with a flag to only return some of the fields?
> ServerMeta should be updated during the session, if the last has been changed.
> --
>
> Key: DRILL-5402
> URL: https://issues.apache.org/jira/browse/DRILL-5402
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - Java, Client - ODBC, Execution - RPC, Metadata
>Affects Versions: 1.10.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Minor
> Fix For: 1.11.0
>
>
> DRILL-5301 introduces a new server metadata API.
> But clients can get from the server the current meta information only once,
> the next request returns the previous meta even if it was updated. For
> example, Identifier quote character can be changed by the user via one
> session.
> It would be good if the session is changed, the "getServerMeta()" will return
> updated server meta information.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5369) Missing initialization for ServerMetaContext
Laurent Goujon created DRILL-5369:
-
Summary: Missing initialization for ServerMetaContext
Key: DRILL-5369
URL: https://issues.apache.org/jira/browse/DRILL-5369
Project: Apache Drill
Issue Type: Bug
Components: Client - C++
Reporter: Laurent Goujon
Assignee: Laurent Goujon
{{ServerMetaContext}} is not initialized properly which might cause some
unexpected issues (like getMetadata() returning before receiving the answer) in
some cases.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5368) Memory leak in C++ server metadata handler
Laurent Goujon created DRILL-5368: - Summary: Memory leak in C++ server metadata handler Key: DRILL-5368 URL: https://issues.apache.org/jira/browse/DRILL-5368 Project: Apache Drill Issue Type: Bug Components: Client - C++ Affects Versions: 1.10.0 Reporter: Laurent Goujon Assignee: Laurent Goujon Priority: Minor When receiving server metadata response, a protobuf ServerMetaResp object is dynamically allocated but never freed. Since for this handler, there's no need to keep the instance attached to the handler (content is copied over by the MetaData class), a reference is enough and allocation can be done on the stack. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5136) Some SQL statements fail due to PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925492#comment-15925492
]
Laurent Goujon commented on DRILL-5136:
---
I might not be the best person to answer this issue unfortunately, as I don't
know in details how SQL parsing is done in Drill, if there's a way to identify
the type of queries submitted to the engine. My guess is that depending on the
type of queries, one could return a predetermined schema vs running a limit 0
query if schema has to be discovered. Maybe this could be an extension of each
SQLHandler?
> Some SQL statements fail due to PreparedStatement
> -
>
> Key: DRILL-5136
> URL: https://issues.apache.org/jira/browse/DRILL-5136
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - ODBC
>Affects Versions: 1.9.0
>Reporter: Robert Hou
>Assignee: Laurent Goujon
>
> "show schemas" does not work.
> SQL>show schemas
> 1: SQLPrepare = [MapR][Drill] (1040) Drill failed to execute the query: show
> schemas
> [30029]Query execution error. Details:[
> PARSE ERROR: Encountered "( show" at line 1, column 15.
> Was expecting one of:
> ...
> ...
> ...
> ...
> ...
> "LATERAL" ...
> "(" "WITH" ...
> "(" "+" ...
> "(" "-" ...
> "(" ...
> "(" ...
> "(" The query profile shows this SQL statement is being executed:
>{color:blue} SELECT * FROM {color} (show schemas) {color:blue} LIMIT 0
> {color}
>
> The blue text has been added when displaying schemas
> "use schema" also does not work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (DRILL-4335) Apache Drill should support network encryption
[ https://issues.apache.org/jira/browse/DRILL-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899831#comment-15899831 ] Laurent Goujon commented on DRILL-4335: --- It looks like from the initial implementation that there are lots of byte copying involved. Any estimation/benchmark to quantify the impact on throughput? wouldn't a SSL/TLS implementation be a more performant alternative here (because of its integration directly into netty?) > Apache Drill should support network encryption > -- > > Key: DRILL-4335 > URL: https://issues.apache.org/jira/browse/DRILL-4335 > Project: Apache Drill > Issue Type: New Feature >Reporter: Keys Botzum >Assignee: Sorabh Hamirwasia > Labels: security > > This is clearly related to Drill-291 but wanted to make explicit that this > needs to include network level encryption and not just authentication. This > is particularly important for the client connection to Drill which will often > be sending passwords in the clear until there is encryption. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5311) C++ connector connect doesn't check handshake result for timeout
[
https://issues.apache.org/jira/browse/DRILL-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894920#comment-15894920
]
Laurent Goujon commented on DRILL-5311:
---
Did some testing using querySubmitter:
before:
{noformat}
$ ./querySubmitter type=server connectStr=local=localhost:31010 api=meta
logLevel=trace
Connector:
name:Apache Drill C++ client
version:1.10.0
Server:
name:
version:
Metadata:
all tables are selectable: 0
catalog separator: .
catalog term: catalog
COLLATE support: 0
correlation names: 3
date time functions: CURDATE, CURTIME, NOW, QUARTER
date time literals support: 65278
GROUP BY support: 0
identifier case: 1
identifier quote string: `
max binary literal length: 0
max catalog name length: 1024
max char literal length: 0
max column name length: 1024
max columns in GROUP BY: 1024
max columns in ORDER BY: 0
max columns in SELECT: 0
max cursor name length: 1024
max logical lob size: 0
max row size: 0
max schema name length: 1024
max statement length: 0
max statements: 0
max table name length: 1024
max tables in SELECT: 0
max user name length: 1024
NULL collation: 1
numeric functions: ABS, EXP, LOG, LOG10, MOD, POWER
OUTER JOIN support: 14
quoted identifier case: 1
SQL keywords:
ABS,ALLOW,ARRAY,ASENSITIVE,ASYMMETRIC,ATOMIC,BIGINT,BINARY,BLOB,BOOLEAN,CALL,CALLED,CARDINALITY,CEIL,CEILING,CLOB,COLLECT,CONDITION,CORR,COVAR_POP,COVAR_SAMP,CUBE,CUME_DIST,CURRENT_CATALOG,CURRENT_DEFAULT_TRANSFORM_GROUP,CURRENT_PATH,CURRENT_ROLE,CURRENT_SCHEMA,CURRENT_TRANSFORM_GROUP_FOR_TYPE,CYCLE,DATABASE,DATABASES,DENSE_RANK,DEREF,DETERMINISTIC,DISALLOW,DYNAMIC,EACH,ELEMENT,EVERY,EXP,EXPLAIN,EXTEND,FILES,FILTER,FIRST_VALUE,FLOOR,FREE,FUNCTION,FUSION,GROUPING,HOLD,IF,IMPORT,INOUT,INTERSECTION,LARGE,LAST_VALUE,LATERAL,LIMIT,LN,LOCALTIME,LOCALTIMESTAMP,MEMBER,MERGE,METADATA,METHOD,MOD,MODIFIES,MULTISET,NCLOB,NEW,NONE,NORMALIZE,OFFSET,OLD,OUT,OVER,OVERLAY,PARAMETER,PARTITION,PERCENTILE_CONT,PERCENTILE_DISC,PERCENT_RANK,POWER,RANGE,RANK,READS,RECURSIVE,REF,REFERENCING,REFRESH,REGR_AVGX,REGR_AVGY,REGR_COUNT,REGR_INTERCEPT,REGR_R2,REGR_SLOPE,REGR_SXX,REGR_SXY,REGR_SYY,RELEASE,REPLACE,RESET,RESULT,RETURN,RETURNS,ROLLUP,ROW,ROW_NUMBER,SAVEPOINT,SCHEMAS,SCOPE,SEARCH,SENSITIVE,SHOW,SIMILAR,SPECIFIC,SPECIFICTYPE,SQLEXCEPTION,SQLWARNING,SQRT,START,STATIC,STDDEV_POP,STDDEV_SAMP,STREAM,SUBMULTISET,SYMMETRIC,SYSTEM,TABLES,TABLESAMPLE,TINYINT,TREAT,TRIGGER,UESCAPE,UNNEST,UPSERT,USE,VARBINARY,VAR_POP,VAR_SAMP,WIDTH_BUCKET,WINDOW,WITHIN,WITHOUT
schema term: schema
search escape string: \
special characters:
string functions:
CONCAT,INSERT,LCASE,LENGTH,LOCATE,LTRIM,RTRIM,SUBSTRING,UCASE
sub query support: 46
system functions:
table term: table
UNION support: 6
BLOB included in max row size: 1
catalog at start: 1
column aliasing supported: 1
LIKE escape clause supported: 1
NULL plus non NULL equals to NULL: 1
read-only: 0
SELECT FOR UPDATE supported: 0
transaction supported: 0
unrelated columns in ORDER BY supported: 1
{noformat}
Note that how the connection didn't fail and it is still possible to query API
(although results are not populated correctly).
after:
{noformat}
$ ./querySubmitter type=server connectStr=local=localhost:31010 api=meta
logLevel=trace
Failed to connect with error: [30015]Handshake Timeout.
(Using:local=localhost:31010)
{noformat}
> C++ connector connect doesn't check handshake result for timeout
>
>
> Key: DRILL-5311
> URL: https://issues.apache.org/jira/browse/DRILL-5311
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Reporter: Laurent Goujon
>Assignee: Sudheesh Katkam
> Labels: ready-to-commit
>
> The C++ connector connect methods returns okay as soon as the tcp connection
> is succesfully established between client and server, and the handshake
> message is sent. However it doesn't wait for handshake to have completed.
> The consequence is that if handshake failed, the error is deferred to the
> first query, which might be unexpected by the application.
> I believe that validateHanshake method in drillClientImpl should wait for the
> handshake to complete, as it seems a bit more saner...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (DRILL-5311) C++ connector connect doesn't check handshake result for timeout
[ https://issues.apache.org/jira/browse/DRILL-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-5311: -- Summary: C++ connector connect doesn't check handshake result for timeout (was: C++ connector connect doesn't wait for handshake to complete) > C++ connector connect doesn't check handshake result for timeout > > > Key: DRILL-5311 > URL: https://issues.apache.org/jira/browse/DRILL-5311 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Reporter: Laurent Goujon >Assignee: Sudheesh Katkam > Labels: ready-to-commit > > The C++ connector connect methods returns okay as soon as the tcp connection > is succesfully established between client and server, and the handshake > message is sent. However it doesn't wait for handshake to have completed. > The consequence is that if handshake failed, the error is deferred to the > first query, which might be unexpected by the application. > I believe that validateHanshake method in drillClientImpl should wait for the > handshake to complete, as it seems a bit more saner... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5311) C++ connector connect doesn't wait for handshake to complete
[ https://issues.apache.org/jira/browse/DRILL-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893725#comment-15893725 ] Laurent Goujon commented on DRILL-5311: --- It looks like there's a check but only enabled on win32 platforms: https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L277 I'm not sure why such restriction, since pError would be set whatever the platform > C++ connector connect doesn't wait for handshake to complete > > > Key: DRILL-5311 > URL: https://issues.apache.org/jira/browse/DRILL-5311 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Reporter: Laurent Goujon > > The C++ connector connect methods returns okay as soon as the tcp connection > is succesfully established between client and server, and the handshake > message is sent. However it doesn't wait for handshake to have completed. > The consequence is that if handshake failed, the error is deferred to the > first query, which might be unexpected by the application. > I believe that validateHanshake method in drillClientImpl should wait for the > handshake to complete, as it seems a bit more saner... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5313) C++ client build failure on linux
[ https://issues.apache.org/jira/browse/DRILL-5313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893707#comment-15893707 ] Laurent Goujon commented on DRILL-5313: --- Proposing an alternative fix (pr #769) which use the per-connection value instead of the default one. > C++ client build failure on linux > - > > Key: DRILL-5313 > URL: https://issues.apache.org/jira/browse/DRILL-5313 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Affects Versions: 1.10 >Reporter: Sorabh Hamirwasia >Assignee: Laurent Goujon > > We are seeing below errors while building Drill C++ client on linux platform: > [root@qa-node161 build]# make > [ 6%] Built target y2038 > [ 38%] Built target protomsgs > [ 41%] Building CXX object > src/clientlib/CMakeFiles/drillClient.dir/drillClientImpl.cpp.o > /root/drill/drill/contrib/native/client/src/clientlib/drillClientImpl.cpp: In > function ‘void Drill::updateLikeFilter(exec::user::LikeFilter&, const > std::string&)’: > /root/drill/drill/contrib/native/client/src/clientlib/drillClientImpl.cpp:782: > error: ‘s_searchEscapeString’ is not a member of ‘Drill::meta::DrillMetadata’ > make[2]: *** [src/clientlib/CMakeFiles/drillClient.dir/drillClientImpl.cpp.o] > Error 1 > make[1]: *** [src/clientlib/CMakeFiles/drillClient.dir/all] Error 2 > make: *** [all] Error 2 > It looks to be related to one of Laurent's pull request below: > https://github.com/apache/drill/pull/712 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5313) C++ client build failure on linux
[ https://issues.apache.org/jira/browse/DRILL-5313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893687#comment-15893687 ] Laurent Goujon commented on DRILL-5313: --- it's a hidden merge conflict with DRILL-5301 C++ change, I'll post a fix quickly. > C++ client build failure on linux > - > > Key: DRILL-5313 > URL: https://issues.apache.org/jira/browse/DRILL-5313 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Affects Versions: 1.10 >Reporter: Sorabh Hamirwasia >Assignee: Laurent Goujon > > We are seeing below errors while building Drill C++ client on linux platform: > [root@qa-node161 build]# make > [ 6%] Built target y2038 > [ 38%] Built target protomsgs > [ 41%] Building CXX object > src/clientlib/CMakeFiles/drillClient.dir/drillClientImpl.cpp.o > /root/drill/drill/contrib/native/client/src/clientlib/drillClientImpl.cpp: In > function ‘void Drill::updateLikeFilter(exec::user::LikeFilter&, const > std::string&)’: > /root/drill/drill/contrib/native/client/src/clientlib/drillClientImpl.cpp:782: > error: ‘s_searchEscapeString’ is not a member of ‘Drill::meta::DrillMetadata’ > make[2]: *** [src/clientlib/CMakeFiles/drillClient.dir/drillClientImpl.cpp.o] > Error 1 > make[1]: *** [src/clientlib/CMakeFiles/drillClient.dir/all] Error 2 > make: *** [all] Error 2 > It looks to be related to one of Laurent's pull request below: > https://github.com/apache/drill/pull/712 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5311) C++ connector connect doesn't wait for handshake to complete
[ https://issues.apache.org/jira/browse/DRILL-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893686#comment-15893686 ] Laurent Goujon commented on DRILL-5311: --- you're correct, I forgot that run is a blocking operation. But I guess in that case, there's no check about the actual completion of the handshake, and SUCCESS is always returned. > C++ connector connect doesn't wait for handshake to complete > > > Key: DRILL-5311 > URL: https://issues.apache.org/jira/browse/DRILL-5311 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Reporter: Laurent Goujon > > The C++ connector connect methods returns okay as soon as the tcp connection > is succesfully established between client and server, and the handshake > message is sent. However it doesn't wait for handshake to have completed. > The consequence is that if handshake failed, the error is deferred to the > first query, which might be unexpected by the application. > I believe that validateHanshake method in drillClientImpl should wait for the > handshake to complete, as it seems a bit more saner... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5311) C++ connector connect doesn't wait for handshake to complete
Laurent Goujon created DRILL-5311: - Summary: C++ connector connect doesn't wait for handshake to complete Key: DRILL-5311 URL: https://issues.apache.org/jira/browse/DRILL-5311 Project: Apache Drill Issue Type: Bug Components: Client - C++ Reporter: Laurent Goujon The C++ connector connect methods returns okay as soon as the tcp connection is succesfully established between client and server, and the handshake message is sent. However it doesn't wait for handshake to have completed. The consequence is that if handshake failed, the error is deferred to the first query, which might be unexpected by the application. I believe that validateHanshake method in drillClientImpl should wait for the handshake to complete, as it seems a bit more saner... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5303) Drillbits fail to start when Drill server built with JDK 8 is deployed on a JDK 7 environment
[
https://issues.apache.org/jira/browse/DRILL-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890875#comment-15890875
]
Laurent Goujon commented on DRILL-5303:
---
Java does not provide such guarantee as compiling against version N+1 jdk (with
source and target class using version N) and being able to run on version N:
you need the three of them (bootclasspath, source and target) to match
versions. And this kind of issues happened with all Java releases.
Coming with Java9 I believe, symbols for previous jdks will also be available
meaning you would not need java8 (just for the boot classpath) in order to
target jdk8.
> Drillbits fail to start when Drill server built with JDK 8 is deployed on a
> JDK 7 environment
> -
>
> Key: DRILL-5303
> URL: https://issues.apache.org/jira/browse/DRILL-5303
> Project: Apache Drill
> Issue Type: Bug
> Components: Server, Tools, Build & Test
>Affects Versions: 1.9.0
>Reporter: Abhishek Girish
>
> When Drill is built on a node configured with JDK 8 and is then deployed in a
> JDK 7 environment, Drillbits fail to start and the following errors are seen
> in Drillbit.out:
> {code}
> Exception in thread "main" java.lang.NoSuchMethodError:
> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
> at
> org.apache.drill.exec.coord.ClusterCoordinator.drillbitRegistered(ClusterCoordinator.java:85)
> at
> org.apache.drill.exec.coord.zk.ZKClusterCoordinator.updateEndpoints(ZKClusterCoordinator.java:266)
> at
> org.apache.drill.exec.coord.zk.ZKClusterCoordinator.start(ZKClusterCoordinator.java:135)
> at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:117)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:292)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:272)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:268)
> {code}
> Workaround is to match the Java versions of build and deployment environments.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (DRILL-4994) Prepared statement stopped working between 1.8.0 client and < 1.8.0 server
[ https://issues.apache.org/jira/browse/DRILL-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890744#comment-15890744 ] Laurent Goujon commented on DRILL-4994: --- Adding ready-to-commit label as it was also added to DRILL-4730 which is part of the same pull request > Prepared statement stopped working between 1.8.0 client and < 1.8.0 server > -- > > Key: DRILL-4994 > URL: https://issues.apache.org/jira/browse/DRILL-4994 > Project: Apache Drill > Issue Type: Bug >Reporter: Laurent Goujon >Assignee: Laurent Goujon > Labels: ready-to-commit > > Older servers (pre-1.8.0) don't support the prepared statement rpc method, > but the JDBC client doesn't check if it is available or not. The end result > is that the statement is stuck as the server is not responding back. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (DRILL-4994) Prepared statement stopped working between 1.8.0 client and < 1.8.0 server
[ https://issues.apache.org/jira/browse/DRILL-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-4994: -- Labels: ready-to-commit (was: ) > Prepared statement stopped working between 1.8.0 client and < 1.8.0 server > -- > > Key: DRILL-4994 > URL: https://issues.apache.org/jira/browse/DRILL-4994 > Project: Apache Drill > Issue Type: Bug >Reporter: Laurent Goujon >Assignee: Laurent Goujon > Labels: ready-to-commit > > Older servers (pre-1.8.0) don't support the prepared statement rpc method, > but the JDBC client doesn't check if it is available or not. The end result > is that the statement is stuck as the server is not responding back. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-3161) Drill JDBC driver not visible/auto-registered via Service Provider Mechanism
[
https://issues.apache.org/jira/browse/DRILL-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laurent Goujon closed DRILL-3161.
-
Resolution: Duplicate
> Drill JDBC driver not visible/auto-registered via Service Provider Mechanism
>
>
> Key: DRILL-3161
> URL: https://issues.apache.org/jira/browse/DRILL-3161
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Reporter: Daniel Barclay
> Fix For: Future
>
>
> Drill's JDBC driver is not automatically made visible to JDBC's DriverManager
> and auto-registered, because it does not use Java's Service Provider
> Mechanism as specified by JDBC 4.0.
> This usually means that instead of just having to put the Drill JDBC driver
> Jar file on the class path and use a Drill JDBC URL (one starting with
> "{{jdbc:drill:}}"), users also have to configure their tools or code with the
> name of the Drill driver class.
>
> The Drill JDBC driver's Jar file should contain a
> {{META-INF/services/java.sql.Driver}} file that contains a line consisting of
> the fully qualified name of the Drill JDBC driver class
> ({{org.apache.drill.jdbc.Driver}}).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (DRILL-4994) Prepared statement stopped working between 1.8.0 client and < 1.8.0 server
[ https://issues.apache.org/jira/browse/DRILL-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886204#comment-15886204 ] Laurent Goujon commented on DRILL-4994: --- Part of this pull request: https://github.com/apache/drill/pull/613 > Prepared statement stopped working between 1.8.0 client and < 1.8.0 server > -- > > Key: DRILL-4994 > URL: https://issues.apache.org/jira/browse/DRILL-4994 > Project: Apache Drill > Issue Type: Bug >Reporter: Laurent Goujon >Assignee: Laurent Goujon > > Older servers (pre-1.8.0) don't support the prepared statement rpc method, > but the JDBC client doesn't check if it is available or not. The end result > is that the statement is stuck as the server is not responding back. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (DRILL-4994) Prepared statement stopped working between 1.8.0 client and < 1.8.0 server
[ https://issues.apache.org/jira/browse/DRILL-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon reassigned DRILL-4994: - Assignee: Laurent Goujon > Prepared statement stopped working between 1.8.0 client and < 1.8.0 server > -- > > Key: DRILL-4994 > URL: https://issues.apache.org/jira/browse/DRILL-4994 > Project: Apache Drill > Issue Type: Bug >Reporter: Laurent Goujon >Assignee: Laurent Goujon > > Older servers (pre-1.8.0) don't support the prepared statement rpc method, > but the JDBC client doesn't check if it is available or not. The end result > is that the statement is stuck as the server is not responding back. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (DRILL-5301) Add server metadata API
[ https://issues.apache.org/jira/browse/DRILL-5301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon reassigned DRILL-5301: - Assignee: Laurent Goujon > Add server metadata API > --- > > Key: DRILL-5301 > URL: https://issues.apache.org/jira/browse/DRILL-5301 > Project: Apache Drill > Issue Type: Improvement > Components: Server, Client - C++, Client - Java, Client - JDBC, > Client - ODBC >Reporter: Laurent Goujon >Assignee: Laurent Goujon > > JDBC and ODBC clients exposes lots of metadata regarding server version and > support of various parts of the SQL standard. > Currently the returned information is hardcoded in both clients/drivers which > means that the infomation returned is support as of the client version, not > the server version. > Instead, a new method should be provided to the clients to query the actual > server support. Support on the client or the server should be optional (for > example a client should not use this API if the server doesn't support it and > fallback to default values). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5301) Add server metadata API
Laurent Goujon created DRILL-5301: - Summary: Add server metadata API Key: DRILL-5301 URL: https://issues.apache.org/jira/browse/DRILL-5301 Project: Apache Drill Issue Type: Improvement Components: Server, Client - C++, Client - Java, Client - JDBC, Client - ODBC Reporter: Laurent Goujon JDBC and ODBC clients exposes lots of metadata regarding server version and support of various parts of the SQL standard. Currently the returned information is hardcoded in both clients/drivers which means that the infomation returned is support as of the client version, not the server version. Instead, a new method should be provided to the clients to query the actual server support. Support on the client or the server should be optional (for example a client should not use this API if the server doesn't support it and fallback to default values). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-4385) Support metadata and prepare operations on User RPC layer
[ https://issues.apache.org/jira/browse/DRILL-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon closed DRILL-4385. - Resolution: Duplicate Assignee: Venki Korukanti > Support metadata and prepare operations on User RPC layer > - > > Key: DRILL-4385 > URL: https://issues.apache.org/jira/browse/DRILL-4385 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Venki Korukanti > > Right now, we don't support prepare and metadata operations are done through > code and queries on the jdbc and odbc drivers. This is an umbrella task to > implement metadata and prepare operations directly at the RPC layer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-4419) JDBC driver should move to using the new metadata methods provided by DRILL-4385
[ https://issues.apache.org/jira/browse/DRILL-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon closed DRILL-4419. - Resolution: Fixed Assignee: Laurent Goujon > JDBC driver should move to using the new metadata methods provided by > DRILL-4385 > > > Key: DRILL-4419 > URL: https://issues.apache.org/jira/browse/DRILL-4419 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Jacques Nadeau >Assignee: Laurent Goujon > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-4385) Support metadata and prepare operations on User RPC layer
[ https://issues.apache.org/jira/browse/DRILL-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886173#comment-15886173 ] Laurent Goujon commented on DRILL-4385: --- I believe this was done as part of DRILL-4728 and DRILL-4729 > Support metadata and prepare operations on User RPC layer > - > > Key: DRILL-4385 > URL: https://issues.apache.org/jira/browse/DRILL-4385 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau > > Right now, we don't support prepare and metadata operations are done through > code and queries on the jdbc and odbc drivers. This is an umbrella task to > implement metadata and prepare operations directly at the RPC layer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-4280) Kerberos Authentication
[ https://issues.apache.org/jira/browse/DRILL-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870738#comment-15870738 ] Laurent Goujon commented on DRILL-4280: --- I'm still reviewing the code. I thought I'd have it done sooner, but unfortunately, I was now able to give it my full attention until now. Let me know if there's hard deadlines for this. > Kerberos Authentication > --- > > Key: DRILL-4280 > URL: https://issues.apache.org/jira/browse/DRILL-4280 > Project: Apache Drill > Issue Type: Improvement >Reporter: Keys Botzum >Assignee: Sudheesh Katkam > Labels: security > > Drill should support Kerberos based authentication from clients. This means > that both the ODBC and JDBC drivers as well as the web/REST interfaces should > support inbound Kerberos. For Web this would most likely be SPNEGO while for > ODBC and JDBC this will be more generic Kerberos. > Since Hive and much of Hadoop supports Kerberos there is a potential for a > lot of reuse of ideas if not implementation. > Note that this is related to but not the same as > https://issues.apache.org/jira/browse/DRILL-3584 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5219) Remove DrillUserProperties filtering in C++ driver
[ https://issues.apache.org/jira/browse/DRILL-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857491#comment-15857491 ] Laurent Goujon commented on DRILL-5219: --- Thanks [~sudheeshkatkam] > Remove DrillUserProperties filtering in C++ driver > -- > > Key: DRILL-5219 > URL: https://issues.apache.org/jira/browse/DRILL-5219 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Reporter: Laurent Goujon >Assignee: Laurent Goujon >Priority: Minor > Labels: ready-to-commit > > Unlike the Java client, the C++ connector filter out unknown Drill user > properties: > https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L374 > This prevents a client (like the ODBC driver) to pass extra properties to the > server (like extra metainformation, or some specific behavior for a given > software) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5219) Remove DrillUserProperties filtering in C++ driver
[ https://issues.apache.org/jira/browse/DRILL-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852976#comment-15852976 ] Laurent Goujon commented on DRILL-5219: --- The idea is that an older client can provide information for a newer server without protocol change like the default quoting mechanism, or some special behavior an application might require. I consider that user properties are similar to HTTP header fields where some are part of the specs, and that client and server should abide by them, but clients can send any header, with the assumption that the server might not recognize it and will ignore it. Having user properties being filtered by the client means that if the end user wants to provide extra context to the server, it needs to update to a new client, which is sometimes an issue in some organizations. Also the ODBC driver release process is a long and complex one, so I believe that there is value of keeping a way to pass informations around without having to change the client. > Remove DrillUserProperties filtering in C++ driver > -- > > Key: DRILL-5219 > URL: https://issues.apache.org/jira/browse/DRILL-5219 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Reporter: Laurent Goujon >Assignee: Laurent Goujon >Priority: Minor > > Unlike the Java client, the C++ connector filter out unknown Drill user > properties: > https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L374 > This prevents a client (like the ODBC driver) to pass extra properties to the > server (like extra metainformation, or some specific behavior for a given > software) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (DRILL-5221) cancel message is delayed until queryid or data is received
Title: Message Title Laurent Goujon assigned an issue to Laurent Goujon Apache Drill / DRILL-5221 cancel message is delayed until queryid or data is received Change By: Laurent Goujon Assignee: Laurent Goujon Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] [Commented] (DRILL-5222) C++ client unable to parse queries with table function
[
https://issues.apache.org/jira/browse/DRILL-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838447#comment-15838447
]
Laurent Goujon commented on DRILL-5222:
---
can you describe the kind of connection (direct connection vs zookeeper), and
if you use any session statement (like USE)?
> C++ client unable to parse queries with table function
> --
>
> Key: DRILL-5222
> URL: https://issues.apache.org/jira/browse/DRILL-5222
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Krystal
>
> The following query failed from was odbc and custom C++ client app:
> SQL>select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
> lineDelimiter=>'\r\n'))
> 1: SQLPrepare = [MapR][Drill] (1040) Drill failed to execute the query:
> select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
> lineDelimiter=>'\r\n'))
> [30027]Query execution error. Details:[
> SYSTEM ERROR: SqlValidatorException: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> Here is the stack trace:
> {code}
>SYSTEM ERROR: SqlValidatorException: No match found for function
> signature table_function/cr_lf.csv(type => , lineDelimiter => )
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> org.apache.drill.exec.work.foreman.Foreman.run():281
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.drill.exec.exception.FunctionNotFoundException) No
> match found for function signature table_function/cr_lf.csv(type => ,
> lineDelimiter => )
> org.apache.drill.exec.planner.sql.SqlConverter.validate():170
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode():606
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():192
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():164
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():122
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():96
> org.apache.drill.exec.work.foreman.Foreman.runSQL():1017
> org.apache.drill.exec.work.foreman.Foreman.run():264
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.calcite.runtime.CalciteContextException) From line 1,
> column 45 to line 1, column 107: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2
> sun.reflect.NativeConstructorAccessorImpl.newInstance():57
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
> java.lang.reflect.Constructor.newInstance():526
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex():405
> org.apache.calcite.sql.SqlUtil.newContextException():765
> org.apache.calcite.sql.SqlUtil.newContextException():753
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError():3974
>
> org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction():1583
> org.apache.calcite.sql.SqlFunction.deriveType():278
> org.apache.calcite.sql.SqlFunction.deriveType():222
>
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4337
>
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4324
> org.apache.calcite.sql.SqlCall.accept():130
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl():1501
> org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl():53
> org.apache.calcite.sql.validate.AbstractNamespace.validate():86
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2806
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2791
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3014
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
> org.apache.calcite.sql.validate.AbstractNamespace.validate():86
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2806
>
[jira] [Commented] (DRILL-5219) Remove DrillUserProperties filtering in C++ driver
[
https://issues.apache.org/jira/browse/DRILL-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838371#comment-15838371
]
Laurent Goujon commented on DRILL-5219:
---
I used the wrong jira id in my commit description
{quote}
GitHub user laurentgo opened a pull request:
https://github.com/apache/drill/pull/727
DRILL-5119: Relax user properties validation in C++ client
Unlike Java client, C++ client only allows user properties present in a
whitelist. Relax this restriction so that user can add extra properties.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/laurentgo/drill laurent/DRILL-5219
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/727.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #727
commit 6cae1ae18c3fd876ba98130971216dbf5cb10324
Author: Laurent Goujon
Date: 2017-01-25T18:32:33Z
DRILL-5119: Relax user properties validation in C++ client
Unlike Java client, C++ client only allows user properties present in a
whitelist. Relax this restriction so that user can add extra properties.
{quote}
> Remove DrillUserProperties filtering in C++ driver
> --
>
> Key: DRILL-5219
> URL: https://issues.apache.org/jira/browse/DRILL-5219
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
>Priority: Minor
>
> Unlike the Java client, the C++ connector filter out unknown Drill user
> properties:
> https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L374
> This prevents a client (like the ODBC driver) to pass extra properties to the
> server (like extra metainformation, or some specific behavior for a given
> software)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5119) Update MapR version to 5.2.0.40963-mapr
[ https://issues.apache.org/jira/browse/DRILL-5119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838365#comment-15838365 ] Laurent Goujon commented on DRILL-5119: --- Sorry, I used the wrong jira id for my pull request. Please ignore latest github comment > Update MapR version to 5.2.0.40963-mapr > --- > > Key: DRILL-5119 > URL: https://issues.apache.org/jira/browse/DRILL-5119 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.10.0 >Reporter: Abhishek Girish >Assignee: Patrick Wong > Labels: ready-to-commit > Fix For: 1.10.0 > > > This is for the "mapr" profile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-5220) Add api to set application name in C++ connector
[ https://issues.apache.org/jira/browse/DRILL-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon reassigned DRILL-5220: - Assignee: Laurent Goujon > Add api to set application name in C++ connector > > > Key: DRILL-5220 > URL: https://issues.apache.org/jira/browse/DRILL-5220 > Project: Apache Drill > Issue Type: Improvement > Components: Client - C++ >Affects Versions: 1.8.0 >Reporter: Laurent Goujon >Assignee: Laurent Goujon >Priority: Minor > > There's no API for a C++ connector user to specify the name of the > application, and to provide it to the server (optional field added in > DRILL-4369) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-5219) Remove DrillUserProperties filtering in C++ driver
[ https://issues.apache.org/jira/browse/DRILL-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon reassigned DRILL-5219: - Assignee: Laurent Goujon > Remove DrillUserProperties filtering in C++ driver > -- > > Key: DRILL-5219 > URL: https://issues.apache.org/jira/browse/DRILL-5219 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Reporter: Laurent Goujon >Assignee: Laurent Goujon >Priority: Minor > > Unlike the Java client, the C++ connector filter out unknown Drill user > properties: > https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L374 > This prevents a client (like the ODBC driver) to pass extra properties to the > server (like extra metainformation, or some specific behavior for a given > software) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-5167) C++ connector does not set escape string for metadata search pattern
[ https://issues.apache.org/jira/browse/DRILL-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-5167: -- Affects Version/s: 1.9.0 > C++ connector does not set escape string for metadata search pattern > > > Key: DRILL-5167 > URL: https://issues.apache.org/jira/browse/DRILL-5167 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0 >Reporter: Laurent Goujon >Assignee: Laurent Goujon >Priority: Minor > > C++ connector does not set the escape string for search pattern when doing > metadata operation (getCatalogs/getSchema/getTables/getColumns). It is > assumed to be '\\' as returned by DrillMetadata::getSearchEscapeString(), but > because this is not sent over the wire, the server will actually consider > that there's no escape character, and might return different or no result > compared to what has been requested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5222) C++ client unable to parse queries with table function
[
https://issues.apache.org/jira/browse/DRILL-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838279#comment-15838279
]
Laurent Goujon commented on DRILL-5222:
---
As the C++ client provides the query as is to the server, it is more likely a
server issue than a client one. Have you tried the same query with the JDBC
client or directly into the UI, or from CLI?
> C++ client unable to parse queries with table function
> --
>
> Key: DRILL-5222
> URL: https://issues.apache.org/jira/browse/DRILL-5222
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Krystal
>
> The following query failed from was odbc and custom C++ client app:
> SQL>select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
> lineDelimiter=>'\r\n'))
> 1: SQLPrepare = [MapR][Drill] (1040) Drill failed to execute the query:
> select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
> lineDelimiter=>'\r\n'))
> [30027]Query execution error. Details:[
> SYSTEM ERROR: SqlValidatorException: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> Here is the stack trace:
> {code}
>SYSTEM ERROR: SqlValidatorException: No match found for function
> signature table_function/cr_lf.csv(type => , lineDelimiter => )
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> org.apache.drill.exec.work.foreman.Foreman.run():281
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.drill.exec.exception.FunctionNotFoundException) No
> match found for function signature table_function/cr_lf.csv(type => ,
> lineDelimiter => )
> org.apache.drill.exec.planner.sql.SqlConverter.validate():170
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode():606
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():192
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():164
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():122
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():96
> org.apache.drill.exec.work.foreman.Foreman.runSQL():1017
> org.apache.drill.exec.work.foreman.Foreman.run():264
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.calcite.runtime.CalciteContextException) From line 1,
> column 45 to line 1, column 107: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2
> sun.reflect.NativeConstructorAccessorImpl.newInstance():57
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
> java.lang.reflect.Constructor.newInstance():526
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex():405
> org.apache.calcite.sql.SqlUtil.newContextException():765
> org.apache.calcite.sql.SqlUtil.newContextException():753
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError():3974
>
> org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction():1583
> org.apache.calcite.sql.SqlFunction.deriveType():278
> org.apache.calcite.sql.SqlFunction.deriveType():222
>
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4337
>
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4324
> org.apache.calcite.sql.SqlCall.accept():130
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl():1501
> org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl():53
> org.apache.calcite.sql.validate.AbstractNamespace.validate():86
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2806
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2791
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3014
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
> org.apache.calcite.sql.validate.AbstractNamespace.validate():86
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
>
[jira] [Commented] (DRILL-5221) cancel message is delayed until queryid or data is received
[ https://issues.apache.org/jira/browse/DRILL-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837076#comment-15837076 ] Laurent Goujon commented on DRILL-5221: --- To clarify the description a bit, the current connector is waiting for the next message (queryId or batch data) to be received to send a cancellation message to the server. What I am proposing is to send immediately the cancel message to the server if the queryId has already been received. Otherwise to wait until the queryId has been received. > cancel message is delayed until queryid or data is received > --- > > Key: DRILL-5221 > URL: https://issues.apache.org/jira/browse/DRILL-5221 > Project: Apache Drill > Issue Type: Improvement > Components: Client - C++ >Affects Versions: 1.9.0 >Reporter: Laurent Goujon > > When user is calling the cancel method of the C++ client, the client wait for > a message from the server to reply back with a cancellation message. > In case of queries taking a long time to return batch results, it means > cancellation won't be effective until the next batch is received, instead of > cancelling right away the query (assuming the query id has already been > received, which is generally the case). > It seems this was foreseen by [~vkorukanti] in his initial patch > (https://github.com/vkorukanti/drill/commit/e0ef6349aac48de5828b6d725c2cf013905d18eb) > but was omitted when I backported it post metadata changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-5221) cancel message is delayed until queryid or data is received
[ https://issues.apache.org/jira/browse/DRILL-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laurent Goujon updated DRILL-5221: -- Description: When user is calling the cancel method of the C++ client, the client wait for a message from the server to reply back with a cancellation message. In case of queries taking a long time to return batch results, it means cancellation won't be effective until the next batch is received, instead of cancelling right away the query (assuming the query id has already been received, which is generally the case). It seems this was foreseen by [~vkorukanti] in his initial patch (https://github.com/vkorukanti/drill/commit/e0ef6349aac48de5828b6d725c2cf013905d18eb) but was omitted when I backported it post metadata changes. was: When user is calling the cancel method of the C++ client, the client wait for a message from the server to reply back with a cancellation message. In case of queries taking a long time to return their first batch, it means cancellation is taking the same amount of time to be effective, instead of cancelling right away the query (assuming the query id has already been received, which is generally the case). It seems this was foreseen by [~vkorukanti] in his initial patch (https://github.com/vkorukanti/drill/commit/e0ef6349aac48de5828b6d725c2cf013905d18eb) but was omitted when I backported it post metadata changes. > cancel message is delayed until queryid or data is received > --- > > Key: DRILL-5221 > URL: https://issues.apache.org/jira/browse/DRILL-5221 > Project: Apache Drill > Issue Type: Improvement > Components: Client - C++ >Affects Versions: 1.9.0 >Reporter: Laurent Goujon > > When user is calling the cancel method of the C++ client, the client wait for > a message from the server to reply back with a cancellation message. > In case of queries taking a long time to return batch results, it means > cancellation won't be effective until the next batch is received, instead of > cancelling right away the query (assuming the query id has already been > received, which is generally the case). > It seems this was foreseen by [~vkorukanti] in his initial patch > (https://github.com/vkorukanti/drill/commit/e0ef6349aac48de5828b6d725c2cf013905d18eb) > but was omitted when I backported it post metadata changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5221) cancel message is delayed until queryid or data is received
Laurent Goujon created DRILL-5221: - Summary: cancel message is delayed until queryid or data is received Key: DRILL-5221 URL: https://issues.apache.org/jira/browse/DRILL-5221 Project: Apache Drill Issue Type: Improvement Components: Client - C++ Affects Versions: 1.9.0 Reporter: Laurent Goujon When user is calling the cancel method of the C++ client, the client wait for a message from the server to reply back with a cancellation message. In case of queries taking a long time to return their first batch, it means cancellation is taking the same amount of time to be effective, instead of cancelling right away the query (assuming the query id has already been received, which is generally the case). It seems this was foreseen by [~vkorukanti] in his initial patch (https://github.com/vkorukanti/drill/commit/e0ef6349aac48de5828b6d725c2cf013905d18eb) but was omitted when I backported it post metadata changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5217) Heartbeat Fails when C++ client receives a large ResultSet
[ https://issues.apache.org/jira/browse/DRILL-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836980#comment-15836980 ] Laurent Goujon commented on DRILL-5217: --- maybe then having two separate services (one for sending, one for receiving) might make sense then. There are also some common locks you would have to figure out how to split... > Heartbeat Fails when C++ client receives a large ResultSet > -- > > Key: DRILL-5217 > URL: https://issues.apache.org/jira/browse/DRILL-5217 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Reporter: Sudheesh Katkam >Priority: Critical > > If the listener thread is occupied for longer than 15 seconds (heartbeat > timeout) while [handling a message from the > drillbit|https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L1286] > e.g. [processing query data blocks if the query result listener's buffer is > full|https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L899], > heartbeats fail because the same thread is responsible for sending > heartbeats! > Fix is to [handle long running > operations|http://stackoverflow.com/questions/17648725/long-running-blocking-operations-in-boost-asio-handlers] > separately using boost asio. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5217) Heartbeat Fails when C++ client receives a large ResultSet
[ https://issues.apache.org/jira/browse/DRILL-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836950#comment-15836950 ] Laurent Goujon commented on DRILL-5217: --- maybe I misunderstood what the issue is: I thought the client was closing the connection because it was not receiving the pong notification in a timely manner. > Heartbeat Fails when C++ client receives a large ResultSet > -- > > Key: DRILL-5217 > URL: https://issues.apache.org/jira/browse/DRILL-5217 > Project: Apache Drill > Issue Type: Bug > Components: Client - C++ >Reporter: Sudheesh Katkam >Priority: Critical > > If the listener thread is occupied for longer than 15 seconds (heartbeat > timeout) while [handling a message from the > drillbit|https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L1286] > e.g. [processing query data blocks if the query result listener's buffer is > full|https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L899], > heartbeats fail because the same thread is responsible for sending > heartbeats! > Fix is to [handle long running > operations|http://stackoverflow.com/questions/17648725/long-running-blocking-operations-in-boost-asio-handlers] > separately using boost asio. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
