[jira] [Commented] (DRILL-8009) DrillConnectionImpl#isValid() doesn't correspond JDBC API
[
https://issues.apache.org/jira/browse/DRILL-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447701#comment-17447701
]
ASF GitHub Bot commented on DRILL-8009:
---
paul-rogers commented on pull request #2333:
URL: https://github.com/apache/drill/pull/2333#issuecomment-976017727
@rymarm & @luocooong sorry, I've been off on other stuff. Before I did into
the code again, I'd like to check one thing. Suppose I build the new JDBC
driver with this fix. Suppose I run it with a previously-released Drill that
does not have the server-side fixes? Will it work? Or, does this JDBC driver
require a matching server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DrillConnectionImpl#isValid() doesn't correspond JDBC API
> -
>
> Key: DRILL-8009
> URL: https://issues.apache.org/jira/browse/DRILL-8009
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
>
> {{DrillConnectionImpl#isValid()}} doesn't correspond [Java JDBC API
> documentation|https://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#isValid(int)].
> Current implementation doesn't do actual connection verify and doesn't cover
> cases like drillbit termination and network issues.
> {{java.sql.Connection#isValid()}} (which extends {{DrillConnectionImpl}})
> widely used in JDBC connection pools like HikariCP and without right
> implementation, Drill JDBC driver cannot be used with them, because
> connections will keep alive forever.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-7983) Add a REST API to support the get running or completed profiles
[
https://issues.apache.org/jira/browse/DRILL-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447698#comment-17447698
]
ASF GitHub Bot commented on DRILL-7983:
---
paul-rogers commented on pull request #2353:
URL: https://github.com/apache/drill/pull/2353#issuecomment-976015866
@CuteKittyhoho, thanks for this PR. I wonder, can we simplify the API a bit.
As you stated your change is to change the original `/profiles.json` to:
```text
/profiles.json
/profiles/running
/profiles/completed
```
One could imagine other filters as well: failed requests, completed
requests, running requests by Charles, etc.
So, I wonder if it makes sense to add the filter as a query option:
`/profiles/json?status=[all|running|completed]&...`
Where the `...` is whatever someone else wants to add later.
Might make the code a bit simpler also.
Then, on the unit tests: testing this one might be hard: there is no good
way to ensure that there are, in fact, running queries when you request them.
If it were me, I'd add a hack to the mock data source. Add a global lock. By
default, it is unlocked so existing code works. A test can set the lock, then
launch a query in one thread, while testing your API in another. The mock data
source, when it fetches a batch, would block if the lock is set, waiting for
the lock to be cleared. Ugly, but it should work.
Now, I'm not sure it's fair for us to ask you to add all that for just this
case, so I guess I'm OK with doing only manual ad-hoc tests for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add a REST API to support the get running or completed profiles
> ---
>
> Key: DRILL-7983
> URL: https://issues.apache.org/jira/browse/DRILL-7983
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Cong Luo
>Priority: Major
> Fix For: 1.20.0
>
>
> Drill have a REST API to get the profiles of running and completed queries.
> this goal of ticket is extended from the `/profiles.json` to support get the
> running or completed profiles (not all of them). the client does not need to
> filter with the state.
> old URI
> {code:java}
> /profiles.json{code}
> new URI
> {code:java}
> /profiles.json
> /profiles/running
> /profiles/completed{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-8037) Add V2 JSON Format Plugin based on EVF
[ https://issues.apache.org/jira/browse/DRILL-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447694#comment-17447694 ] ASF GitHub Bot commented on DRILL-8037: --- paul-rogers commented on pull request #2364: URL: https://github.com/apache/drill/pull/2364#issuecomment-976012501 @vdiravka, thanks for doing this! It will take me a bit to remember where I left this. I'll do the review in small bits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add V2 JSON Format Plugin based on EVF > -- > > Key: DRILL-8037 > URL: https://issues.apache.org/jira/browse/DRILL-8037 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Vitalii Diravka >Assignee: Vitalii Diravka >Priority: Major > > This adds new V2 beta JSON Format Plugin based on the "Extended Vector > Framework". > This is follow up DRILL-6953 (was closed with the decision to merge it by > small pieces). > So it is based on [https://github.com/apache/drill/pull/1913] and > [https://github.com/paul-rogers/drill/tree/DRILL-6953-rev2] work. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (DRILL-7844) Some GitHub Actions builds intermittently fail
[
https://issues.apache.org/jira/browse/DRILL-7844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka resolved DRILL-7844.
Resolution: Done
The problem is no longer observed after DRILL-7973
> Some GitHub Actions builds intermittently fail
> --
>
> Key: DRILL-7844
> URL: https://issues.apache.org/jira/browse/DRILL-7844
> Project: Apache Drill
> Issue Type: Test
> Components: Tools, Build & Test
>Affects Versions: 1.18.0
>Reporter: Vitalii Diravka
>Priority: Minor
> Labels: github, github-actions
> Fix For: Future
>
>
>
> *[Very minor, but can be good for newcomers]*
> GitHub Action intermitently fails due to different issues:
> * on JVM 1.8 with:
> {code:java}
> [INFO]
> 2955[INFO] Results:
> 2956[INFO]
> 2957Error: Errors:
> 2958Error:StatementTest.testClientTriggeredQueryTimeout:152 » SqlTimeout
> Query timed out...
> 2959[INFO]
> 2960Error: Tests run: 1812, Failures: 0, Errors: 1, Skipped: 370
> 2961[INFO]
> 2962[INFO]
>
> 2963[INFO] Reactor Summary for Apache Drill Root POM 1.19.0-SNAPSHOT:
> 2964[INFO]
> 2965[INFO] Apache Drill Root POM .. SUCCESS [
> 12.859 s]
> 2966[INFO] tools/Parent Pom ... SUCCESS [
> 0.410 s]
> 2967[INFO] tools/freemarker codegen tooling ... SUCCESS [
> 6.684 s]
> 2968[INFO] Drill Protocol . SUCCESS [
> 7.467 s]
> 2969[INFO] Common (Logical Plan, Base expressions) SUCCESS [
> 9.096 s]
> 2970[INFO] Logical Plan, Base expressions . SUCCESS [
> 9.730 s]
> 2971[INFO] exec/Parent Pom SUCCESS [
> 0.307 s]
> 2972[INFO] exec/memory/Parent Pom . SUCCESS [
> 0.296 s]
> 2973[INFO] exec/memory/base ... SUCCESS [
> 6.691 s]
> 2974[INFO] exec/rpc ... SUCCESS [
> 3.329 s]
> 2975[INFO] exec/Vectors ... SUCCESS
> [01:30 min]
> 2976[INFO] contrib/Parent Pom . SUCCESS [
> 0.260 s]
> 2977[INFO] contrib/data/Parent Pom SUCCESS [
> 0.283 s]
> 2978[INFO] contrib/data/tpch-sample-data .. SUCCESS [
> 2.177 s]
> 2979[INFO] metastore/Parent Pom ... SUCCESS [
> 0.275 s]
> 2980[INFO] metastore/Drill Metastore API .. SUCCESS [
> 9.111 s]
> 2981[INFO] metastore/Drill Iceberg Metastore .. SUCCESS [
> 21.273 s]
> 2982[INFO] exec/Java Execution Engine . SUCCESS
> [44:14 min]
> 2983[INFO] exec/JDBC Driver using dependencies FAILURE
> [02:11 min]
> 2984[INFO] JDBC JAR with all dependencies . SKIPPED
> 2985[INFO] Drill-on-YARN .. SKIPPED
> 2986[INFO] metastore/Drill RDBMS Metastore SKIPPED
> 2987[INFO] contrib/kudu-storage-plugin SKIPPED
> 2988[INFO] contrib/format-xml . SKIPPED
> 2989[INFO] contrib/http-storage-plugin SKIPPED
> 2990[INFO] contrib/opentsdb-storage-plugin SKIPPED
> 2991[INFO] contrib/mongo-storage-plugin ... SKIPPED
> 2992[INFO] contrib/hbase-storage-plugin ... SKIPPED
> 2993[INFO] contrib/jdbc-storage-plugin SKIPPED
> 2994[INFO] contrib/hive-storage-plugin/Parent Pom . SKIPPED
> 2995[INFO] contrib/hive-storage-plugin/hive-exec-shaded ... SKIPPED
> 2996[INFO] contrib/hive-storage-plugin/core ... SKIPPED
> 2997[INFO] contrib/kafka-storage-plugin ... SKIPPED
> 2998[INFO] contrib/drill-udfs . SKIPPED
> 2999[INFO] contrib/format-syslog .. SKIPPED
> 3000[INFO] contrib/httpd-format-plugin SKIPPED
> 3001[INFO] contrib/format-hdf5 SKIPPED
> 3002[INFO] contrib/format-spss SKIPPED
> 3003[INFO] contrib/ltsv-format-plugin . SKIPPED
> 3004[INFO] contrib/format-esri SKIPPED
> 3005[INFO] contrib/format-excel ... SKIPPED
> 3006[INFO] contrib/druid-storage-plugin ... SKIPPED
> 3007[INFO] Packaging and Distribution Assembly SKIPPED
> 3008[INFO] contrib/mapr-format-plugin . SKIPPED
> 3009[INFO]
>
> 3010[INFO
[jira] [Resolved] (DRILL-3192) TestDrillbitResilience#cancelWhenQueryIdArrives hangs
[ https://issues.apache.org/jira/browse/DRILL-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka resolved DRILL-3192. Resolution: Done Not an issue after DRILL-7973 and DRILL-8030 > TestDrillbitResilience#cancelWhenQueryIdArrives hangs > - > > Key: DRILL-3192 > URL: https://issues.apache.org/jira/browse/DRILL-3192 > Project: Apache Drill > Issue Type: Bug > Components: Execution - RPC >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam >Priority: Critical > Fix For: Future > > > TestDrillbitResilience#cancelWhenQueryIdArrives (previously named > cancelBeforeAnyResultsArrive) hangs when the test is run multiple times. > -(Will add more information)- > *Configuration: BIT_SERVER_RPC_THREADS = 1* > The remote RPC thread with a cancel signal is waiting for the fragment to > start accepting external events. The fragment is waiting for a > ControlConnection to the Foreman node (through ReconnectingConnection). The > Foreman node is waiting for the remote node to accept the connection, which > happens through an RPC thread. Distributed deadlock. > DRILL-3242 should solve this problem. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (DRILL-8051) Update the JQuery for Vulnerability issue
[ https://issues.apache.org/jira/browse/DRILL-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reassigned DRILL-8051: --- Assignee: James Turton > Update the JQuery for Vulnerability issue > - > > Key: DRILL-8051 > URL: https://issues.apache.org/jira/browse/DRILL-8051 > Project: Apache Drill > Issue Type: Improvement > Components: 1.19 >Affects Versions: 1.19.0 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: Future, 1.19.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Apache drill used JQuery 3.4.1 which has the vulnerability report form CVE. > [https://snyk.io/vuln/npm:jquery] > Needs to update JQuery version from 3.4.1 to 3.6.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (DRILL-8051) Update the JQuery for Vulnerability issue
Jingchuan Hu created DRILL-8051: --- Summary: Update the JQuery for Vulnerability issue Key: DRILL-8051 URL: https://issues.apache.org/jira/browse/DRILL-8051 Project: Apache Drill Issue Type: Improvement Components: 1.19 Affects Versions: 1.19.0 Reporter: Jingchuan Hu Fix For: Future, 1.19.0 Apache drill used JQuery 3.4.1 which has the vulnerability report form CVE. [https://snyk.io/vuln/npm:jquery] Needs to update JQuery version from 3.4.1 to 3.6.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8015) Add MongoDB Metastore implementation
[ https://issues.apache.org/jira/browse/DRILL-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447437#comment-17447437 ] ASF GitHub Bot commented on DRILL-8015: --- Leon-WTF commented on pull request #2384: URL: https://github.com/apache/drill/pull/2384#issuecomment-975571642 > @Leon-WTF Thanks for the contribution, this pull request is great! After completing the unit test and docs, please let me know, we can start a review. I have finished the ut, they are all passed. You can start the review, I will update the doc while replying the comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add MongoDB Metastore implementation > > > Key: DRILL-8015 > URL: https://issues.apache.org/jira/browse/DRILL-8015 > Project: Apache Drill > Issue Type: Improvement > Components: Metadata >Reporter: wtf >Assignee: wtf >Priority: Minor > > Add a NoSQL Metastore implementation -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8015) Add MongoDB Metastore implementation
[ https://issues.apache.org/jira/browse/DRILL-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447438#comment-17447438 ] ASF GitHub Bot commented on DRILL-8015: --- Leon-WTF edited a comment on pull request #2384: URL: https://github.com/apache/drill/pull/2384#issuecomment-975571642 > @Leon-WTF Thanks for the contribution, this pull request is great! After completing the unit test and docs, please let me know, we can start a review. @luocooong I have finished the ut, they are all passed. You can start the review, I will update the doc while replying the comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add MongoDB Metastore implementation > > > Key: DRILL-8015 > URL: https://issues.apache.org/jira/browse/DRILL-8015 > Project: Apache Drill > Issue Type: Improvement > Components: Metadata >Reporter: wtf >Assignee: wtf >Priority: Minor > > Add a NoSQL Metastore implementation -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8015) Add MongoDB Metastore implementation
[ https://issues.apache.org/jira/browse/DRILL-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447349#comment-17447349 ] ASF GitHub Bot commented on DRILL-8015: --- luocooong commented on pull request #2384: URL: https://github.com/apache/drill/pull/2384#issuecomment-975432320 @Leon-WTF Thanks for the contribution, this pull request is great! After completing the unit test and docs, please let me know, we can start a review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add MongoDB Metastore implementation > > > Key: DRILL-8015 > URL: https://issues.apache.org/jira/browse/DRILL-8015 > Project: Apache Drill > Issue Type: Improvement > Components: Metadata >Reporter: wtf >Assignee: wtf >Priority: Minor > > Add a NoSQL Metastore implementation -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8009) DrillConnectionImpl#isValid() doesn't correspond JDBC API
[
https://issues.apache.org/jira/browse/DRILL-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447339#comment-17447339
]
ASF GitHub Bot commented on DRILL-8009:
---
luocooong commented on pull request #2333:
URL: https://github.com/apache/drill/pull/2333#issuecomment-975419903
@rymarm Hello, sorry for the delay. Paul will take a look in the next couple
of days. The pull request is associated with the core module, Paul has the
most powerful voice. Thank you for understanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DrillConnectionImpl#isValid() doesn't correspond JDBC API
> -
>
> Key: DRILL-8009
> URL: https://issues.apache.org/jira/browse/DRILL-8009
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
>
> {{DrillConnectionImpl#isValid()}} doesn't correspond [Java JDBC API
> documentation|https://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#isValid(int)].
> Current implementation doesn't do actual connection verify and doesn't cover
> cases like drillbit termination and network issues.
> {{java.sql.Connection#isValid()}} (which extends {{DrillConnectionImpl}})
> widely used in JDBC connection pools like HikariCP and without right
> implementation, Drill JDBC driver cannot be used with them, because
> connections will keep alive forever.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-1282) Add read and write support for Parquet v2
[ https://issues.apache.org/jira/browse/DRILL-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447332#comment-17447332 ] ASF GitHub Bot commented on DRILL-1282: --- lgtm-com[bot] commented on pull request #2351: URL: https://github.com/apache/drill/pull/2351#issuecomment-975400967 This pull request **fixes 1 alert** when merging 021be9a7e0d47c2e97ee5e2a6d9da9187cd4b794 into baf698f63a1e549346fba59254472995e0f7be16 - [view on LGTM.com](https://lgtm.com/projects/g/apache/drill/rev/pr-78a017446219374fcc2f484dbb2cec6b1aafebff) **fixed alerts:** * 1 for Unused format argument -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add read and write support for Parquet v2 > - > > Key: DRILL-1282 > URL: https://issues.apache.org/jira/browse/DRILL-1282 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Reporter: Jacques Nadeau >Assignee: James Turton >Priority: Minor > Fix For: Future > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8009) DrillConnectionImpl#isValid() doesn't correspond JDBC API
[
https://issues.apache.org/jira/browse/DRILL-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447298#comment-17447298
]
ASF GitHub Bot commented on DRILL-8009:
---
rymarm commented on pull request #2333:
URL: https://github.com/apache/drill/pull/2333#issuecomment-975385292
@luocooong Hello! Can someone take a look at this PR? I pinged @paul-rogers
but seems he is busy, it has passed 18 days since my remind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DrillConnectionImpl#isValid() doesn't correspond JDBC API
> -
>
> Key: DRILL-8009
> URL: https://issues.apache.org/jira/browse/DRILL-8009
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
>
> {{DrillConnectionImpl#isValid()}} doesn't correspond [Java JDBC API
> documentation|https://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#isValid(int)].
> Current implementation doesn't do actual connection verify and doesn't cover
> cases like drillbit termination and network issues.
> {{java.sql.Connection#isValid()}} (which extends {{DrillConnectionImpl}})
> widely used in JDBC connection pools like HikariCP and without right
> implementation, Drill JDBC driver cannot be used with them, because
> connections will keep alive forever.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-1282) Add read and write support for Parquet v2
[
https://issues.apache.org/jira/browse/DRILL-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447273#comment-17447273
]
ASF GitHub Bot commented on DRILL-1282:
---
dzamo commented on a change in pull request #2351:
URL: https://github.com/apache/drill/pull/2351#discussion_r754090152
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestParquetWriterConfig.java
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.parquet;
+
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.nio.file.Paths;
+
+public class TestParquetWriterConfig extends ClusterTest {
Review comment:
@vdiravka Okay, I decided to upgrade `TestParquetWriter` to
`ClusterTest` so that this test can live in there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add read and write support for Parquet v2
> -
>
> Key: DRILL-1282
> URL: https://issues.apache.org/jira/browse/DRILL-1282
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
>Reporter: Jacques Nadeau
>Assignee: James Turton
>Priority: Minor
> Fix For: Future
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-1282) Add read and write support for Parquet v2
[
https://issues.apache.org/jira/browse/DRILL-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447272#comment-17447272
]
ASF GitHub Bot commented on DRILL-1282:
---
dzamo commented on a change in pull request #2351:
URL: https://github.com/apache/drill/pull/2351#discussion_r754090152
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestParquetWriterConfig.java
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.parquet;
+
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.nio.file.Paths;
+
+public class TestParquetWriterConfig extends ClusterTest {
Review comment:
Okay, I decided to upgrade `TestParquetWriter` to `ClusterTest` so that
this test can live in there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add read and write support for Parquet v2
> -
>
> Key: DRILL-1282
> URL: https://issues.apache.org/jira/browse/DRILL-1282
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
>Reporter: Jacques Nadeau
>Assignee: James Turton
>Priority: Minor
> Fix For: Future
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-1282) Add read and write support for Parquet v2
[
https://issues.apache.org/jira/browse/DRILL-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447252#comment-17447252
]
ASF GitHub Bot commented on DRILL-1282:
---
dzamo commented on a change in pull request #2351:
URL: https://github.com/apache/drill/pull/2351#discussion_r754050973
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java
##
@@ -141,40 +145,94 @@ public AbstractWriter getWriter(PhysicalOperator child,
String location, List options = new HashMap<>();
+Map writerOpts = new HashMap<>();
+OptionManager contextOpts = context.getOptions();
-options.put("location", writer.getLocation());
+writerOpts.put("location", writer.getLocation());
FragmentHandle handle = context.getHandle();
String fragmentId = String.format("%d_%d", handle.getMajorFragmentId(),
handle.getMinorFragmentId());
-options.put("prefix", fragmentId);
-
-options.put(ExecConstants.PARQUET_BLOCK_SIZE,
context.getOptions().getOption(ExecConstants.PARQUET_BLOCK_SIZE).num_val.toString());
-options.put(ExecConstants.PARQUET_WRITER_USE_SINGLE_FS_BLOCK,
-
context.getOptions().getOption(ExecConstants.PARQUET_WRITER_USE_SINGLE_FS_BLOCK).bool_val.toString());
-options.put(ExecConstants.PARQUET_PAGE_SIZE,
context.getOptions().getOption(ExecConstants.PARQUET_PAGE_SIZE).num_val.toString());
-options.put(ExecConstants.PARQUET_DICT_PAGE_SIZE,
context.getOptions().getOption(ExecConstants.PARQUET_DICT_PAGE_SIZE).num_val.toString());
-
-options.put(ExecConstants.PARQUET_WRITER_COMPRESSION_TYPE,
-
context.getOptions().getOption(ExecConstants.PARQUET_WRITER_COMPRESSION_TYPE).string_val);
-
-options.put(ExecConstants.PARQUET_WRITER_ENABLE_DICTIONARY_ENCODING,
-
context.getOptions().getOption(ExecConstants.PARQUET_WRITER_ENABLE_DICTIONARY_ENCODING).bool_val.toString());
-
-options.put(ExecConstants.PARQUET_WRITER_LOGICAL_TYPE_FOR_DECIMALS,
-
context.getOptions().getOption(ExecConstants.PARQUET_WRITER_LOGICAL_TYPE_FOR_DECIMALS).string_val);
-
-options.put(ExecConstants.PARQUET_WRITER_USE_PRIMITIVE_TYPES_FOR_DECIMALS,
-
context.getOptions().getOption(ExecConstants.PARQUET_WRITER_USE_PRIMITIVE_TYPES_FOR_DECIMALS).bool_val.toString());
+writerOpts.put("prefix", fragmentId);
+
+// Many options which follow may be set as Drill config options or in the
parquet format
+// plugin config. If there is a Drill option set at session scope or
narrower it takes precendence.
+OptionValue.OptionScope minScope = OptionValue.OptionScope.SESSION;
Review comment:
@vdiravka Okay let's discuss a design for this for a new PR. I guess it
would impact all the existing format plugins, and how they access format
configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add read and write support for Parquet v2
> -
>
> Key: DRILL-1282
> URL: https://issues.apache.org/jira/browse/DRILL-1282
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
>Reporter: Jacques Nadeau
>Assignee: James Turton
>Priority: Minor
> Fix For: Future
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-1282) Add read and write support for Parquet v2
[
https://issues.apache.org/jira/browse/DRILL-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447248#comment-17447248
]
ASF GitHub Bot commented on DRILL-1282:
---
dzamo commented on a change in pull request #2351:
URL: https://github.com/apache/drill/pull/2351#discussion_r754042756
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderConfig.java
##
@@ -186,11 +187,13 @@ public ParquetReaderConfig build() {
readerConfig.enableTimeReadCounter =
conf.getBoolean(ENABLE_TIME_READ_COUNTER, readerConfig.enableTimeReadCounter);
}
- // last assign values from session options, session options have higher
priority than other configurations
+ // last assign values from session or query scoped options which have
higher priority than other configurations
if (options != null) {
-String option =
options.getOption(ExecConstants.PARQUET_READER_STRINGS_SIGNED_MIN_MAX_VALIDATOR);
-if (!option.isEmpty()) {
- readerConfig.enableStringsSignedMinMax = Boolean.valueOf(option);
+String optVal = (String) options.getOption(
+ ExecConstants.PARQUET_READER_STRINGS_SIGNED_MIN_MAX
+).getValueMinScope(OptionValue.OptionScope.SESSION);
Review comment:
@vdiravka The idea for `getValueMinScope` is that it returns the value
of an option only if it is defined at scope as least as narrow as the passed-in
minimum. If you call it with `SESSION` as the minimum while the identified
option only has a value in `SYSTEM` or `BOOT` then null is returned. This is
useful for implementing the option priority QUERY > SESSION > FORMAT > SYSTEM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add read and write support for Parquet v2
> -
>
> Key: DRILL-1282
> URL: https://issues.apache.org/jira/browse/DRILL-1282
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
>Reporter: Jacques Nadeau
>Assignee: James Turton
>Priority: Minor
> Fix For: Future
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
