[jira] [Created] (DRILL-6915) Unit test mysql-test-data.sql in contrib/jdbc-storage-plugin fails on newer MacOS
Boaz Ben-Zvi created DRILL-6915:
---
Summary: Unit test mysql-test-data.sql in
contrib/jdbc-storage-plugin fails on newer MacOS
Key: DRILL-6915
URL: https://issues.apache.org/jira/browse/DRILL-6915
Project: Apache Drill
Issue Type: Bug
Components: Storage - JDBC
Affects Versions: 1.14.0
Environment: MacOS, either High Sierra (10.13) or Mojave (10.14).
Reporter: Boaz Ben-Zvi
The newer MacOS file systems (10.13 and above) are case-insensitive by default.
This leads to the following unit test failure:
{code:java}
~/drill > mvn clean install -rf :drill-jdbc-storage
[INFO] Scanning for projects...
[INFO]
[INFO] Detecting the operating system and CPU architecture
[INFO]
[INFO] os.detected.name: osx
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 10.14
.
[INFO]
[INFO] Building contrib/jdbc-storage-plugin 1.15.0-SNAPSHOT
[INFO]
.
[INFO] >> 2018-12-19 15:11:32 7136 [Warning] Setting lower_case_table_names=2
because file system for __drill/contrib/storage-jdbc/target/mysql-data/data/ is
case insensitive
.
[ERROR] Failed to execute:
create table CASESENSITIVETABLE (
a BLOB,
b BLOB
)
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] contrib/jdbc-storage-plugin FAILURE [01:30 min]
...
[ERROR] Failed to execute goal org.codehaus.mojo:sql-maven-plugin:1.5:execute
(create-tables) on project drill-jdbc-storage: Table 'casesensitivetable'
already exists -> [Help 1]{code}
in the test file *mysql-test-data.sql*, where +both+ tables
*caseSensitiveTable* and *CASESENSITIVETABLE* are created.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi resolved DRILL-6838.
--
Resolution: Fixed
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6914) Query with RuntimeFilter and SemiJoin fails with IllegalStateException: Memory was leaked by query
Abhishek Ravi created DRILL-6914:
Summary: Query with RuntimeFilter and SemiJoin fails with
IllegalStateException: Memory was leaked by query
Key: DRILL-6914
URL: https://issues.apache.org/jira/browse/DRILL-6914
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.15.0
Reporter: Abhishek Ravi
Fix For: 1.16.0
Following query fails on TPC-H SF 100 dataset when
exec.hashjoin.enable.runtime_filter = true AND planner.enable_semijoin = true.
Note that the query does not fail if any one of them or both are disabled.
{code:sql}
set `exec.hashjoin.enable.runtime_filter` = true;
set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
set `planner.enable_broadcast_join` = false;
set `planner.enable_semijoin` = true;
select
count(*) as row_count
from
lineitem l1
where
l1.l_shipdate IN (
select
distinct(cast(l2.l_shipdate as date))
from
lineitem l2);
reset `exec.hashjoin.enable.runtime_filter`;
reset `exec.hashjoin.runtime_filter.max.waiting.time`;
reset `planner.enable_broadcast_join`;
reset `planner.enable_semijoin`;
{code}
{noformat}
Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory
leaked: (134217728)
Allocator(frag:1:0) 800/134217728/172453568/70126322567
(res/actual/peak/limit)
Fragment 1:0
Please, refer to logs for more information.
[Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
(state=,code=0)
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
by query. Memory leaked: (134217728)
Allocator(frag:1:0) 800/134217728/172453568/70126322567
(res/actual/peak/limit)
Fragment 1:0
Please, refer to logs for more information.
[Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:640)
at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at sqlline.BufferedRows.(BufferedRows.java:37)
at sqlline.SqlLine.print(SqlLine.java:1716)
at sqlline.Commands.execute(Commands.java:949)
at sqlline.Commands.sql(Commands.java:882)
at sqlline.SqlLine.dispatch(SqlLine.java:725)
at sqlline.SqlLine.runCommands(SqlLine.java:1779)
at sqlline.Commands.run(Commands.java:1485)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:722)
at sqlline.SqlLine.initArgs(SqlLine.java:458)
at sqlline.SqlLine.begin(SqlLine.java:514)
at sqlline.SqlLine.start(SqlLine.java:264)
at sqlline.SqlLine.main(SqlLine.java:195)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
(134217728)
Allocator(frag:1:0) 800/134217728/172453568/70126322567
(res/actual/peak/limit)
Fragment 1:0
Please, refer to logs for more information.
[Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
at
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
[GitHub] vvysotskyi commented on issue #1570: Replace git-wip-us reference with gitbox
vvysotskyi commented on issue #1570: Replace git-wip-us reference with gitbox URL: https://github.com/apache/drill/pull/1570#issuecomment-448676231 @bbevens, did you have a chance to take a look at this? In this PR was only replaced obsolete links to the remote repository, since it caused problems for new drill users: https://stackoverflow.com/questions/53516086/apache-drill-assertionerror-relational-expression/53525547#comment94039836_53525547 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] arina-ielchiieva opened a new pull request #1578: DRILL-6913: Fix multiple error output in the console
arina-ielchiieva opened a new pull request #1578: DRILL-6913: Fix multiple error output in the console URL: https://github.com/apache/drill/pull/1578 1. Bump up Calcite version to 1.17.0-drill-r2 (includes CALCITE-2463). 2. Optimized exception handling in DrillSqlWorker getPlan method. Details in [DRILL-6913](https://issues.apache.org/jira/browse/DRILL-6913). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] arina-ielchiieva commented on a change in pull request #1577: DRILL-6912: NPE when other drillbit is already running
arina-ielchiieva commented on a change in pull request #1577: DRILL-6912: NPE
when other drillbit is already running
URL: https://github.com/apache/drill/pull/1577#discussion_r242989144
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestTwoDrillbitsWithSamePort.java
##
@@ -0,0 +1,68 @@
+package org.apache.drill.exec.server;
+
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.Test;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+public class TestTwoDrillbitsWithSamePort extends ClusterTest {
Review comment:
Is there a reason to keep this test separately from other Graceful Shutdown
tests?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] arina-ielchiieva commented on a change in pull request #1577: DRILL-6912: NPE when other drillbit is already running
arina-ielchiieva commented on a change in pull request #1577: DRILL-6912: NPE when other drillbit is already running URL: https://github.com/apache/drill/pull/1577#discussion_r242989504 ## File path: exec/java-exec/src/test/java/org/apache/drill/exec/server/TestTwoDrillbitsWithSamePort.java ## @@ -0,0 +1,68 @@ +package org.apache.drill.exec.server; + +import org.apache.drill.test.ClusterFixture; +import org.apache.drill.test.ClusterFixtureBuilder; +import org.apache.drill.test.ClusterTest; +import org.junit.Test; + +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +/* Review comment: License should be before package. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] arina-ielchiieva commented on a change in pull request #1577: DRILL-6912: NPE when other drillbit is already running
arina-ielchiieva commented on a change in pull request #1577: DRILL-6912: NPE
when other drillbit is already running
URL: https://github.com/apache/drill/pull/1577#discussion_r242989610
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestTwoDrillbitsWithSamePort.java
##
@@ -0,0 +1,68 @@
+package org.apache.drill.exec.server;
+
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.Test;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+public class TestTwoDrillbitsWithSamePort extends ClusterTest {
+
+
+ /**
+ * Provides test coverage for issue: DRILL-6912
+ */
+ @Test
+ public void gracefulShutdownThreadShouldBeInitializedBeforeClosingDrillbit()
throws Exception {
+Drillbit drillbit = null;
+Drillbit drillbit1 = null;
+
+try {
+ ClusterFixtureBuilder fixtureBuilder =
ClusterFixture.bareBuilder(dirTestWatcher).withLocalZk();
+ ClusterFixture clusterFixture = fixtureBuilder.build();
+ drillbit = clusterFixture.drillbit();
+
+ // creating another drillbit instance using same config
+ drillbit1 = new Drillbit(clusterFixture.config(),
fixtureBuilder.configBuilder().getDefinitions(),
+ clusterFixture.serviceSet());
+
+ try {
+drillbit1.run();
+ } catch (org.apache.drill.common.exceptions.UserException be) {
+// it's expected that second drillbit can't be started because port is
busy
+assertTrue(be.getMessage().startsWith("RESOURCE ERROR: Drillbit could
not bind to port 31010"));
+ }
+} finally {
+ // preconditions
+ assertNotNull(drillbit);
+ assertNotNull(drillbit1);
+ assertNotNull("gracefulShutdownThread should be initialized, otherwise
npe will be thrown from close()",
+ drillbit.gracefulShutdownThread);
+ // main test case
+ assertNotNull("gracefulShutdownThread should be initialized, otherwise
npe will be thrown from close()",
Review comment:
npe -> NPE
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] arina-ielchiieva commented on a change in pull request #1577: DRILL-6912: NPE when other drillbit is already running
arina-ielchiieva commented on a change in pull request #1577: DRILL-6912: NPE
when other drillbit is already running
URL: https://github.com/apache/drill/pull/1577#discussion_r242989350
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestTwoDrillbitsWithSamePort.java
##
@@ -0,0 +1,68 @@
+package org.apache.drill.exec.server;
+
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.Test;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+public class TestTwoDrillbitsWithSamePort extends ClusterTest {
+
+
+ /**
+ * Provides test coverage for issue: DRILL-6912
+ */
+ @Test
+ public void gracefulShutdownThreadShouldBeInitializedBeforeClosingDrillbit()
throws Exception {
+Drillbit drillbit = null;
+Drillbit drillbit1 = null;
+
+try {
+ ClusterFixtureBuilder fixtureBuilder =
ClusterFixture.bareBuilder(dirTestWatcher).withLocalZk();
+ ClusterFixture clusterFixture = fixtureBuilder.build();
+ drillbit = clusterFixture.drillbit();
+
+ // creating another drillbit instance using same config
+ drillbit1 = new Drillbit(clusterFixture.config(),
fixtureBuilder.configBuilder().getDefinitions(),
+ clusterFixture.serviceSet());
+
+ try {
+drillbit1.run();
+ } catch (org.apache.drill.common.exceptions.UserException be) {
Review comment:
Can path be imported?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] arina-ielchiieva commented on issue #1577: DRILL-6912: NPE when other drillbit is already running
arina-ielchiieva commented on issue #1577: DRILL-6912: NPE when other drillbit is already running URL: https://github.com/apache/drill/pull/1577#issuecomment-448636567 @ihuzenko please check `TestGracefulShutdown` class. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ihuzenko commented on issue #1577: DRILL-6912: NPE when other drillbit is already running
ihuzenko commented on issue #1577: DRILL-6912: NPE when other drillbit is already running URL: https://github.com/apache/drill/pull/1577#issuecomment-448628839 @vvysotskyi can you suggest how to assert this ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vvysotskyi commented on issue #1577: DRILL-6912: NPE when other drillbit is already running
vvysotskyi commented on issue #1577: DRILL-6912: NPE when other drillbit is already running URL: https://github.com/apache/drill/pull/1577#issuecomment-448622987 Changes look good, but can we add some tests for this issue? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Oracle JDBC storage plugin
I've configured drill to use an JDBC storage plugin. My connection string is for an Oracle database. I have included the Oracle JDBC driver to my drill deployment. The connection is established correctly. However the storage plugin fails to retrieve the schema of the database. The JDBC API provides two ways to get metadata about the database: getSchemas() and getCatalogs(). Depending on the database vendor one of those calls will return the information. The storage plugin correctly tries both calls. However I believe there is an issue when using Oracle. The JDBC API claims that getSchemas() returns a set of two columns "table schema" and "table catalog" but in reality it just returns "table schema" as a first column of the result set. I believe this to be a bug. Has anyone tested the JDBC storage plugin against an Oracle DB and successfully retrieved the list of schema and tables? This article explains in details the issue in retrieving schema information using JDBC. https://www.wisdomjobs.com/e-university/jdbc-tutorial-278/what-are-catalogs-and-schemas-1147.html Thank you jc
[GitHub] ihuzenko opened a new pull request #1577: DRILL-6912: NPE when other drillbit is already running
ihuzenko opened a new pull request #1577: DRILL-6912: NPE when other drillbit is already running URL: https://github.com/apache/drill/pull/1577 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ihuzenko commented on issue #1559: DRILL-540: Allow querying hive views in Drill
ihuzenko commented on issue #1559: DRILL-540: Allow querying hive views in Drill URL: https://github.com/apache/drill/pull/1559#issuecomment-448602644 @vdiravka I've addressed comments, review please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1559: DRILL-540: Allow querying hive views in Drill
ihuzenko commented on a change in pull request #1559: DRILL-540: Allow querying
hive views in Drill
URL: https://github.com/apache/drill/pull/1559#discussion_r242921323
##
File path:
contrib/storage-hive/core/src/test/java/org/apache/drill/exec/impersonation/hive/TestStorageBasedHiveAuthorization.java
##
@@ -145,6 +148,41 @@ private static void
setStorabaseBasedAuthorizationInHiveConf() {
return hiveConfig;
}
+ /**
+ * User Groups
+ * ---
Review comment:
added
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1559: DRILL-540: Allow querying hive views in Drill
ihuzenko commented on a change in pull request #1559: DRILL-540: Allow querying
hive views in Drill
URL: https://github.com/apache/drill/pull/1559#discussion_r242920937
##
File path:
contrib/storage-hive/core/src/test/java/org/apache/drill/exec/impersonation/hive/TestSqlStdBasedAuthorization.java
##
@@ -137,15 +173,10 @@ private static void generateTestData() throws Exception {
String.format("SELECT rownum, name, age FROM %s.%s.%s",
MINI_DFS_STORAGE_PLUGIN_NAME, "tmp", v_student_u0g0_750));
}
- private static void createTbl(final Driver driver, final String db, final
String tbl, final String tblDef,
- final String data) throws Exception {
-executeQuery(driver, String.format(tblDef, db, tbl));
-executeQuery(driver, String.format("LOAD DATA LOCAL INPATH '%s' INTO TABLE
%s.%s", data, db, tbl));
- }
-
// Irrespective of each db permissions, all dbs show up in "SHOW SCHEMAS"
@Test
public void showSchemas() throws Exception {
+test("USE " + MINI_DFS_STORAGE_PLUGIN_NAME);
Review comment:
Added fix with comment in HiveAuthorizationHelper .
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1559: DRILL-540: Allow querying hive views in Drill
ihuzenko commented on a change in pull request #1559: DRILL-540: Allow querying
hive views in Drill
URL: https://github.com/apache/drill/pull/1559#discussion_r242920560
##
File path:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/types/HiveToRelDataTypeConverter.java
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.types;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.SqlCollation;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.util.Util;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector.Category;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class is responsible for data type conversions
+ * from {@link org.apache.hadoop.hive.metastore.api.FieldSchema} instances
+ * to {@link org.apache.calcite.rel.type.RelDataType} instances
+ */
+public class HiveToRelDataTypeConverter {
+
+ private static final Logger logger =
LoggerFactory.getLogger(HiveToRelDataTypeConverter.class);
+
+ private static final String UNSUPPORTED_HIVE_DATA_TYPE_ERROR_MSG =
"Unsupported Hive data type %s. %n" +
+ "Following Hive data types are supported in Drill INFORMATION_SCHEMA: " +
+ "BOOLEAN, BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP,
BINARY, DECIMAL, STRING, " +
+ "VARCHAR, CHAR, LIST, MAP, STRUCT and UNION";
+
+
+ private final RelDataTypeFactory typeFactory;
+
+ public HiveToRelDataTypeConverter(RelDataTypeFactory typeFactory) {
+this.typeFactory = typeFactory;
+ }
+
+ /**
+ * Performs conversion from Hive field to nullable RelDataType
+ *
+ * @param field - representation of data type in Hive Metastore
+ * @return appropriate nullable RelDataType for using with Calcite
+ * @throws RuntimeException for unsupported data types, check
+ * {@link
HiveToRelDataTypeConverter#UNSUPPORTED_HIVE_DATA_TYPE_ERROR_MSG}
+ * for details about supported hive types
+ */
+ public RelDataType convertToNullableRelDataType(FieldSchema field) {
+TypeInfo fieldTypeInfo =
TypeInfoUtils.getTypeInfoFromTypeString(field.getType());
+RelDataType relDataType = convertToRelDataType(fieldTypeInfo);
+return typeFactory.createTypeWithNullability(relDataType, true);
+ }
+
+ private RelDataType convertToRelDataType(TypeInfo typeInfo) {
+final Category typeCategory = typeInfo.getCategory();
+switch (typeCategory) {
+ case PRIMITIVE:
+return getRelDataType((PrimitiveTypeInfo) typeInfo);
+ case LIST:
+return getRelDataType((ListTypeInfo) typeInfo);
+ case MAP:
+return getRelDataType((MapTypeInfo) typeInfo);
+ case STRUCT:
+return getRelDataType((StructTypeInfo) typeInfo);
+ case UNION:
+logger.warn("There is no UNION data type in SQL. Converting it to Sql
type OTHER to avoid " +
+"breaking INFORMATION_SCHEMA queries");
+return typeFactory.createSqlType(SqlTypeName.OTHER);
+}
+throw new
RuntimeException(String.format(UNSUPPORTED_HIVE_DATA_TYPE_ERROR_MSG,
typeCategory));
+ }
+
+ private RelDataType getRelDataType(StructTypeInfo structTypeInfo) {
+final List fieldNames = structTypeInfo.getAllStructFieldNames();
+final List relDataTypes =
structTypeInfo.getAllStructFieldTypeInfos()
+.stream()
+.map(this::convertToRelDataType)
+.collect(Collectors.toList());
+
[GitHub] ihuzenko commented on a change in pull request #1559: DRILL-540: Allow querying hive views in Drill
ihuzenko commented on a change in pull request #1559: DRILL-540: Allow querying
hive views in Drill
URL: https://github.com/apache/drill/pull/1559#discussion_r242546811
##
File path:
contrib/storage-hive/core/src/test/java/org/apache/drill/exec/impersonation/hive/TestSqlStdBasedAuthorization.java
##
@@ -137,15 +173,10 @@ private static void generateTestData() throws Exception {
String.format("SELECT rownum, name, age FROM %s.%s.%s",
MINI_DFS_STORAGE_PLUGIN_NAME, "tmp", v_student_u0g0_750));
}
- private static void createTbl(final Driver driver, final String db, final
String tbl, final String tblDef,
- final String data) throws Exception {
-executeQuery(driver, String.format(tblDef, db, tbl));
-executeQuery(driver, String.format("LOAD DATA LOCAL INPATH '%s' INTO TABLE
%s.%s", data, db, tbl));
- }
-
// Irrespective of each db permissions, all dbs show up in "SHOW SCHEMAS"
@Test
public void showSchemas() throws Exception {
+test("USE " + MINI_DFS_STORAGE_PLUGIN_NAME);
Review comment:
without this option test is failing, possibly one of another tests has call
to USE hive; which has negative impact on the test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
Re: [VOTE] Apache Drill release 1.15.0 - RC0
@Aman Sinha I am investigating, which maven plugin
causes creating this dir.
I guess this sinks rc0. I'll prepare new release candidate once
issues in DRILL-6912 [1] and DRILL-6913 [2] are fixed.
[1] https://issues.apache.org/jira/browse/DRILL-6912
[2] https://issues.apache.org/jira/browse/DRILL-6913
Kind regards
Vitalii
On Wed, Dec 19, 2018 at 1:58 PM Arina Ielchiieva wrote:
> I think above issues are blockers, Vitalii please cancel the vote
>
> Kind regards,
> Arina
>
> On Wed, Dec 19, 2018 at 6:09 AM Aman Sinha wrote:
>
> > @vita...@apache.org any idea why there's an
> > extraneous directory in the source ?
> > drwxrwxr-x vitalii/vitalii 0 2018-12-18 03:48
> > apache-drill-1.15.0-src/${project.*basedir*}/
> >
> > drwxrwxr-x vitalii/vitalii 0 2018-12-18 03:48
> > apache-drill-1.15.0-src/${project.*basedir*}/src/
> >
> > drwxrwxr-x vitalii/vitalii 0 2018-12-18 03:48
> > apache-drill-1.15.0-src/${project.*basedir*}/src/site/
> >
> > drwxrwxr-x vitalii/vitalii 0 2018-12-18 03:48
> > apache-drill-1.15.0-src/${project.*basedir*}/src/site/resources/
> >
> > drwxrwxr-x vitalii/vitalii 0 2018-12-18 03:48
> > apache-drill-1.15.0-src/${project.*basedir*}/src/site/resources/repo/
> >
> > On Tue, Dec 18, 2018 at 10:58 AM Vitalii Diravka
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to propose the first release candidate (RC0) of Apache Drill,
> > > version 1.15.0.
> > >
> > > The release candidate covers a total of 185 resolved JIRAs [1].
> > > Thanks to everyone who contributed to this release.
> > >
> > > The tarball artifacts are hosted at [2] and the maven artifacts are
> > hosted
> > > at [3].
> > >
> > > This release candidate is based on commit
> > > ff797695de0e27a732c22e2410cbef58abbfcef3 located at [4].
> > >
> > > Please download and try out the release.
> > >
> > > The vote ends at 7:00 PM UTC (11:00 AM PDT, 9:00 PM EET, 0:30 AM IST),
> > Dec
> > > 21th, 2018
> > >
> > > [ ] +1
> > > [ ] +0
> > > [ ] -1
> > >
> > > I have found two issues, which are not observed earlier [5], [6].
> Should
> > we
> > > consider them as blockers?
> > >
> > > [1]
> > > *
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343317==12313820
> > > <
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343317==12313820
> > > >*
> > >
> > > [2] *http://home.apache.org/~vitalii/drill/releases/1.15.0/rc0/
> > > *
> > >
> > > [5] https://issues.apache.org/jira/browse/DRILL-6912
> > >
> > > [6] https://issues.apache.org/jira/browse/DRILL-6913
> > >
> > > Kind regards
> > > Vitalii
> > >
> >
>
Re: [VOTE] Apache Drill release 1.15.0 - RC0
I think above issues are blockers, Vitalii please cancel the vote
Kind regards,
Arina
On Wed, Dec 19, 2018 at 6:09 AM Aman Sinha wrote:
> @vita...@apache.org any idea why there's an
> extraneous directory in the source ?
> drwxrwxr-x vitalii/vitalii 0 2018-12-18 03:48
> apache-drill-1.15.0-src/${project.*basedir*}/
>
> drwxrwxr-x vitalii/vitalii 0 2018-12-18 03:48
> apache-drill-1.15.0-src/${project.*basedir*}/src/
>
> drwxrwxr-x vitalii/vitalii 0 2018-12-18 03:48
> apache-drill-1.15.0-src/${project.*basedir*}/src/site/
>
> drwxrwxr-x vitalii/vitalii 0 2018-12-18 03:48
> apache-drill-1.15.0-src/${project.*basedir*}/src/site/resources/
>
> drwxrwxr-x vitalii/vitalii 0 2018-12-18 03:48
> apache-drill-1.15.0-src/${project.*basedir*}/src/site/resources/repo/
>
> On Tue, Dec 18, 2018 at 10:58 AM Vitalii Diravka
> wrote:
>
> > Hi all,
> >
> > I'd like to propose the first release candidate (RC0) of Apache Drill,
> > version 1.15.0.
> >
> > The release candidate covers a total of 185 resolved JIRAs [1].
> > Thanks to everyone who contributed to this release.
> >
> > The tarball artifacts are hosted at [2] and the maven artifacts are
> hosted
> > at [3].
> >
> > This release candidate is based on commit
> > ff797695de0e27a732c22e2410cbef58abbfcef3 located at [4].
> >
> > Please download and try out the release.
> >
> > The vote ends at 7:00 PM UTC (11:00 AM PDT, 9:00 PM EET, 0:30 AM IST),
> Dec
> > 21th, 2018
> >
> > [ ] +1
> > [ ] +0
> > [ ] -1
> >
> > I have found two issues, which are not observed earlier [5], [6]. Should
> we
> > consider them as blockers?
> >
> > [1]
> > *
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343317==12313820
> > <
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343317==12313820
> > >*
> >
> > [2] *http://home.apache.org/~vitalii/drill/releases/1.15.0/rc0/
> > >*
> >
> > [4] *https://github.com/vdiravka/drill/commits/drill-1.15.0
> > *
> >
> > [5] https://issues.apache.org/jira/browse/DRILL-6912
> >
> > [6] https://issues.apache.org/jira/browse/DRILL-6913
> >
> > Kind regards
> > Vitalii
> >
>
