[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[ https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922644#comment-16922644 ] ASF GitHub Bot commented on DRILL-7343: --- cgivre commented on issue #1840: DRILL-7343: Add User-Agent UDFs to Drill URL: https://github.com/apache/drill/pull/1840#issuecomment-527979825 @KazydubB I opened a JIRA for the null issues, but this PR should be ready to go. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add User-Agent UDFs to Drill > > > Key: DRILL-7343 > URL: https://issues.apache.org/jira/browse/DRILL-7343 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.17.0 > > > This collection of UDFs adds the ability to parse user agent strings which is > useful for security data analysis. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DRILL-7366) Improve Null Handling for UDFs with Complex Output
Charles Givre created DRILL-7366:
Summary: Improve Null Handling for UDFs with Complex Output
Key: DRILL-7366
URL: https://issues.apache.org/jira/browse/DRILL-7366
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.16.0
Reporter: Charles Givre
Fix For: 1.17.0
If there is a UDF which has a complex field (Map or List) as output, Drill does
not allow the UDF to have nullable input and it creates additional complexity
when writing these kinds of UDFs.
I therefore would like to propose that two options be added to the
FunctionTemplate for null handling: {{EMPTY_LIST_IF_NULL}}, and
{{EMPTY_MAP_IF_NULL}} which, would simplify UDF creation. I'm envisioning that
if either of these options were selected, and the UDF receives any null value
as input, the UDF will return either an empty map or list.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[ https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922585#comment-16922585 ] ASF GitHub Bot commented on DRILL-7343: --- KazydubB commented on issue #1840: DRILL-7343: Add User-Agent UDFs to Drill URL: https://github.com/apache/drill/pull/1840#issuecomment-527940960 @cgivre I agree that having two functions for `OPTIONAL` and `REQUIRED` cases is not the best experience, but returning an empty list on `NULL` is not enough, because there may be functions that will need to return a `MAP` for example. So, there's a need to provide `NullHandling` for `NULL` input for other cases too (`EMPTY_LIST_IF_NULL`, `EMPTY_MAP_IF_NULL` etc.). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add User-Agent UDFs to Drill > > > Key: DRILL-7343 > URL: https://issues.apache.org/jira/browse/DRILL-7343 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.17.0 > > > This collection of UDFs adds the ability to parse user agent strings which is > useful for security data analysis. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[ https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922580#comment-16922580 ] ASF GitHub Bot commented on DRILL-7343: --- cgivre commented on issue #1840: DRILL-7343: Add User-Agent UDFs to Drill URL: https://github.com/apache/drill/pull/1840#issuecomment-527935677 @KazydubB What happens in the `parse_query` function is that there is a duplicate function of the same name which allows null input which returns an empty list. I had tried this before, but apparently didn't use the correct parameters so it didn't work for me, BUT, now it does so yay! In any event, this seems like horrible design. I'm going to open a JIRA to create a new function handler which returns an empty list on null. That seems like a better approach than having to write 2 UDFS for every UDF with a complex writer output. I also added a series of unit tests to test this functionality. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add User-Agent UDFs to Drill > > > Key: DRILL-7343 > URL: https://issues.apache.org/jira/browse/DRILL-7343 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.17.0 > > > This collection of UDFs adds the ability to parse user agent strings which is > useful for security data analysis. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922552#comment-16922552
]
ASF GitHub Bot commented on DRILL-7343:
---
KazydubB commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to
Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r320787922
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
+ .baselineColumns("DeviceClass", "DeviceName",
"DeviceBrand","DeviceCpuBits","OperatingSystemClass",
"OperatingSystemName","OperatingSystemVersion",
+
"OperatingSystemVersionMajor","OperatingSystemNameVersion","OperatingSystemNameVersionMajor","LayoutEngineClass","LayoutEngineName","LayoutEngineVersion",
+
"LayoutEngineVersionMajor","LayoutEngineNameVersion","LayoutEngineBuild","AgentClass","AgentName","AgentVersion","AgentVersionMajor","AgentNameVersionMajor",
+"AgentLanguage","AgentLanguageCode","AgentSecurity")
+ .baselineValues("Desktop","Desktop", "Unknown","32", "Desktop", "Windows
NT", "XP", "XP", "Windows XP", "Windows XP", "Browser", "Gecko", "1.8.1.11",
"1", "Gecko 1.8.1.11",
+"20071127", "Browser", "Firefox", "2.0.0.11", "2", "Firefox 2",
"English (United States)", "en-us","Strong security")
+ .go();
+ }
+
+ @Test
+ public void testGetHostName() throws Exception {
+String query = "SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11', 'AgentSecurity')
AS agent FROM " +
+ "(values" +
+ "(1))";
+
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922550#comment-16922550
]
ASF GitHub Bot commented on DRILL-7343:
---
KazydubB commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to
Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r320787922
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
+ .baselineColumns("DeviceClass", "DeviceName",
"DeviceBrand","DeviceCpuBits","OperatingSystemClass",
"OperatingSystemName","OperatingSystemVersion",
+
"OperatingSystemVersionMajor","OperatingSystemNameVersion","OperatingSystemNameVersionMajor","LayoutEngineClass","LayoutEngineName","LayoutEngineVersion",
+
"LayoutEngineVersionMajor","LayoutEngineNameVersion","LayoutEngineBuild","AgentClass","AgentName","AgentVersion","AgentVersionMajor","AgentNameVersionMajor",
+"AgentLanguage","AgentLanguageCode","AgentSecurity")
+ .baselineValues("Desktop","Desktop", "Unknown","32", "Desktop", "Windows
NT", "XP", "XP", "Windows XP", "Windows XP", "Browser", "Gecko", "1.8.1.11",
"1", "Gecko 1.8.1.11",
+"20071127", "Browser", "Firefox", "2.0.0.11", "2", "Firefox 2",
"English (United States)", "en-us","Strong security")
+ .go();
+ }
+
+ @Test
+ public void testGetHostName() throws Exception {
+String query = "SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11', 'AgentSecurity')
AS agent FROM " +
+ "(values" +
+ "(1))";
+
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922551#comment-16922551
]
ASF GitHub Bot commented on DRILL-7343:
---
KazydubB commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to
Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r320787922
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
+ .baselineColumns("DeviceClass", "DeviceName",
"DeviceBrand","DeviceCpuBits","OperatingSystemClass",
"OperatingSystemName","OperatingSystemVersion",
+
"OperatingSystemVersionMajor","OperatingSystemNameVersion","OperatingSystemNameVersionMajor","LayoutEngineClass","LayoutEngineName","LayoutEngineVersion",
+
"LayoutEngineVersionMajor","LayoutEngineNameVersion","LayoutEngineBuild","AgentClass","AgentName","AgentVersion","AgentVersionMajor","AgentNameVersionMajor",
+"AgentLanguage","AgentLanguageCode","AgentSecurity")
+ .baselineValues("Desktop","Desktop", "Unknown","32", "Desktop", "Windows
NT", "XP", "XP", "Windows XP", "Windows XP", "Browser", "Gecko", "1.8.1.11",
"1", "Gecko 1.8.1.11",
+"20071127", "Browser", "Firefox", "2.0.0.11", "2", "Firefox 2",
"English (United States)", "en-us","Strong security")
+ .go();
+ }
+
+ @Test
+ public void testGetHostName() throws Exception {
+String query = "SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11', 'AgentSecurity')
AS agent FROM " +
+ "(values" +
+ "(1))";
+
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922541#comment-16922541
]
ASF GitHub Bot commented on DRILL-7343:
---
cgivre commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r320784690
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
+ .baselineColumns("DeviceClass", "DeviceName",
"DeviceBrand","DeviceCpuBits","OperatingSystemClass",
"OperatingSystemName","OperatingSystemVersion",
+
"OperatingSystemVersionMajor","OperatingSystemNameVersion","OperatingSystemNameVersionMajor","LayoutEngineClass","LayoutEngineName","LayoutEngineVersion",
+
"LayoutEngineVersionMajor","LayoutEngineNameVersion","LayoutEngineBuild","AgentClass","AgentName","AgentVersion","AgentVersionMajor","AgentNameVersionMajor",
+"AgentLanguage","AgentLanguageCode","AgentSecurity")
+ .baselineValues("Desktop","Desktop", "Unknown","32", "Desktop", "Windows
NT", "XP", "XP", "Windows XP", "Windows XP", "Browser", "Gecko", "1.8.1.11",
"1", "Gecko 1.8.1.11",
+"20071127", "Browser", "Firefox", "2.0.0.11", "2", "Firefox 2",
"English (United States)", "en-us","Strong security")
+ .go();
+ }
+
+ @Test
+ public void testGetHostName() throws Exception {
+String query = "SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11', 'AgentSecurity')
AS agent FROM " +
+ "(values" +
+ "(1))";
+
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922538#comment-16922538
]
ASF GitHub Bot commented on DRILL-7343:
---
cgivre commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r320782860
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
+ .baselineColumns("DeviceClass", "DeviceName",
"DeviceBrand","DeviceCpuBits","OperatingSystemClass",
"OperatingSystemName","OperatingSystemVersion",
+
"OperatingSystemVersionMajor","OperatingSystemNameVersion","OperatingSystemNameVersionMajor","LayoutEngineClass","LayoutEngineName","LayoutEngineVersion",
+
"LayoutEngineVersionMajor","LayoutEngineNameVersion","LayoutEngineBuild","AgentClass","AgentName","AgentVersion","AgentVersionMajor","AgentNameVersionMajor",
+"AgentLanguage","AgentLanguageCode","AgentSecurity")
+ .baselineValues("Desktop","Desktop", "Unknown","32", "Desktop", "Windows
NT", "XP", "XP", "Windows XP", "Windows XP", "Browser", "Gecko", "1.8.1.11",
"1", "Gecko 1.8.1.11",
+"20071127", "Browser", "Firefox", "2.0.0.11", "2", "Firefox 2",
"English (United States)", "en-us","Strong security")
+ .go();
+ }
+
+ @Test
+ public void testGetHostName() throws Exception {
+String query = "SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11', 'AgentSecurity')
AS agent FROM " +
+ "(values" +
+ "(1))";
+
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922533#comment-16922533
]
ASF GitHub Bot commented on DRILL-7343:
---
KazydubB commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to
Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r320781926
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
+ .baselineColumns("DeviceClass", "DeviceName",
"DeviceBrand","DeviceCpuBits","OperatingSystemClass",
"OperatingSystemName","OperatingSystemVersion",
+
"OperatingSystemVersionMajor","OperatingSystemNameVersion","OperatingSystemNameVersionMajor","LayoutEngineClass","LayoutEngineName","LayoutEngineVersion",
+
"LayoutEngineVersionMajor","LayoutEngineNameVersion","LayoutEngineBuild","AgentClass","AgentName","AgentVersion","AgentVersionMajor","AgentNameVersionMajor",
+"AgentLanguage","AgentLanguageCode","AgentSecurity")
+ .baselineValues("Desktop","Desktop", "Unknown","32", "Desktop", "Windows
NT", "XP", "XP", "Windows XP", "Windows XP", "Browser", "Gecko", "1.8.1.11",
"1", "Gecko 1.8.1.11",
+"20071127", "Browser", "Firefox", "2.0.0.11", "2", "Firefox 2",
"English (United States)", "en-us","Strong security")
+ .go();
+ }
+
+ @Test
+ public void testGetHostName() throws Exception {
+String query = "SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11', 'AgentSecurity')
AS agent FROM " +
+ "(values" +
+ "(1))";
+
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922491#comment-16922491
]
ASF GitHub Bot commented on DRILL-7343:
---
cgivre commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r320763598
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
+ .baselineColumns("DeviceClass", "DeviceName",
"DeviceBrand","DeviceCpuBits","OperatingSystemClass",
"OperatingSystemName","OperatingSystemVersion",
+
"OperatingSystemVersionMajor","OperatingSystemNameVersion","OperatingSystemNameVersionMajor","LayoutEngineClass","LayoutEngineName","LayoutEngineVersion",
+
"LayoutEngineVersionMajor","LayoutEngineNameVersion","LayoutEngineBuild","AgentClass","AgentName","AgentVersion","AgentVersionMajor","AgentNameVersionMajor",
+"AgentLanguage","AgentLanguageCode","AgentSecurity")
+ .baselineValues("Desktop","Desktop", "Unknown","32", "Desktop", "Windows
NT", "XP", "XP", "Windows XP", "Windows XP", "Browser", "Gecko", "1.8.1.11",
"1", "Gecko 1.8.1.11",
+"20071127", "Browser", "Firefox", "2.0.0.11", "2", "Firefox 2",
"English (United States)", "en-us","Strong security")
+ .go();
+ }
+
+ @Test
+ public void testGetHostName() throws Exception {
+String query = "SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11', 'AgentSecurity')
AS agent FROM " +
+ "(values" +
+ "(1))";
+
[jira] [Updated] (DRILL-7365) Failed to read column added to existing Hive partition
[
https://issues.apache.org/jira/browse/DRILL-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Guzenko updated DRILL-7365:
Description:
*Prerequisities:*
Enable ACID in Hive
[https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions].
*Steps to reproduce:*
1) create table hive_bucketed2 (emp_id int, first_name string) PARTITIONED BY
(`col_year_month` string) clustered by (emp_id) into 4 buckets stored as orc
tblproperties ('transactional'='true');
2) insert into hive_bucketed2 PARTITION (col_year_month = '2019-09') values
(1, 'A'),(2, 'B');
3) alter table hive_bucketed2 add columns (age INT);
4) insert into hive_bucketed2 PARTITION (col_year_month = '2019-09') values
(11, '1A', 10),(12, '1B', 22);
5) select * from hive.hive_bucketed2;
*Workaround* (may be a little bit {color:#de350b}risky{color}:) :
1. Connect to Hive metastore database.
[https://analyticsanvil.files.wordpress.com/2016/08/hive_metastore_database_diagram.png]
2. Find SDS linked to desired PARTITIONS . Actually you need CD_ID's for such
SDS.
3. Insert your column into COLUMNS_V2 with CD_ID found at previous step.
was:
Prerequisities:
Enable ACID in Hive
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions.
Steps to reproduce:
1) create table hive_bucketed2 (emp_id int, first_name string) PARTITIONED BY
(`col_year_month` string) clustered by (emp_id) into 4 buckets stored as orc
tblproperties ('transactional'='true');
2) insert into hive_bucketed2 PARTITION (col_year_month = '2019-09') values (1,
'A'),(2, 'B');
3) alter table hive_bucketed2 add columns (age INT);
4) insert into hive_bucketed2 PARTITION (col_year_month = '2019-09') values
(11, '1A', 10),(12, '1B', 22);
5) select * from hive.hive_bucketed2;
Workaround (may be a little bit risky:) :
1. Connect to Hive metastore database.
https://analyticsanvil.files.wordpress.com/2016/08/hive_metastore_database_diagram.png
2. Find SDS linked to desired PARTITIONS . Actually you need CD_ID's for such
SDS.
3. Insert your column into COLUMNS_V2 with CD_ID found at previous step.
> Failed to read column added to existing Hive partition
> --
>
> Key: DRILL-7365
> URL: https://issues.apache.org/jira/browse/DRILL-7365
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Reporter: Igor Guzenko
>Priority: Major
>
> *Prerequisities:*
> Enable ACID in Hive
> [https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions].
> *Steps to reproduce:*
> 1) create table hive_bucketed2 (emp_id int, first_name string) PARTITIONED BY
> (`col_year_month` string) clustered by (emp_id) into 4 buckets stored as orc
> tblproperties ('transactional'='true');
> 2) insert into hive_bucketed2 PARTITION (col_year_month = '2019-09') values
> (1, 'A'),(2, 'B');
> 3) alter table hive_bucketed2 add columns (age INT);
> 4) insert into hive_bucketed2 PARTITION (col_year_month = '2019-09') values
> (11, '1A', 10),(12, '1B', 22);
> 5) select * from hive.hive_bucketed2;
> *Workaround* (may be a little bit {color:#de350b}risky{color}:) :
> 1. Connect to Hive metastore database.
> [https://analyticsanvil.files.wordpress.com/2016/08/hive_metastore_database_diagram.png]
> 2. Find SDS linked to desired PARTITIONS . Actually you need CD_ID's for such
> SDS.
> 3. Insert your column into COLUMNS_V2 with CD_ID found at previous step.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (DRILL-7365) Failed to read column added to existing Hive partition
Igor Guzenko created DRILL-7365:
---
Summary: Failed to read column added to existing Hive partition
Key: DRILL-7365
URL: https://issues.apache.org/jira/browse/DRILL-7365
Project: Apache Drill
Issue Type: Bug
Components: Storage - Hive
Reporter: Igor Guzenko
Prerequisities:
Enable ACID in Hive
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions.
Steps to reproduce:
1) create table hive_bucketed2 (emp_id int, first_name string) PARTITIONED BY
(`col_year_month` string) clustered by (emp_id) into 4 buckets stored as orc
tblproperties ('transactional'='true');
2) insert into hive_bucketed2 PARTITION (col_year_month = '2019-09') values (1,
'A'),(2, 'B');
3) alter table hive_bucketed2 add columns (age INT);
4) insert into hive_bucketed2 PARTITION (col_year_month = '2019-09') values
(11, '1A', 10),(12, '1B', 22);
5) select * from hive.hive_bucketed2;
Workaround (may be a little bit risky:) :
1. Connect to Hive metastore database.
https://analyticsanvil.files.wordpress.com/2016/08/hive_metastore_database_diagram.png
2. Find SDS linked to desired PARTITIONS . Actually you need CD_ID's for such
SDS.
3. Insert your column into COLUMNS_V2 with CD_ID found at previous step.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
