[
https://issues.apache.org/jira/browse/BEAM-3773?focusedWorklogId=107724&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107724
]
ASF GitHub Bot logged work on BEAM-3773:
----------------------------------------
Author: ASF GitHub Bot
Created on: 31/May/18 16:06
Start Date: 31/May/18 16:06
Worklog Time Spent: 10m
Work Description: kennknowles closed pull request #5506: [BEAM-3773]
Build JDBC jar
URL: https://github.com/apache/beam/pull/5506
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/LICENSE b/LICENSE
index 4dda9e3e5dd..a2e93e2819c 100644
--- a/LICENSE
+++ b/LICENSE
@@ -298,4 +298,107 @@
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
- IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
\ No newline at end of file
+ IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ jline:
+ Copyright (c) 2002-2016, the original author or authors.
+ All rights reserved.
+
+ http://www.opensource.org/licenses/bsd-license.php
+
+ Redistribution and use in source and binary forms, with or
+ without modification, are permitted provided that the following
+ conditions are met:
+
+ Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+
+ Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer
+ in the documentation and/or other materials provided with
+ the distribution.
+
+ Neither the name of JLine nor the names of its contributors
+ may be used to endorse or promote products derived from this
+ software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
+ BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
+ AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
+ EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
+ FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
+ OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
+ IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+ OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ sqlline:
+ SQLLine - Shell for issuing SQL to relational databases via JDBC
+
+ Copyright (c) 2002,2003,2004,2005,2006,2007 Marc Prud'hommeaux
+ Copyright (c) 2004-2010 The Eigenbase Project
+ Copyright (c) 2013-2017 Julian Hyde
+ All rights reserved.
+
+
===============================================================================
+
+ Licensed under the Modified BSD License (the "License"); you may not
+ use this file except in compliance with the License. You may obtain a
+ copy of the License at:
+
+ http://opensource.org/licenses/BSD-3-Clause
+
+ Redistribution and use in source and binary forms,
+ with or without modification, are permitted provided
+ that the following conditions are met:
+
+ (1) Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+
+ (2) Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in the
+ documentation and/or other materials provided with the
+ distribution.
+
+ (3) The name of the author may not be used to endorse or promote
+ products derived from this software without specific prior written
+ permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ slf4j:
+ Copyright (c) 2004-2017 QOS.ch
+ All rights reserved.
+
+ Permission is hereby granted, free of charge, to any person obtaining
+ a copy of this software and associated documentation files (the
+ "Software"), to deal in the Software without restriction, including
+ without limitation the rights to use, copy, modify, merge, publish,
+ distribute, sublicense, and/or sell copies of the Software, and to
+ permit persons to whom the Software is furnished to do so, subject to
+ the following conditions:
+
+ The above copyright notice and this permission notice shall be
+ included in all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE
diff --git a/build_rules.gradle b/build_rules.gradle
index 6f38a84bca9..219afe58bc5 100644
--- a/build_rules.gradle
+++ b/build_rules.gradle
@@ -344,6 +344,7 @@ class JavaNatureConfiguration {
//TODO(https://issues.apache.org/jira/browse/BEAM-4394): Should this default
to true?
boolean enableSpotless = false // Controls whether spotless plugin enforces
autoformat
+ boolean testShadowJar = false // Controls whether tests are run with
shadowJar
// The shadowJar / shadowTestJar tasks execute the following closure to
configure themselves.
// Users can compose their closure with the default closure via:
@@ -581,6 +582,22 @@ ext.applyJavaNature = {
artifacts.shadow shadowJar
artifacts.shadowTest shadowTestJar
+ if (configuration.testShadowJar) {
+ // Use a configuration and dependency set which represents the execution
classpath using shaded artifacts for tests.
+ configurations {
+ shadowTestRuntimeClasspath
+ }
+
+ dependencies {
+ shadowTestRuntimeClasspath project(path: project.path, configuration:
"shadowTest")
+ shadowTestRuntimeClasspath project(path: project.path, configuration:
"provided")
+ }
+
+ test {
+ classpath = configurations.shadowTestRuntimeClasspath
+ }
+ }
+
if (isRelease() || project.hasProperty('publishing')) {
apply plugin: "maven-publish"
@@ -632,21 +649,6 @@ artifactId=${project.name}
}
artifacts.archives javadocJar
- // Use a configuration and dependency set which represents the execution
classpath using shaded artifacts for tests.
- configurations {
- shadowTestRuntimeClasspath
- }
-
- dependencies {
- shadowTestRuntimeClasspath project(path: project.path, configuration:
"shadowTest")
- shadowTestRuntimeClasspath project(path: project.path, configuration:
"provided")
- shadowTestRuntimeClasspath project(path: project.path, configuration:
"runtime")
- }
-
- test {
- classpath = configurations.shadowTestRuntimeClasspath
- }
-
// Only sign artifacts if we are performing a release
if (isRelease()) {
apply plugin: "signing"
diff --git a/sdks/java/extensions/sql/build.gradle
b/sdks/java/extensions/sql/build.gradle
index 44441442944..5cdf5fd74e2 100644
--- a/sdks/java/extensions/sql/build.gradle
+++ b/sdks/java/extensions/sql/build.gradle
@@ -19,7 +19,7 @@ import groovy.json.JsonOutput
*/
apply from: project(":").file("build_rules.gradle")
-applyJavaNature(enableSpotless: true, shadowClosure: DEFAULT_SHADOW_CLOSURE <<
{
+applyJavaNature(enableSpotless: true, testShadowJar: true, shadowClosure:
DEFAULT_SHADOW_CLOSURE << {
dependencies {
include(dependency(library.java.protobuf_java))
include(dependency(library.java.protobuf_java_util))
diff --git a/sdks/java/extensions/sql/jdbc/build.gradle
b/sdks/java/extensions/sql/jdbc/build.gradle
new file mode 100644
index 00000000000..cd9959c7ea9
--- /dev/null
+++ b/sdks/java/extensions/sql/jdbc/build.gradle
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply from: project(":").file("build_rules.gradle")
+applyJavaNature(failOnWarning: true, enableSpotless: true, testShadowJar:
true, shadowClosure: {})
+
+configurations {
+ integrationTest
+}
+
+dependencies {
+ compile project(path: ":beam-sdks-java-extensions-sql", configuration:
"shadow")
+ compile "jline:jline:2.14.6"
+ compile "sqlline:sqlline:1.3.0"
+ compile library.java.slf4j_jdk14
+ compileOnly library.java.findbugs_annotations
+ testCompileOnly library.java.findbugs_annotations
+ // Depending on outputs so integrationTest can run with only test
dependencies.
+ // This enables us to test the JDBC jar being loaded on a custom classloader.
+ integrationTest sourceSets.test.output
+ integrationTest sourceSets.main.output
+ integrationTest library.java.junit
+}
+
+shadowJar {
+ manifest {
+ attributes "Main-Class":
"org.apache.beam.sdk.extensions.sql.jdbc.BeamSqlLine"
+ }
+}
+
+test {
+ doFirst {
+ // Assert everything is in the output or test jar
+ assert classpath.size() == 2
+ }
+}
+
+task integrationTest(type: Test, dependsOn:
":beam-sdks-java-extensions-sql-jdbc:shadowJar") {
+ group = "Verification"
+
+ systemProperty "driver.jar", configurations.shadowTestRuntimeClasspath[1]
+
+ include '**/*IT.class'
+ classpath = configurations.integrationTest
+ useJUnit { }
+}
diff --git
a/sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
b/sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
new file mode 100644
index 00000000000..7952b0620c3
--- /dev/null
+++
b/sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.jdbc;
+
+import static
org.apache.beam.sdk.extensions.sql.impl.JdbcDriver.CONNECT_STRING_PREFIX;
+
+import java.io.IOException;
+import sqlline.SqlLine;
+
+/** {@link BeamSqlLine} provides default arguments to SqlLine. */
+public class BeamSqlLine {
+ public static void main(String[] args) throws IOException {
+ String[] args2 = new String[2 + args.length];
+ args2[0] = "-u";
+ args2[1] = CONNECT_STRING_PREFIX;
+ System.arraycopy(args, 0, args2, 2, args.length);
+
+ SqlLine.main(args2);
+ }
+}
diff --git
a/sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/package-info.java
b/sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/package-info.java
new file mode 100644
index 00000000000..ae3eaff16af
--- /dev/null
+++
b/sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/package-info.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Provides default commandline for sqlline. */
+package org.apache.beam.sdk.extensions.sql.jdbc;
diff --git
a/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
b/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
new file mode 100644
index 00000000000..cee3aa7142d
--- /dev/null
+++
b/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.jdbc;
+
+import org.junit.Test;
+
+/**
+ * Test for {@link org.apache.beam.sdk.extensions.sql.jdbc.BeamSqlLine}. Note
that this test only
+ * tests for crashes (due to ClassNotFoundException for example). It does not
test output.
+ */
+public class BeamSqlLineTest {
+
+ @Test
+ public void testSqlLine_emptyArgs() throws Exception {
+ BeamSqlLine.main(new String[] {});
+ }
+
+ @Test
+ public void testSqlLine_nullCommand() throws Exception {
+ BeamSqlLine.main(new String[] {"-e", ""});
+ }
+
+ @Test
+ public void testSqlLine_simple() throws Exception {
+ BeamSqlLine.main(new String[] {"-e", "SELECT 1;"});
+ }
+
+ @Test
+ public void testSqlLine_parse() throws Exception {
+ BeamSqlLine.main(new String[] {"-e", "SELECT 'beam';"});
+ }
+
+ @Test
+ public void testSqlLine_ddl() throws Exception {
+ BeamSqlLine.main(
+ new String[] {
+ "-e", "CREATE TABLE test (id INTEGER) TYPE 'text';", "-e", "DROP
TABLE test;"
+ });
+ }
+}
diff --git
a/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/JdbcIT.java
b/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/JdbcIT.java
new file mode 100644
index 00000000000..08cd9a0a4d6
--- /dev/null
+++
b/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/JdbcIT.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.jdbc;
+
+import static org.junit.Assert.assertTrue;
+
+import java.io.File;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.sql.Connection;
+import java.sql.Driver;
+import java.sql.Statement;
+import java.util.Properties;
+import java.util.ServiceLoader;
+import org.junit.Test;
+
+/** Test for JDBC via {@link java.net.URLClassLoader}. */
+public class JdbcIT {
+ private static final String DRIVER_URL = "jdbc:beam:";
+
+ private Connection getConnection() throws Exception {
+ URL jdbcUrl = new File(System.getProperty("driver.jar")).toURI().toURL();
+ ClassLoader classLoader =
+ new URLClassLoader(new URL[] {jdbcUrl},
ClassLoader.getSystemClassLoader());
+
+ ServiceLoader<Driver> loader = ServiceLoader.load(Driver.class,
classLoader);
+ Driver driver = loader.iterator().next();
+ return driver.connect(DRIVER_URL, new Properties());
+ }
+
+ @Test
+ public void classLoader_simple() throws Exception {
+ Connection connection = getConnection();
+ Statement statement = connection.createStatement();
+ // SELECT 1 is a special case and does not reach the parser
+ assertTrue(statement.execute("SELECT 1"));
+ }
+}
diff --git a/settings.gradle b/settings.gradle
index 670a513fbfb..0b89a609a33 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -84,6 +84,8 @@ include "beam-sdks-java-extensions-sorter"
project(":beam-sdks-java-extensions-sorter").dir =
file("sdks/java/extensions/sorter")
include "beam-sdks-java-extensions-sql"
project(":beam-sdks-java-extensions-sql").dir =
file("sdks/java/extensions/sql")
+include "beam-sdks-java-extensions-sql-jdbc"
+project(":beam-sdks-java-extensions-sql-jdbc").dir =
file("sdks/java/extensions/sql/jdbc")
include "beam-sdks-java-fn-execution"
project(":beam-sdks-java-fn-execution").dir = file("sdks/java/fn-execution")
include "beam-sdks-java-harness"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 107724)
Time Spent: 10h 40m (was: 10.5h)
> [SQL] Investigate JDBC interface for Beam SQL
> ---------------------------------------------
>
> Key: BEAM-3773
> URL: https://issues.apache.org/jira/browse/BEAM-3773
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
> Reporter: Anton Kedin
> Assignee: Andrew Pilloud
> Priority: Major
> Time Spent: 10h 40m
> Remaining Estimate: 0h
>
> JDBC allows integration with a lot of third-party tools, e.g
> [Zeppelin|https://zeppelin.apache.org/docs/0.7.0/manual/interpreters.html],
> [sqlline|https://github.com/julianhyde/sqlline]. We should look into how
> feasible it is to implement a JDBC interface for Beam SQL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
