[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=125121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125121
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 19/Jul/18 17:10
Start Date: 19/Jul/18 17:10
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #5969: [BEAM-4806][SQL]
improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/java/extensions/sql/jdbc/build.gradle
b/sdks/java/extensions/sql/jdbc/build.gradle
index f7dbb7cf66c..3a894cc01bd 100644
--- a/sdks/java/extensions/sql/jdbc/build.gradle
+++ b/sdks/java/extensions/sql/jdbc/build.gradle
@@ -30,6 +30,8 @@ dependencies {
compile library.java.slf4j_jdk14
compile library.java.guava
testCompile library.java.junit
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
// Depending on outputs so integrationTest can run with only test
dependencies.
// This enables us to test the JDBC jar being loaded on a custom classloader.
integrationTest sourceSets.test.output
diff --git
a/sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
b/sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
index c47ab4106fe..966253a27ae 100644
---
a/sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
+++
b/sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
@@ -19,12 +19,16 @@
import static
org.apache.beam.sdk.extensions.sql.impl.JdbcDriver.CONNECT_STRING_PREFIX;
-import com.google.common.collect.ImmutableList;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import javax.annotation.Nullable;
import sqlline.SqlLine;
+import sqlline.SqlLine.Status;
/** {@link BeamSqlLine} provides default arguments to SqlLine. */
public class BeamSqlLine {
@@ -32,28 +36,42 @@
private static final String NICKNAME = "BeamSQL";
public static void main(String[] args) throws IOException {
+runSqlLine(args, null, System.out, System.err);
+ }
+
+ private static String[] checkConnectionArgs(String[] args) {
+List argsList = new ArrayList(Arrays.asList(args));
-// Until we learn otherwise, we expect to add -nn -u
-@Nullable String databaseUrl = null;
-@Nullable String nickname = null;
-
-// Provide -u and -nn only if they do not exist
-for (int i = 0; i < args.length; i++) {
- if (args[i].equals("-u")) {
-databaseUrl = args[++i];
- } else if (args[i].equals("-nn")) {
-nickname = args[++i];
- }
+if (!argsList.contains("-nn")) {
+ argsList.add("-nn");
+ argsList.add(NICKNAME);
}
-ImmutableList.Builder wrappedArgs =
ImmutableList.builder().addAll(Arrays.asList(args));
-if (databaseUrl == null) {
- wrappedArgs.add("-u").add(CONNECT_STRING_PREFIX);
+if (!argsList.contains("-u")) {
+ argsList.add("-u");
+ argsList.add(CONNECT_STRING_PREFIX);
}
-if (nickname == null) {
- wrappedArgs.add("-nn").add(NICKNAME);
+
+return argsList.toArray(new String[argsList.size()]);
+ }
+
+ static Status runSqlLine(
+ String[] args,
+ InputStream inputStream,
+ @Nullable OutputStream outputStream,
+ @Nullable OutputStream errorStream)
+ throws IOException {
+String[] modifiedArgs = checkConnectionArgs(args);
+SqlLine sqlLine = new SqlLine();
+
+if (outputStream != null) {
+ sqlLine.setOutputStream(new PrintStream(outputStream));
+}
+
+if (errorStream != null) {
+ sqlLine.setErrorStream(new PrintStream(errorStream));
}
-List wrappedArgList = wrappedArgs.build();
-SqlLine.main(wrappedArgList.toArray(new String[wrappedArgList.size()]));
+
+return sqlLine.begin(modifiedArgs, inputStream, true);
}
}
diff --git
a/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
b/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
index 1ad19b4512e..f8ec21ce735 100644
---
a/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
+++
b/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
@@ -17,7 +17,16 @@
*/
package
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=125108=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125108 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 19/Jul/18 16:46 Start Date: 19/Jul/18 16:46 Worklog Time Spent: 10m Work Description: akedin commented on issue #5969: [BEAM-4806][SQL] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406342270 lgtm This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 125108) Time Spent: 5h 20m (was: 5h 10m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124767=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124767 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 19/Jul/18 03:15 Start Date: 19/Jul/18 03:15 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #5969: [BEAM-4806][SQL] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406141425 run java postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124767) Time Spent: 5h (was: 4h 50m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124768 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 19/Jul/18 03:15 Start Date: 19/Jul/18 03:15 Worklog Time Spent: 10m Work Description: amaliujia removed a comment on issue #5969: [BEAM-4806][SQL] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406123306 run java postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124768) Time Spent: 5h 10m (was: 5h) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124739 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 19/Jul/18 01:16 Start Date: 19/Jul/18 01:16 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #5969: [BEAM-4806][SQL] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406123306 run java postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124739) Time Spent: 4h 50m (was: 4h 40m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124679 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 22:48 Start Date: 18/Jul/18 22:48 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #5969: [BEAM-4806][SQL] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406097978 @apilloud FYI, I added the JdbcJarTest patch in this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124679) Time Spent: 4h 40m (was: 4.5h) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124677=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124677
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 22:47
Start Date: 18/Jul/18 22:47
Worklog Time Spent: 10m
Work Description: amaliujia commented on a change in pull request #5969:
[BEAM-4806][SQL] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203554858
##
File path:
sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
##
@@ -74,4 +89,143 @@ public void classLoader_readFile() throws Exception {
"DROP TABLE test;"
});
}
+
+ @Test
+ public void testSqlLine_select() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args = buildArgs("SELECT 3, 'hello', DATE '2018-05-28';");
+
+BeamSqlLine.runSqlLine(args, null, new PrintStream(byteArrayOutputStream),
null);
+
+List> lines = toLines(byteArrayOutputStream);
+assertThat(
+Arrays.asList(Arrays.asList("", "3", "hello", "2018-05-28")),
+everyItem(IsIn.isOneOf(lines.toArray(;
+ }
+
+ @Test
+ public void testSqlLine_selectFromTable() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args =
+buildArgs(
+"CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR, "
++ "col_c VARCHAR, col_x TINYINT, col_y INT, col_z BIGINT) TYPE
'test';",
+"INSERT INTO table_test VALUES ('a', 'b', 'c', 1, 2, 3);",
+"SELECT * FROM table_test;");
+
+BeamSqlLine.runSqlLine(args, null, new PrintStream(byteArrayOutputStream),
null);
+
+List> lines = toLines(byteArrayOutputStream);
+assertThat(
+Arrays.asList(
+Arrays.asList("", "col_a", "col_b", "col_c", "col_x", "col_y",
"col_z"),
+Arrays.asList("", "a", "b", "c", "1", "2", "3")),
+everyItem(IsIn.isOneOf(lines.toArray(;
+ }
+
+ @Test
+ public void testSqlLine_insertSelect() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args =
+buildArgs(
+"CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE
'test';",
+"INSERT INTO table_test SELECT '3', 'hello';",
+"SELECT * FROM table_test;");
+
+BeamSqlLine.runSqlLine(args, null, new PrintStream(byteArrayOutputStream),
null);
+
+List> lines = toLines(byteArrayOutputStream);
+assertThat(
+Arrays.asList(Arrays.asList("", "3", "hello")),
everyItem(IsIn.isOneOf(lines.toArray(;
+ }
+
+ @Test
+ public void testSqlLine_GroupBy() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args =
+buildArgs(
+"CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE
'test';",
+"INSERT INTO table_test SELECT '3', 'foo';",
+"INSERT INTO table_test SELECT '3', 'bar';",
+"INSERT INTO table_test SELECT '4', 'foo';",
+"SELECT col_a, count(*) FROM table_test GROUP BY col_a;");
+
+BeamSqlLine.runSqlLine(args, null, new PrintStream(byteArrayOutputStream),
null);
+
+List> lines = toLines(byteArrayOutputStream);
+assertThat(
+Arrays.asList(Arrays.asList("", "3", "2"), Arrays.asList("", "4",
"1")),
+everyItem(IsIn.isOneOf(lines.toArray(;
+ }
+
+ @Test
+ public void testSqlLine_fixedWindow() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args =
+buildArgs(
+"CREATE TABLE table_test (col_a VARCHAR, col_b TIMESTAMP) TYPE
'test';",
+"INSERT INTO table_test SELECT '3', TIMESTAMP '2018-07-01
21:26:06';",
+"INSERT INTO table_test SELECT '3', TIMESTAMP '2018-07-01
21:26:07';",
+"SELECT TUMBLE_START(col_b, INTERVAL '1' SECOND), count(*) FROM
table_test "
++ "GROUP BY TUMBLE(col_b, INTERVAL '1' SECOND);");
+
+BeamSqlLine.runSqlLine(args, null, new PrintStream(byteArrayOutputStream),
null);
+
+List> lines = toLines(byteArrayOutputStream);
+assertThat(
+Arrays.asList(
+Arrays.asList("", "2018-07-01 21:26:06", "1"),
+Arrays.asList("", "2018-07-01 21:26:07", "1")),
+everyItem(IsIn.isOneOf(lines.toArray(;
+ }
+
+ @Test
+ public void testSqlLine_slidingWindow() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args =
+buildArgs(
+"CREATE TABLE table_test (col_a VARCHAR, col_b TIMESTAMP) TYPE
'test';",
+"INSERT INTO table_test SELECT
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124678=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124678
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 22:47
Start Date: 18/Jul/18 22:47
Worklog Time Spent: 10m
Work Description: amaliujia commented on a change in pull request #5969:
[BEAM-4806][SQL] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203554949
##
File path:
sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
##
@@ -19,41 +19,57 @@
import static
org.apache.beam.sdk.extensions.sql.impl.JdbcDriver.CONNECT_STRING_PREFIX;
-import com.google.common.collect.ImmutableList;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintStream;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import javax.annotation.Nullable;
import sqlline.SqlLine;
+import sqlline.SqlLine.Status;
/** {@link BeamSqlLine} provides default arguments to SqlLine. */
public class BeamSqlLine {
private static final String NICKNAME = "BeamSQL";
public static void main(String[] args) throws IOException {
+runSqlLine(checkConnectionArgs(args), null, null, null);
+ }
+
+ private static String[] checkConnectionArgs(String[] args) {
+List argsList = new ArrayList(Arrays.asList(args));
-// Until we learn otherwise, we expect to add -nn -u
-@Nullable String databaseUrl = null;
-@Nullable String nickname = null;
-
-// Provide -u and -nn only if they do not exist
-for (int i = 0; i < args.length; i++) {
- if (args[i].equals("-u")) {
-databaseUrl = args[++i];
- } else if (args[i].equals("-nn")) {
-nickname = args[++i];
- }
+if (!argsList.contains("-nn")) {
+ argsList.add("-nn");
+ argsList.add(NICKNAME);
}
-ImmutableList.Builder wrappedArgs =
ImmutableList.builder().addAll(Arrays.asList(args));
-if (databaseUrl == null) {
- wrappedArgs.add("-u").add(CONNECT_STRING_PREFIX);
+if (!argsList.contains("-u")) {
+ argsList.add("-u");
+ argsList.add(CONNECT_STRING_PREFIX);
}
-if (nickname == null) {
- wrappedArgs.add("-nn").add(NICKNAME);
+
+return argsList.toArray(new String[argsList.size()]);
+ }
+
+ static Status runSqlLine(
+ String[] args,
+ InputStream inputStream,
+ @Nullable PrintStream outputStream,
+ @Nullable PrintStream errorStream)
Review comment:
Use `OutputStream` now. This is a good idea to simplify tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124678)
Time Spent: 4.5h (was: 4h 20m)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124676
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 22:47
Start Date: 18/Jul/18 22:47
Worklog Time Spent: 10m
Work Description: amaliujia commented on a change in pull request #5969:
[BEAM-4806][SQL] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203554776
##
File path:
sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
##
@@ -53,7 +58,14 @@ public static void main(String[] args) throws IOException {
if (nickname == null) {
wrappedArgs.add("-nn").add(NICKNAME);
}
-List wrappedArgList = wrappedArgs.build();
-SqlLine.main(wrappedArgList.toArray(new String[wrappedArgList.size()]));
+
+return wrappedArgs.build();
+ }
+
+ public static void testMain(String[] args, PrintStream outputStream) throws
IOException {
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124676)
Time Spent: 4h 10m (was: 4h)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124668=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124668
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 22:27
Start Date: 18/Jul/18 22:27
Worklog Time Spent: 10m
Work Description: amaliujia commented on a change in pull request #5969:
[BEAM-4806][SQL] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203549851
##
File path:
sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
##
@@ -19,41 +19,57 @@
import static
org.apache.beam.sdk.extensions.sql.impl.JdbcDriver.CONNECT_STRING_PREFIX;
-import com.google.common.collect.ImmutableList;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintStream;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import javax.annotation.Nullable;
import sqlline.SqlLine;
+import sqlline.SqlLine.Status;
/** {@link BeamSqlLine} provides default arguments to SqlLine. */
public class BeamSqlLine {
private static final String NICKNAME = "BeamSQL";
public static void main(String[] args) throws IOException {
+runSqlLine(checkConnectionArgs(args), null, null, null);
Review comment:
Great suggestion. Putting `checkConnectionArgs()` in `runSqlLine()` also
simplify tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124668)
Time Spent: 4h (was: 3h 50m)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124663 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 22:23 Start Date: 18/Jul/18 22:23 Worklog Time Spent: 10m Work Description: amaliujia commented on a change in pull request #5969: [BEAM-4806][SQL] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#discussion_r203549018 ## File path: sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java ## @@ -30,6 +30,7 @@ import org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl; import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils; import org.apache.beam.sdk.extensions.sql.meta.Table; +import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider; Review comment: Did a check and seems like here is a conflict of using TestTableProvider but creating a `Text` table. This is a slightly refactoring of old code but not relevant to this PR. I prefer to fix this conflict in a separate PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124663) Time Spent: 3h 40m (was: 3.5h) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124665 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 22:23 Start Date: 18/Jul/18 22:23 Worklog Time Spent: 10m Work Description: amaliujia commented on a change in pull request #5969: [BEAM-4806][SQL] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#discussion_r203549018 ## File path: sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java ## @@ -30,6 +30,7 @@ import org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl; import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils; import org.apache.beam.sdk.extensions.sql.meta.Table; +import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider; Review comment: Did a check and seems like here is a conflict of using `TestTableProvider` but creating a `Text` table. This is a slightly refactoring of old code but not relevant to this PR. I prefer to fix this conflict in a separate PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124665) Time Spent: 3h 50m (was: 3h 40m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124657=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124657
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 22:11
Start Date: 18/Jul/18 22:11
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806][SQL] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203546370
##
File path:
sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
##
@@ -19,41 +19,57 @@
import static
org.apache.beam.sdk.extensions.sql.impl.JdbcDriver.CONNECT_STRING_PREFIX;
-import com.google.common.collect.ImmutableList;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintStream;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import javax.annotation.Nullable;
import sqlline.SqlLine;
+import sqlline.SqlLine.Status;
/** {@link BeamSqlLine} provides default arguments to SqlLine. */
public class BeamSqlLine {
private static final String NICKNAME = "BeamSQL";
public static void main(String[] args) throws IOException {
+runSqlLine(checkConnectionArgs(args), null, null, null);
Review comment:
i think it's better to call `checkConnectionArgs()` within `runSqlLine()`,
it's basically the only thing that happens in this class, we probably should
exercise it
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124657)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124655 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 22:11 Start Date: 18/Jul/18 22:11 Worklog Time Spent: 10m Work Description: akedin commented on a change in pull request #5969: [BEAM-4806][SQL] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#discussion_r203537090 ## File path: sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java ## @@ -30,6 +30,7 @@ import org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl; import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils; import org.apache.beam.sdk.extensions.sql.meta.Table; +import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider; Review comment: why these imports are needed? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124655) Time Spent: 3h 10m (was: 3h) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124656=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124656
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 22:11
Start Date: 18/Jul/18 22:11
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806][SQL] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203544612
##
File path:
sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
##
@@ -74,4 +89,143 @@ public void classLoader_readFile() throws Exception {
"DROP TABLE test;"
});
}
+
+ @Test
+ public void testSqlLine_select() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args = buildArgs("SELECT 3, 'hello', DATE '2018-05-28';");
+
+BeamSqlLine.runSqlLine(args, null, new PrintStream(byteArrayOutputStream),
null);
+
+List> lines = toLines(byteArrayOutputStream);
+assertThat(
+Arrays.asList(Arrays.asList("", "3", "hello", "2018-05-28")),
+everyItem(IsIn.isOneOf(lines.toArray(;
+ }
+
+ @Test
+ public void testSqlLine_selectFromTable() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args =
+buildArgs(
+"CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR, "
++ "col_c VARCHAR, col_x TINYINT, col_y INT, col_z BIGINT) TYPE
'test';",
+"INSERT INTO table_test VALUES ('a', 'b', 'c', 1, 2, 3);",
+"SELECT * FROM table_test;");
+
+BeamSqlLine.runSqlLine(args, null, new PrintStream(byteArrayOutputStream),
null);
+
+List> lines = toLines(byteArrayOutputStream);
+assertThat(
+Arrays.asList(
+Arrays.asList("", "col_a", "col_b", "col_c", "col_x", "col_y",
"col_z"),
+Arrays.asList("", "a", "b", "c", "1", "2", "3")),
+everyItem(IsIn.isOneOf(lines.toArray(;
+ }
+
+ @Test
+ public void testSqlLine_insertSelect() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args =
+buildArgs(
+"CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE
'test';",
+"INSERT INTO table_test SELECT '3', 'hello';",
+"SELECT * FROM table_test;");
+
+BeamSqlLine.runSqlLine(args, null, new PrintStream(byteArrayOutputStream),
null);
+
+List> lines = toLines(byteArrayOutputStream);
+assertThat(
+Arrays.asList(Arrays.asList("", "3", "hello")),
everyItem(IsIn.isOneOf(lines.toArray(;
+ }
+
+ @Test
+ public void testSqlLine_GroupBy() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args =
+buildArgs(
+"CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE
'test';",
+"INSERT INTO table_test SELECT '3', 'foo';",
+"INSERT INTO table_test SELECT '3', 'bar';",
+"INSERT INTO table_test SELECT '4', 'foo';",
+"SELECT col_a, count(*) FROM table_test GROUP BY col_a;");
+
+BeamSqlLine.runSqlLine(args, null, new PrintStream(byteArrayOutputStream),
null);
+
+List> lines = toLines(byteArrayOutputStream);
+assertThat(
+Arrays.asList(Arrays.asList("", "3", "2"), Arrays.asList("", "4",
"1")),
+everyItem(IsIn.isOneOf(lines.toArray(;
+ }
+
+ @Test
+ public void testSqlLine_fixedWindow() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args =
+buildArgs(
+"CREATE TABLE table_test (col_a VARCHAR, col_b TIMESTAMP) TYPE
'test';",
+"INSERT INTO table_test SELECT '3', TIMESTAMP '2018-07-01
21:26:06';",
+"INSERT INTO table_test SELECT '3', TIMESTAMP '2018-07-01
21:26:07';",
+"SELECT TUMBLE_START(col_b, INTERVAL '1' SECOND), count(*) FROM
table_test "
++ "GROUP BY TUMBLE(col_b, INTERVAL '1' SECOND);");
+
+BeamSqlLine.runSqlLine(args, null, new PrintStream(byteArrayOutputStream),
null);
+
+List> lines = toLines(byteArrayOutputStream);
+assertThat(
+Arrays.asList(
+Arrays.asList("", "2018-07-01 21:26:06", "1"),
+Arrays.asList("", "2018-07-01 21:26:07", "1")),
+everyItem(IsIn.isOneOf(lines.toArray(;
+ }
+
+ @Test
+ public void testSqlLine_slidingWindow() throws Exception {
+ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
+String[] args =
+buildArgs(
+"CREATE TABLE table_test (col_a VARCHAR, col_b TIMESTAMP) TYPE
'test';",
+"INSERT INTO table_test SELECT '3',
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124658 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 22:11 Start Date: 18/Jul/18 22:11 Worklog Time Spent: 10m Work Description: akedin commented on a change in pull request #5969: [BEAM-4806][SQL] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#discussion_r203536893 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/package-info.java ## @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** Table schema for text files. */ Review comment: not for text files, for in-memory test data? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124658) Time Spent: 3h 20m (was: 3h 10m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124659
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 22:11
Start Date: 18/Jul/18 22:11
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806][SQL] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203545649
##
File path:
sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
##
@@ -19,41 +19,57 @@
import static
org.apache.beam.sdk.extensions.sql.impl.JdbcDriver.CONNECT_STRING_PREFIX;
-import com.google.common.collect.ImmutableList;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintStream;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import javax.annotation.Nullable;
import sqlline.SqlLine;
+import sqlline.SqlLine.Status;
/** {@link BeamSqlLine} provides default arguments to SqlLine. */
public class BeamSqlLine {
private static final String NICKNAME = "BeamSQL";
public static void main(String[] args) throws IOException {
+runSqlLine(checkConnectionArgs(args), null, null, null);
+ }
+
+ private static String[] checkConnectionArgs(String[] args) {
+List argsList = new ArrayList(Arrays.asList(args));
-// Until we learn otherwise, we expect to add -nn -u
-@Nullable String databaseUrl = null;
-@Nullable String nickname = null;
-
-// Provide -u and -nn only if they do not exist
-for (int i = 0; i < args.length; i++) {
- if (args[i].equals("-u")) {
-databaseUrl = args[++i];
- } else if (args[i].equals("-nn")) {
-nickname = args[++i];
- }
+if (!argsList.contains("-nn")) {
+ argsList.add("-nn");
+ argsList.add(NICKNAME);
}
-ImmutableList.Builder wrappedArgs =
ImmutableList.builder().addAll(Arrays.asList(args));
-if (databaseUrl == null) {
- wrappedArgs.add("-u").add(CONNECT_STRING_PREFIX);
+if (!argsList.contains("-u")) {
+ argsList.add("-u");
+ argsList.add(CONNECT_STRING_PREFIX);
}
-if (nickname == null) {
- wrappedArgs.add("-nn").add(NICKNAME);
+
+return argsList.toArray(new String[argsList.size()]);
+ }
+
+ static Status runSqlLine(
+ String[] args,
+ InputStream inputStream,
+ @Nullable PrintStream outputStream,
+ @Nullable PrintStream errorStream)
Review comment:
i think it makes sense to accept `OutputStream` instead of `PrintStream`,
every consumer of this method will need to wrap it. I think it's better to have
one place in `main` to specify `System.out` etc than have to create
`PrintStream` in every test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124659)
Time Spent: 3.5h (was: 3h 20m)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124619=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124619 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 21:14 Start Date: 18/Jul/18 21:14 Worklog Time Spent: 10m Work Description: amaliujia removed a comment on issue #5969: [BEAM-4806] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-405856734 run java postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124619) Time Spent: 3h (was: 2h 50m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124618 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 21:14 Start Date: 18/Jul/18 21:14 Worklog Time Spent: 10m Work Description: amaliujia removed a comment on issue #5969: [BEAM-4806] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406020991 run java postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124618) Time Spent: 2h 50m (was: 2h 40m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124532=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124532 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 17:59 Start Date: 18/Jul/18 17:59 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #5969: [BEAM-4806] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406020991 run java postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124532) Time Spent: 2h 40m (was: 2.5h) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124529=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124529 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 17:58 Start Date: 18/Jul/18 17:58 Worklog Time Spent: 10m Work Description: amaliujia removed a comment on issue #5969: [BEAM-4806] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406018394 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124529) Time Spent: 2h 20m (was: 2h 10m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124528=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124528 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 17:58 Start Date: 18/Jul/18 17:58 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #5969: [BEAM-4806] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406020512 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124528) Time Spent: 2h 10m (was: 2h) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124530 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 17:58 Start Date: 18/Jul/18 17:58 Worklog Time Spent: 10m Work Description: amaliujia removed a comment on issue #5969: [BEAM-4806] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406020512 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124530) Time Spent: 2.5h (was: 2h 20m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124524=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124524 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 17:51 Start Date: 18/Jul/18 17:51 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #5969: [BEAM-4806] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-406018394 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124524) Time Spent: 2h (was: 1h 50m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124359=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124359 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 08:43 Start Date: 18/Jul/18 08:43 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #5969: [BEAM-4806] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-405856734 run java postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124359) Time Spent: 1h 50m (was: 1h 40m) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[ https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124357=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124357 ] ASF GitHub Bot logged work on BEAM-4806: Author: ASF GitHub Bot Created on: 18/Jul/18 08:40 Start Date: 18/Jul/18 08:40 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #5969: [BEAM-4806] improve unittest for BeamSqlLine URL: https://github.com/apache/beam/pull/5969#issuecomment-405856064 @akedin Thanks for the great comments! I refactored the code a lot. Can you take a another look? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 124357) Time Spent: 1h 40m (was: 1.5h) > Improve BeamSqlLine unit tests > -- > > Key: BEAM-4806 > URL: https://issues.apache.org/jira/browse/BEAM-4806 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124338
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 06:24
Start Date: 18/Jul/18 06:24
Worklog Time Spent: 10m
Work Description: amaliujia commented on a change in pull request #5969:
[BEAM-4806] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203262754
##
File path:
sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
##
@@ -74,4 +84,128 @@ public void classLoader_readFile() throws Exception {
"DROP TABLE test;"
});
}
+
+ @Test
+ public void testSqlLine_select() throws Exception {
+BeamSqlLine.testMain(
+new String[] {"-e", "SELECT 3, 'hello', DATE '2018-05-28';"},
+new PrintStream(byteArrayOutputStream));
+
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "3", "hello",
"2018-05-28");
+verifyStringOutputFalse(byteArrayOutputStream.toString(), "4", "hella",
"2018-09-09");
+ }
+
+ @Test
+ public void testSqlLine_selectFromTable() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR, col_c
VARCHAR, col_x TINYINT, col_y INT, col_z BIGINT) TYPE 'test';",
+ "-e",
+ "INSERT INTO table_test VALUES ('a', 'b', 'c', 1, 2, 3);",
+ "-e",
+ "SELECT * FROM table_test;"
+},
+new PrintStream(byteArrayOutputStream));
+
+verifyStringOutputTrue(
+byteArrayOutputStream.toString(), "col_a", "col_b", "col_c", "col_x",
"col_y", "col_z");
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "a", "b", "c",
"1", "2", "3");
+ }
+
+ @Test
+ public void testSqlLine_insertSelect() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE
'test';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', 'hello';",
+ "-e",
+ "SELECT * FROM table_test;"
+},
+new PrintStream(byteArrayOutputStream));
+
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "3", "hello");
+ }
+
+ @Test
+ public void testSqlLine_GroupBy() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE
'test';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', 'foo';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', 'bar';",
+ "-e",
+ "INSERT INTO table_test SELECT '4', 'foo';",
Review comment:
Seems like it does not work if queries are combined.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124338)
Time Spent: 1.5h (was: 1h 20m)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124308
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 03:26
Start Date: 18/Jul/18 03:26
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203237624
##
File path:
sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
##
@@ -53,7 +58,14 @@ public static void main(String[] args) throws IOException {
if (nickname == null) {
wrappedArgs.add("-nn").add(NICKNAME);
}
-List wrappedArgList = wrappedArgs.build();
-SqlLine.main(wrappedArgList.toArray(new String[wrappedArgList.size()]));
+
+return wrappedArgs.build();
+ }
+
+ public static void testMain(String[] args, PrintStream outputStream) throws
IOException {
+List wrappedArgList = mainUtil(args);
+SqlLine sqlLine = new SqlLine();
+sqlLine.setOutputStream(outputStream);
+sqlLine.begin(wrappedArgList.toArray(new String[wrappedArgList.size()]),
null, true);
Review comment:
You expose 2 independent ways to start `sqlline`. Breaking `testMain` might
not break `main`, so there doesn't seem to be a lot of value here. I think
it's better to combine the paths:
* Create a package-private method to create and start `sqlline` instance
from args:
```java
static Status runSqlLine(
String[] rawArgs,
@Nullable InputStream in,
@Nullable OutputStream out,
@Nullable OutputStream err) {
String[] args = ensureConnectionArgs(rawArgs);
SqlLine sqlLine = new SqlLine();
if (out != null) {
sqlLine.setOutputStream(new PrintStream(out));
}
if (err != null) {
sqlLine.setErrorStream(new PrintStream(err));
}
return sqlLine.begin(args, in, true);
}
```
* `main` then only calls this method
```java
public static void main(String[] args) {
runSqlLine(args, null, System.out, System.err);
}
```
* .This way you can test exactly same functionality while injecting
fakes/mocks, which is cleaner then creating an unrelated public interface just
for testing:
```java
String [] EMPTY_ARGS = new String[] {};
InputStream NULL_INPUT = null;
@Test
void testRunsMain() {
OutputStream mainOutput = createOutputStream();
BeamSqlLine.runSqlLine(EMPTY_ARGS, NULL_INPUT, mainOutput);
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124308)
Time Spent: 1h 20m (was: 1h 10m)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124303=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124303
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 03:25
Start Date: 18/Jul/18 03:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203238008
##
File path:
sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
##
@@ -29,6 +33,12 @@
public class BeamSqlLineTest {
@Rule public TemporaryFolder folder = new TemporaryFolder();
+ public ByteArrayOutputStream byteArrayOutputStream;
+
+ @Before
+ public void setUp() {
+byteArrayOutputStream = new ByteArrayOutputStream();
Review comment:
It is a better practice to create everything needed within the test
(wrapping complex initialization in utility methods), instead of keeping state
in class fields. This way it's easier to understand what happens and reason
about it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124303)
Time Spent: 40m (was: 0.5h)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124302=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124302
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 03:25
Start Date: 18/Jul/18 03:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203214140
##
File path:
sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
##
@@ -53,7 +58,14 @@ public static void main(String[] args) throws IOException {
if (nickname == null) {
wrappedArgs.add("-nn").add(NICKNAME);
}
-List wrappedArgList = wrappedArgs.build();
-SqlLine.main(wrappedArgList.toArray(new String[wrappedArgList.size()]));
+
+return wrappedArgs.build();
+ }
+
+ public static void testMain(String[] args, PrintStream outputStream) throws
IOException {
Review comment:
This should not be public. At most this should be package-private.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124302)
Time Spent: 40m (was: 0.5h)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124304=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124304
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 03:25
Start Date: 18/Jul/18 03:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203217244
##
File path:
sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
##
@@ -32,7 +33,11 @@
private static final String NICKNAME = "BeamSQL";
public static void main(String[] args) throws IOException {
+List wrappedArgList = mainUtil(args);
+SqlLine.main(wrappedArgList.toArray(new String[wrappedArgList.size()]));
+ }
+ private static List mainUtil(String[] args) {
Review comment:
I think `mainUtil()` is a bad name for this. I'd call it something like
`ensureConnectionArgs()` to reflect what it does.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124304)
Time Spent: 50m (was: 40m)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124305=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124305
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 03:25
Start Date: 18/Jul/18 03:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203241230
##
File path:
sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
##
@@ -74,4 +84,128 @@ public void classLoader_readFile() throws Exception {
"DROP TABLE test;"
});
}
+
+ @Test
+ public void testSqlLine_select() throws Exception {
+BeamSqlLine.testMain(
+new String[] {"-e", "SELECT 3, 'hello', DATE '2018-05-28';"},
+new PrintStream(byteArrayOutputStream));
+
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "3", "hello",
"2018-05-28");
+verifyStringOutputFalse(byteArrayOutputStream.toString(), "4", "hella",
"2018-09-09");
+ }
+
+ @Test
+ public void testSqlLine_selectFromTable() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR, col_c
VARCHAR, col_x TINYINT, col_y INT, col_z BIGINT) TYPE 'test';",
+ "-e",
+ "INSERT INTO table_test VALUES ('a', 'b', 'c', 1, 2, 3);",
+ "-e",
+ "SELECT * FROM table_test;"
+},
+new PrintStream(byteArrayOutputStream));
+
+verifyStringOutputTrue(
+byteArrayOutputStream.toString(), "col_a", "col_b", "col_c", "col_x",
"col_y", "col_z");
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "a", "b", "c",
"1", "2", "3");
+ }
+
+ @Test
+ public void testSqlLine_insertSelect() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE
'test';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', 'hello';",
+ "-e",
+ "SELECT * FROM table_test;"
+},
+new PrintStream(byteArrayOutputStream));
+
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "3", "hello");
+ }
+
+ @Test
+ public void testSqlLine_GroupBy() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE
'test';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', 'foo';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', 'bar';",
+ "-e",
+ "INSERT INTO table_test SELECT '4', 'foo';",
+ "-e",
+ "SELECT col_a, count(*) FROM table_test GROUP BY col_a;"
+},
+new PrintStream(byteArrayOutputStream));
+
+// col_a 3 has count(*) = 2
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "3", "2");
+// col_a 4 has count(*) = 1
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "4", "1");
+ }
+
+ @Test
+ public void testSqlLine_fixedWindow() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b TIMESTAMP) TYPE
'test';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', TIMESTAMP '2018-07-01
21:26:06';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', TIMESTAMP '2018-07-01
21:26:07';",
+ "-e",
+ "SELECT TUMBLE_START(col_b, INTERVAL '1' SECOND), count(*) FROM
table_test GROUP BY TUMBLE(col_b, INTERVAL '1' SECOND);"
+},
+new PrintStream(byteArrayOutputStream));
+
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "2018-07-01
21:26:06", "1");
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "2018-07-01
21:26:07", "1");
+ }
+
+ @Test
+ public void testSqlLine_slidingWindow() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b TIMESTAMP) TYPE
'test';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', TIMESTAMP '2018-07-01
21:26:06';",
+ "-e",
+ "INSERT INTO table_test SELECT '4', TIMESTAMP '2018-07-01
21:26:07';",
+ "-e",
+ "INSERT INTO table_test SELECT '6', TIMESTAMP '2018-07-01
21:26:08';",
+ "-e",
+ "INSERT INTO table_test SELECT '7', TIMESTAMP '2018-07-01
21:26:09';",
+ "-e",
+ "SELECT HOP_END(col_b, INTERVAL '1' SECOND, INTERVAL '2' SECOND),
count(*) FROM table_test GROUP BY HOP(col_b, INTERVAL '1' SECOND, INTERVAL '2'
SECOND);"
+},
+new PrintStream(byteArrayOutputStream));
+
+
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124307
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 03:25
Start Date: 18/Jul/18 03:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203232220
##
File path:
sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
##
@@ -53,7 +58,14 @@ public static void main(String[] args) throws IOException {
if (nickname == null) {
wrappedArgs.add("-nn").add(NICKNAME);
}
Review comment:
I think this method can be simplified a bit:
```java
public String[] ensureConnectionArgs(String[] rawArgs) {
List argsList = Arrays.asList(rawArgs);
if (!argsList.contains("-nn") {
argsList.add("-nn");
argsList.add(NICKNAME);
}
if (!argsList.contains("-u") {
argsList.add("-u");
argsList.add(CONNECT_STRING_PREFIX);
}
return argsList.toArray(new String[argsList.size()]);
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124307)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124301
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 03:25
Start Date: 18/Jul/18 03:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203238298
##
File path:
sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
##
@@ -74,4 +84,128 @@ public void classLoader_readFile() throws Exception {
"DROP TABLE test;"
});
}
+
+ @Test
+ public void testSqlLine_select() throws Exception {
+BeamSqlLine.testMain(
+new String[] {"-e", "SELECT 3, 'hello', DATE '2018-05-28';"},
+new PrintStream(byteArrayOutputStream));
+
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "3", "hello",
"2018-05-28");
+verifyStringOutputFalse(byteArrayOutputStream.toString(), "4", "hella",
"2018-09-09");
+ }
+
+ @Test
+ public void testSqlLine_selectFromTable() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR, col_c
VARCHAR, col_x TINYINT, col_y INT, col_z BIGINT) TYPE 'test';",
+ "-e",
+ "INSERT INTO table_test VALUES ('a', 'b', 'c', 1, 2, 3);",
+ "-e",
+ "SELECT * FROM table_test;"
+},
+new PrintStream(byteArrayOutputStream));
+
+verifyStringOutputTrue(
+byteArrayOutputStream.toString(), "col_a", "col_b", "col_c", "col_x",
"col_y", "col_z");
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "a", "b", "c",
"1", "2", "3");
+ }
+
+ @Test
+ public void testSqlLine_insertSelect() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE
'test';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', 'hello';",
+ "-e",
+ "SELECT * FROM table_test;"
+},
+new PrintStream(byteArrayOutputStream));
+
+verifyStringOutputTrue(byteArrayOutputStream.toString(), "3", "hello");
+ }
+
+ @Test
+ public void testSqlLine_GroupBy() throws Exception {
+BeamSqlLine.testMain(
+new String[] {
+ "-e",
+ "CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE
'test';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', 'foo';",
+ "-e",
+ "INSERT INTO table_test SELECT '3', 'bar';",
+ "-e",
+ "INSERT INTO table_test SELECT '4', 'foo';",
Review comment:
Can we combine queries with `;`? E.g.:
```java
BeamSqlLine.testMain(
new String[] {
"-e",
"CREATE TABLE table_test (col_a VARCHAR, col_b VARCHAR) TYPE 'test'; \n"
+ "INSERT INTO table_test SELECT '3', 'foo'; \n"
+ "INSERT INTO table_test SELECT '3', 'bar'; \n"
+ "SELECT col_a, count(*) FROM table_test GROUP BY col_a;" });
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124301)
Time Spent: 0.5h (was: 20m)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4806) Improve BeamSqlLine unit tests
[
https://issues.apache.org/jira/browse/BEAM-4806?focusedWorklogId=124306=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124306
]
ASF GitHub Bot logged work on BEAM-4806:
Author: ASF GitHub Bot
Created on: 18/Jul/18 03:25
Start Date: 18/Jul/18 03:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5969:
[BEAM-4806] improve unittest for BeamSqlLine
URL: https://github.com/apache/beam/pull/5969#discussion_r203237624
##
File path:
sdks/java/extensions/sql/jdbc/src/main/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java
##
@@ -53,7 +58,14 @@ public static void main(String[] args) throws IOException {
if (nickname == null) {
wrappedArgs.add("-nn").add(NICKNAME);
}
-List wrappedArgList = wrappedArgs.build();
-SqlLine.main(wrappedArgList.toArray(new String[wrappedArgList.size()]));
+
+return wrappedArgs.build();
+ }
+
+ public static void testMain(String[] args, PrintStream outputStream) throws
IOException {
+List wrappedArgList = mainUtil(args);
+SqlLine sqlLine = new SqlLine();
+sqlLine.setOutputStream(outputStream);
+sqlLine.begin(wrappedArgList.toArray(new String[wrappedArgList.size()]),
null, true);
Review comment:
This way you expose 2 independent ways to start `sqlline`. Breaking
`testMain` might not break `main`, so there doesn't seem to be a lot of value
here. I think it's better to combine the paths:
* Create a package-private method to create and start `sqlline` instance
from args:
```java
static Status runSqlLine(
String[] rawArgs,
@Nullable InputStream in,
@Nullable OutputStream out,
@Nullable OutputStream err) {
String[] args = ensureConnectionArgs(rawArgs);
SqlLine sqlLine = new SqlLine();
if (out != null) {
sqlLine.setOutputStream(new PrintStream(out));
}
if (err != null) {
sqlLine.setErrorStream(new PrintStream(err));
}
return sqlLine.begin(args, in, true);
}
```
* `main` then only calls this method
```java
public static void main(String[] args) {
runSqlLine(args, null, System.out, System.err);
}
```
* .This way you can test exactly same functionality while injecting
fakes/mocks, which is cleaner then creating an unrelated public interface just
for testing:
```java
String [] EMPTY_ARGS = new String[] {};
InputStream NULL_INPUT = null;
@Test
void testRunsMain() {
OutputStream mainOutput = createOutputStream();
BeamSqlLine.runSqlLine(EMPTY_ARGS, NULL_INPUT, mainOutput);
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 124306)
Time Spent: 1h 10m (was: 1h)
> Improve BeamSqlLine unit tests
> --
>
> Key: BEAM-4806
> URL: https://issues.apache.org/jira/browse/BEAM-4806
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
