[jira] [Commented] (CALCITE-1190) Cross-Version Compatibility Test Harness
[
https://issues.apache.org/jira/browse/CALCITE-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15231144#comment-15231144
]
Josh Elser commented on CALCITE-1190:
-
Some progress on what I've been hacking together. A simple Java program which
can find a bunch of JUnit tests and invoke them given a Driver against a given
Avatica server (a URL). We can enumerate many versions to test via a YAML
config file:
{noformat}
tck_jar:
/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar
versions:
v1.6.0:
client_jar:
/Users/jelser/.m2/repository/org/apache/calcite/calcite-avatica/1.6.0/calcite-avatica-1.6.0.jar
server_url: http://localhost:8765
client_url_template: jdbc:avatica:remote:url=;serialization=PROTOBUF
v1.7.1:
client_jar:
/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.7.1/avatica-1.7.1.jar
server_url: http://localhost:8766
client_url_template: jdbc:avatica:remote:url=;serialization=PROTOBUF
v1.8.0-SNAPSHOT:
client_jar:
/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.8.0-SNAPSHOT/avatica-1.8.0-SNAPSHOT.jar
server_url: http://localhost:8767
client_url_template: jdbc:avatica:remote:url=;serialization=PROTOBUF
{noformat}
Then, a small ruby script will take the versions specified in the above YAML
file, generate a cartesian product of all of the versions (removing identity
mappings so those are already implicitly tested), and invoke the java tests
against each client=>server pair.
I need to write some more tests (just made a simple insert test so far), but
I'm pretty happy with it so far). Some example output:
{noformat}
Running v1.6.0 against v1.7.1
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/calcite-avatica/1.6.0/calcite-avatica-1.6.0.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8766;serialization=PROTOBUF''
2016-04-07 17:41:15,073 [main] INFO tck.TestRunner - Running
org.apache.calcite.avatica.tck.tests.InsertTest
2016-04-07 17:41:15,208 [main] INFO tck.TestRunner - Tests run: 1, Failures:
0, Skipped: 0, Time elapsed: 0 - in
org.apache.calcite.avatica.tck.tests.InsertTest
Tests run: 1, Failures: 0
Test of v1.6.0 against v1.7.1 PASSED!
Running v1.6.0 against v1.8.0-SNAPSHOT
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/calcite-avatica/1.6.0/calcite-avatica-1.6.0.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8766;serialization=PROTOBUF''
2016-04-07 17:41:16,020 [main] INFO tck.TestRunner - Running
org.apache.calcite.avatica.tck.tests.InsertTest
2016-04-07 17:41:16,163 [main] INFO tck.TestRunner - Tests run: 1, Failures:
0, Skipped: 0, Time elapsed: 0 - in
org.apache.calcite.avatica.tck.tests.InsertTest
Tests run: 1, Failures: 0
Test of v1.6.0 against v1.8.0-SNAPSHOT PASSED!
Running v1.7.1 against v1.6.0
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.7.1/avatica-1.7.1.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8765;serialization=PROTOBUF''
2016-04-07 17:41:17,044 [main] INFO tck.TestRunner - Running
org.apache.calcite.avatica.tck.tests.InsertTest
2016-04-07 17:41:17,176 [main] INFO tck.TestRunner - Tests run: 1, Failures:
0, Skipped: 0, Time elapsed: 0 - in
org.apache.calcite.avatica.tck.tests.InsertTest
Tests run: 1, Failures: 0
Test of v1.7.1 against v1.6.0 PASSED!
Running v1.7.1 against v1.8.0-SNAPSHOT
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.7.1/avatica-1.7.1.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8765;serialization=PROTOBUF''
2016-04-07 17:41:18,012 [main] INFO tck.TestRunner - Running
org.apache.calcite.avatica.tck.tests.InsertTest
2016-04-07 17:41:18,150 [main] INFO tck.TestRunner - Tests run: 1, Failures:
0, Skipped: 0, Time elapsed: 0 - in
org.apache.calcite.avatica.tck.tests.InsertTest
Tests run: 1, Failures: 0
Test of v1.7.1 against v1.8.0-SNAPSHOT PASSED!
Running v1.8.0-SNAPSHOT against v1.6.0
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.8.0-SNAPSHOT/avatica-1.8.0-SNAPSHOT.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8765;serialization=PROTOBUF''
2016-04-07 17:41:18,980 [main]
[jira] [Commented] (CALCITE-1190) Cross-Version Compatibility Test Harness
[ https://issues.apache.org/jira/browse/CALCITE-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228875#comment-15228875 ] Julian Hyde commented on CALCITE-1190: -- Wikipedia has a page on TCK https://en.wikipedia.org/wiki/Technology_Compatibility_Kit but I don't believe that the term is specific to the Java platform. It's just a test suite with inversion of control. Apache Harmony (mentioned in the wikipedia page) is an interesting (and sad) case, because it was a war fought over a TCK. > Cross-Version Compatibility Test Harness > > > Key: CALCITE-1190 > URL: https://issues.apache.org/jira/browse/CALCITE-1190 > Project: Calcite > Issue Type: Test > Components: avatica >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: avatica-1.8.0 > > > One thing that the Protobuf serialization aimed to provide was a library > which provides us the tools to make Avatica compatible across versions. > However, Protobuf is just a tool and the developers can still misuse protobuf > in such a way that breaks compatibility across versions. Not to mention, > compatibility isn't even remotely certain without any tests. > Because of Avatica's position as a library less than a product, we have to > defer some logic to the concrete product being tested (e.g. Phoenix or > Drill). I'm thinking something like the following: > The user provides pairs of client and server "definitions" for a given > version of compatibility. This would include some version of Avatica and some > backing database. For example, Avatica-1.7.1 and Phoenix-4.7.0 or > Avatica-1.8.0-SNAPSHOT and HSQLDB-2.3.1. > The client half would be some template for the appropriate JDBC url to use > (sans the URL to the Avatica server) and a JAR file containing the necessary > classes to run the j.s.Driver. The server half would just be a URL to the > Avatica server instance. > The test harness itself can provide the logic to test the remote driver > against the other servers, enumerating the possible combinations of > client-server communication. Starting the server for each version to test is > likely too difficult to automate well given the unknown of what the server > will be, so that will be left as a prerequisite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CALCITE-1190) Cross-Version Compatibility Test Harness
[ https://issues.apache.org/jira/browse/CALCITE-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228867#comment-15228867 ] Josh Elser commented on CALCITE-1190: - Hah, I was going to go digging through some projects that you contributed to looking for some inspiration. Thanks for saving me the effort! > Cross-Version Compatibility Test Harness > > > Key: CALCITE-1190 > URL: https://issues.apache.org/jira/browse/CALCITE-1190 > Project: Calcite > Issue Type: Test > Components: avatica >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: avatica-1.8.0 > > > One thing that the Protobuf serialization aimed to provide was a library > which provides us the tools to make Avatica compatible across versions. > However, Protobuf is just a tool and the developers can still misuse protobuf > in such a way that breaks compatibility across versions. Not to mention, > compatibility isn't even remotely certain without any tests. > Because of Avatica's position as a library less than a product, we have to > defer some logic to the concrete product being tested (e.g. Phoenix or > Drill). I'm thinking something like the following: > The user provides pairs of client and server "definitions" for a given > version of compatibility. This would include some version of Avatica and some > backing database. For example, Avatica-1.7.1 and Phoenix-4.7.0 or > Avatica-1.8.0-SNAPSHOT and HSQLDB-2.3.1. > The client half would be some template for the appropriate JDBC url to use > (sans the URL to the Avatica server) and a JAR file containing the necessary > classes to run the j.s.Driver. The server half would just be a URL to the > Avatica server instance. > The test harness itself can provide the logic to test the remote driver > against the other servers, enumerating the possible combinations of > client-server communication. Starting the server for each version to test is > likely too difficult to automate well given the unknown of what the server > will be, so that will be left as a prerequisite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CALCITE-1190) Cross-Version Compatibility Test Harness
[ https://issues.apache.org/jira/browse/CALCITE-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228857#comment-15228857 ] Julian Hyde commented on CALCITE-1190: -- Makes sense. You're talking about what I call a TCK. It is a module that can be used by another product to create a test suite. The other product implements callbacks to provide the test environment. I just created olap4j-tck which is very similar: https://github.com/olap4j/olap4j/tree/olap4j2/tck. Note that the code lives under src/main/java (not src/test/java) and it depends on junit (not in test scope). > Cross-Version Compatibility Test Harness > > > Key: CALCITE-1190 > URL: https://issues.apache.org/jira/browse/CALCITE-1190 > Project: Calcite > Issue Type: Test > Components: avatica >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: avatica-1.8.0 > > > One thing that the Protobuf serialization aimed to provide was a library > which provides us the tools to make Avatica compatible across versions. > However, Protobuf is just a tool and the developers can still misuse protobuf > in such a way that breaks compatibility across versions. Not to mention, > compatibility isn't even remotely certain without any tests. > Because of Avatica's position as a library less than a product, we have to > defer some logic to the concrete product being tested (e.g. Phoenix or > Drill). I'm thinking something like the following: > The user provides pairs of client and server "definitions" for a given > version of compatibility. This would include some version of Avatica and some > backing database. For example, Avatica-1.7.1 and Phoenix-4.7.0 or > Avatica-1.8.0-SNAPSHOT and HSQLDB-2.3.1. > The client half would be some template for the appropriate JDBC url to use > (sans the URL to the Avatica server) and a JAR file containing the necessary > classes to run the j.s.Driver. The server half would just be a URL to the > Avatica server instance. > The test harness itself can provide the logic to test the remote driver > against the other servers, enumerating the possible combinations of > client-server communication. Starting the server for each version to test is > likely too difficult to automate well given the unknown of what the server > will be, so that will be left as a prerequisite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CALCITE-1190) Cross-Version Compatibility Test Harness
[ https://issues.apache.org/jira/browse/CALCITE-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228679#comment-15228679 ] Josh Elser commented on CALCITE-1190: - Actually, maybe a good first stab is to just implement an HSQLDB avatica server impl that we can provide avatica jars at runtime? Same approach, but we wouldn't have to depend on Phoenix/Drill/etc to wire it up. > Cross-Version Compatibility Test Harness > > > Key: CALCITE-1190 > URL: https://issues.apache.org/jira/browse/CALCITE-1190 > Project: Calcite > Issue Type: Test > Components: avatica >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: avatica-1.8.0 > > > One thing that the Protobuf serialization aimed to provide was a library > which provides us the tools to make Avatica compatible across versions. > However, Protobuf is just a tool and the developers can still misuse protobuf > in such a way that breaks compatibility across versions. Not to mention, > compatibility isn't even remotely certain without any tests. > Because of Avatica's position as a library less than a product, we have to > defer some logic to the concrete product being tested (e.g. Phoenix or > Drill). I'm thinking something like the following: > The user provides pairs of client and server "definitions" for a given > version of compatibility. This would include some version of Avatica and some > backing database. For example, Avatica-1.7.1 and Phoenix-4.7.0 or > Avatica-1.8.0-SNAPSHOT and HSQLDB-2.3.1. > The client half would be some template for the appropriate JDBC url to use > (sans the URL to the Avatica server) and a JAR file containing the necessary > classes to run the j.s.Driver. The server half would just be a URL to the > Avatica server instance. > The test harness itself can provide the logic to test the remote driver > against the other servers, enumerating the possible combinations of > client-server communication. Starting the server for each version to test is > likely too difficult to automate well given the unknown of what the server > will be, so that will be left as a prerequisite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
