[
https://issues.apache.org/jira/browse/KNOX-2023?focusedWorklogId=323524&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323524
]
ASF GitHub Bot logged work on KNOX-2023:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Oct/19 16:11
Start Date: 04/Oct/19 16:11
Worklog Time Spent: 10m
Work Description: smolnar82 commented on pull request #162: KNOX-2023 -
Recording KnoxShellTable builder/filter chain and providing rollback/replay
capabilities using the call history as well as allowing end-users to export
JSON without data (in this case only the call history will be serialized)
URL: https://github.com/apache/knox/pull/162
## What changes were proposed in this pull request?
The following features have been added with this change in `knoxshell`:
- recording KnoxShellTable builder/filter chain
- providing rollback/replay capabilities using the call history
- allowing end-users to export JSON without data (in this case, only the
call history will be serialized)
## How was this patch tested?
Executing a full build (where I added/updated new/existing JUnit tests):
```
$ mvn clean -Dshellcheck=true -T1C verify -Prelease,package
...
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 17:58 min (Wall Clock)
[INFO] Finished at: 2019-10-04T18:08:41+02:00
[INFO] Final Memory: 389M/1837M
[INFO]
------------------------------------------------------------------------
```
In addition to running JUnit tests the following manual test steps were
executed:
1.) Building a table with the JDBC builder:
```
locations =
KnoxShellTable.builder().jdbc().driver("org.apache.derby.jdbc.EmbeddedDriver").connectTo("jdbc:derby:/Users/smolnar/test/derbyDb").sql("SELECT
* FROM sample.locations where zip < 15")
===> LOCATIONS
+--------+------------+----------+----------+--------------+
| ZIP | COUNTRY | STATE | CITY | POPULATION |
+--------+------------+----------+----------+--------------+
| 1 | US | NY | City1 | 100000 |
| 2 | US | NY | City2 | 100000 |
| 3 | US | NY | City3 | 100000 |
| 4 | US | NY | City4 | 100000 |
| 5 | US | NY | City5 | 100000 |
| 6 | US | NY | City6 | 100000 |
| 7 | US | NY | City7 | 100000 |
| 8 | US | NY | City8 | 100000 |
| 9 | US | NY | City9 | 100000 |
| 10 | US | NY | City10 | 100000 |
| 11 | US | NY | City11 | 100000 |
| 12 | US | NY | City12 | 100000 |
| 13 | US | NY | City13 | 100000 |
| 14 | US | NY | City14 | 100000 |
+--------+------------+----------+----------+--------------+
knox:000>
knox:000> locations.getCallHistory()
===> Call history (id=1570194513709)
Step 1:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableBuilder
method=jdbc
builderMethod=false
params={}
Step 2:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=driver
builderMethod=false
params={org.apache.derby.jdbc.EmbeddedDriver=class java.lang.String}
Step 3:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=connectTo
builderMethod=false
params={jdbc:derby:/Users/smolnar/test/derbyDb=class java.lang.String}
Step 4:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=sql
builderMethod=true
params={SELECT * FROM sample.locations where zip < 15=class java.lang.String}
```
2.) Filtering the previously built table:
```
knox:000> filteredLocations = locations.filter().name("ZIP").greaterThan(5)
===> LOCATIONS
+--------+------------+----------+----------+--------------+
| ZIP | COUNTRY | STATE | CITY | POPULATION |
+--------+------------+----------+----------+--------------+
| 6 | US | NY | City6 | 100000 |
| 7 | US | NY | City7 | 100000 |
| 8 | US | NY | City8 | 100000 |
| 9 | US | NY | City9 | 100000 |
| 10 | US | NY | City10 | 100000 |
| 11 | US | NY | City11 | 100000 |
| 12 | US | NY | City12 | 100000 |
| 13 | US | NY | City13 | 100000 |
| 14 | US | NY | City14 | 100000 |
+--------+------------+----------+----------+--------------+
knox:000>
knox:000> filteredLocations.getCallHistory()
===> Call history (id=1570194514968)
Step 1:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableBuilder
method=jdbc
builderMethod=false
params={}
Step 2:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=driver
builderMethod=false
params={org.apache.derby.jdbc.EmbeddedDriver=class java.lang.String}
Step 3:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=connectTo
builderMethod=false
params={jdbc:derby:/Users/smolnar/test/derbyDb=class java.lang.String}
Step 4:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=sql
builderMethod=true
params={SELECT * FROM sample.locations where zip < 15=class java.lang.String}
Step 5:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTable
method=filter
builderMethod=false
params={}
Step 6:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableFilter
method=name
builderMethod=false
params={ZIP=class java.lang.String}
Step 7:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableFilter
method=greaterThan
builderMethod=true
params={5=class java.lang.Integer}
knox:000>
knox:000> filteredLocations.toJSON()
===> {
"headers" : [ "ZIP", "COUNTRY", "STATE", "CITY", "POPULATION" ],
"rows" : [ [ 6, "US", "NY", "City6", 100000 ], [ 7, "US", "NY", "City7",
100000 ], [ 8, "US", "NY", "City8", 100000 ], [ 9, "US", "NY", "City9", 100000
], [ 10, "US", "NY", "City10", 100000 ], [ 11, "US", "NY", "City11", 100000 ],
[ 12, "US", "NY", "City12", 100000 ], [ 13, "US", "NY", "City13", 100000 ], [
14, "US", "NY", "City14", 100000 ] ],
"title" : "LOCATIONS",
"id" : 1570194514968
}
knox:000>
knox:000> filteredLocations.toJSON(false)
===> {
"callHistoryList" : [ {
"invokerClass" :
"org.apache.knox.gateway.shell.table.KnoxShellTableBuilder",
"method" : "jdbc",
"builderMethod" : false,
"params" : { }
}, {
"invokerClass" :
"org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder",
"method" : "driver",
"builderMethod" : false,
"params" : {
"org.apache.derby.jdbc.EmbeddedDriver" : "java.lang.String"
}
}, {
"invokerClass" :
"org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder",
"method" : "connectTo",
"builderMethod" : false,
"params" : {
"jdbc:derby:/Users/smolnar/test/derbyDb" : "java.lang.String"
}
}, {
"invokerClass" :
"org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder",
"method" : "sql",
"builderMethod" : true,
"params" : {
"SELECT * FROM sample.locations where zip < 15" : "java.lang.String"
}
}, {
"invokerClass" : "org.apache.knox.gateway.shell.table.KnoxShellTable",
"method" : "filter",
"builderMethod" : false,
"params" : { }
}, {
"invokerClass" :
"org.apache.knox.gateway.shell.table.KnoxShellTableFilter",
"method" : "name",
"builderMethod" : false,
"params" : {
"ZIP" : "java.lang.String"
}
}, {
"invokerClass" :
"org.apache.knox.gateway.shell.table.KnoxShellTableFilter",
"method" : "greaterThan",
"builderMethod" : true,
"params" : {
"5" : "java.lang.Integer"
}
} ]
}
```
3.) Replaying the filtered table up to `step 4`:
```
knox:000> replayed = filteredLocations.replay(4)
===> LOCATIONS
+--------+------------+----------+----------+--------------+
| ZIP | COUNTRY | STATE | CITY | POPULATION |
+--------+------------+----------+----------+--------------+
| 1 | US | NY | City1 | 100000 |
| 2 | US | NY | City2 | 100000 |
| 3 | US | NY | City3 | 100000 |
| 4 | US | NY | City4 | 100000 |
| 5 | US | NY | City5 | 100000 |
| 6 | US | NY | City6 | 100000 |
| 7 | US | NY | City7 | 100000 |
| 8 | US | NY | City8 | 100000 |
| 9 | US | NY | City9 | 100000 |
| 10 | US | NY | City10 | 100000 |
| 11 | US | NY | City11 | 100000 |
| 12 | US | NY | City12 | 100000 |
| 13 | US | NY | City13 | 100000 |
| 14 | US | NY | City14 | 100000 |
+--------+------------+----------+----------+--------------+
knox:000>
knox:000> replayed.getCallHistory()
===> Call history (id=1570194515303)
Step 1:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableBuilder
method=jdbc
builderMethod=false
params={}
Step 2:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=driver
builderMethod=false
params={org.apache.derby.jdbc.EmbeddedDriver=class java.lang.String}
Step 3:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=connectTo
builderMethod=false
params={jdbc:derby:/Users/smolnar/test/derbyDb=class java.lang.String}
Step 4:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=sql
builderMethod=true
params={SELECT * FROM sample.locations where zip < 15=class java.lang.String}
```
4.) Replaying the filtered table entirely:
```
knox:000> replayed = filteredLocations.replayAll()
===> LOCATIONS
+--------+------------+----------+----------+--------------+
| ZIP | COUNTRY | STATE | CITY | POPULATION |
+--------+------------+----------+----------+--------------+
| 6 | US | NY | City6 | 100000 |
| 7 | US | NY | City7 | 100000 |
| 8 | US | NY | City8 | 100000 |
| 9 | US | NY | City9 | 100000 |
| 10 | US | NY | City10 | 100000 |
| 11 | US | NY | City11 | 100000 |
| 12 | US | NY | City12 | 100000 |
| 13 | US | NY | City13 | 100000 |
| 14 | US | NY | City14 | 100000 |
+--------+------------+----------+----------+--------------+
knox:000>
knox:000> replayed.getCallHistory()
===> Call history (id=1570194515309)
Step 1:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableBuilder
method=jdbc
builderMethod=false
params={}
Step 2:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=driver
builderMethod=false
params={org.apache.derby.jdbc.EmbeddedDriver=class java.lang.String}
Step 3:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=connectTo
builderMethod=false
params={jdbc:derby:/Users/smolnar/test/derbyDb=class java.lang.String}
Step 4:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=sql
builderMethod=true
params={SELECT * FROM sample.locations where zip < 15=class java.lang.String}
Step 5:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTable
method=filter
builderMethod=false
params={}
Step 6:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableFilter
method=name
builderMethod=false
params={ZIP=class java.lang.String}
Step 7:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableFilter
method=greaterThan
builderMethod=true
params={5=class java.lang.Integer}
```
5.) Rolled back to the previous valid state
```
knox:000> replayed.rollback()
===> Successfully rolled back
knox:000> replayed.getCallHistory()
===> Call history (id=1570195550441)
Step 1:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableBuilder
method=jdbc
builderMethod=false
params={}
Step 2:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=driver
builderMethod=false
params={org.apache.derby.jdbc.EmbeddedDriver=class java.lang.String}
Step 3:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=connectTo
builderMethod=false
params={jdbc:derby:/Users/smolnar/test/derbyDb=class java.lang.String}
Step 4:
invokerClass=org.apache.knox.gateway.shell.table.JDBCKnoxShellTableBuilder
method=sql
builderMethod=true
params={SELECT * FROM sample.locations where zip < 15=class java.lang.String}
```
6.) Building a table from JSON source holding pure data:
```
knox:000> loadedFromJsonWithData =
KnoxShellTable.builder().json().path("/Users/smolnar/sampleTableWithData.json")
===> +--------+------------+----------+----------+--------------+
| ZIP | COUNTRY | STATE | CITY | POPULATION |
+--------+------------+----------+----------+--------------+
| 6 | US | NY | City6 | 100000 |
| 7 | US | NY | City7 | 100000 |
| 8 | US | NY | City8 | 100000 |
| 9 | US | NY | City9 | 100000 |
| 10 | US | NY | City10 | 100000 |
| 11 | US | NY | City11 | 100000 |
| 12 | US | NY | City12 | 100000 |
| 13 | US | NY | City13 | 100000 |
| 14 | US | NY | City14 | 100000 |
+--------+------------+----------+----------+--------------+
knox:000>
knox:000> loadedFromJsonWithData.getCallHistory()
===> Call history (id=1570194515913)
Step 1:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableBuilder
method=json
builderMethod=false
params={}
Step 2:
invokerClass=org.apache.knox.gateway.shell.table.JSONKnoxShellTableBuilder
method=path
builderMethod=true
params={/Users/smolnar/sampleTableWithData.json=class java.lang.String}
```
7.) Building a table from JSON source holding call history:
```
knox:000> loadedFromJsonWithCallHistory =
KnoxShellTable.builder().json().path("/Users/smolnar/sampleTableWithCallHistory.json")
===> LOCATIONS
+--------+------------+----------+----------+--------------+
| ZIP | COUNTRY | STATE | CITY | POPULATION |
+--------+------------+----------+----------+--------------+
| 6 | US | NY | City6 | 100000 |
| 7 | US | NY | City7 | 100000 |
| 8 | US | NY | City8 | 100000 |
| 9 | US | NY | City9 | 100000 |
| 10 | US | NY | City10 | 100000 |
| 11 | US | NY | City11 | 100000 |
| 12 | US | NY | City12 | 100000 |
| 13 | US | NY | City13 | 100000 |
| 14 | US | NY | City14 | 100000 |
+--------+------------+----------+----------+--------------+
knox:000>
knox:000> loadedFromJsonWithCallHistory.getCallHistory()
===> Call history (id=1570194516193)
Step 1:
invokerClass=org.apache.knox.gateway.shell.table.KnoxShellTableBuilder
method=json
builderMethod=false
params={}
Step 2:
invokerClass=org.apache.knox.gateway.shell.table.JSONKnoxShellTableBuilder
method=path
builderMethod=true
params={/Users/smolnar/sampleTableWithCallHistory.json=class
java.lang.String}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 323524)
Remaining Estimate: 0h
Time Spent: 10m
> KnoxShellTable - Record the Creation Sequence of Tables
> -------------------------------------------------------
>
> Key: KNOX-2023
> URL: https://issues.apache.org/jira/browse/KNOX-2023
> Project: Apache Knox
> Issue Type: Improvement
> Components: KnoxShell
> Reporter: Larry McCay
> Assignee: Sandor Molnar
> Priority: Major
> Fix For: 1.4.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Each KnoxShellTable instance goes through a number of builders, filters,
> sorts, etc in order to get to their final state.
> Currently, those builder classes just go out of scope as they complete and
> return the resulting table.
> This JIRA represents the thought of recording them in an ArrayList within the
> table so that we could do a number of interesting things:
> * persist JSON representation that doesn't require the actual data
> * materialize tables from that persisted state by replaying the builders in
> order
> * possibly undo or rollback to a previous state
> * possibly providing interesting provenance capabilities with confidence
> factors based on sources
> Interesting challenges here will be when authentication is required for a SQL
> query in the JDBC builder or in accessing data accessed through Knox gateway
> instances.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
