GitHub user jackylk opened a pull request:
https://github.com/apache/carbondata/pull/2328
Support StreamSQL for streaming job
Currently, user need to write Spark Streaming APP to use carbon streaming
ingest feature, which is not so easy for some users. By providing StreamSQL,
user can manage the streaming job more easily.
1. CREATE STREAM SOURCE
```SQL
CREATE STREAM SOURCE iot.sensor_event(
time LONG,
device_id INT,
temperature double,
humidity double)
TBLPROPERTIES(
'streaming'='source'
'format'='kafka',
'kafka.bootstrap.servers'='host1:port1,host2:port2',
'topic'='update)
```
2. CREATE SINK TABLE
```SQL
CREATE TABLE iot.room_temperature(
time LONG,
room_id INT,
temperature double)
STORED AS carbondata
TBLPROPERTIES('streaming'='sink') #compatible with 'true'
```
3. CREATE STREAM (Start a streaming job)
```SQL
CREATE STREAM ON TABLE iot.room_temperature
STMPROPERTIES(
'trigger'='ProcessingTime/OneTimeTrigger',
'interval'='5 seconds')
AS
SELECT time, lookup_room_id(device_id) as room_id, temperature
FROM iot.sensor_event
WHERE temperature between 0 and 40
```
## Streaming Job Management
1. SHOW STREAMS
```SQL
SHOW STREAMS [ON TABLE dbName.tableName]
```
It will print
| Jobid | status | Source | Sink | start time |
time elapsed |
| ------------------ | ------- | ------ | ---- | ------------------- |
------------ |
| jkfldsjkad-fkwelj0 | Started | device | fact | 2018-02-03 14:32:42 |
10d2h32m |
2. Kill STREAM
```SQL
KILL STREAM ON TABLE dbName.tableName
```
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance
test report.
- Any additional information to help reviewers in testing this
change.
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata stream
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2328.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2328
----
commit 2482c0367deb17b1d46b7676a1339d42e8329456
Author: Jacky Li <jacky.likun@...>
Date: 2018-05-21T13:49:33Z
support StreamSQL
----
---
