GitHub user jackylk opened a pull request:
https://github.com/apache/carbondata/pull/2695
[CARBONDATA-2919] Support ingest from Kafka in StreamSQL
WIP
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance
test report.
- Any additional information to help reviewers in testing this
change.
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata kafka
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2695.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2695
----
commit 19dd6c641b9faa03da8b7a713873f398cd3b97d4
Author: Ajith <ajith2489@...>
Date: 2018-07-12T03:47:22Z
[CARBONDATA-2736][CARBONSTORE] Kafka integration with Carbon StreamSQL
Modification in this PR:
1.Pass source table properties to streamReader.load()
2.Do not pass schema when sparkSession.readStream
3.Remove querySchema validation against sink as dataFrame made from kafka
source will not have schema ( its written in value column of schema )
4.Extract the dataframe from kafka source which contain actual data schema
@ writeStream
This closes #2495
commit c2148e7e4fb6d7d5917a674075a7afdf941538c2
Author: Jacky Li <jacky.likun@...>
Date: 2018-09-05T17:45:40Z
wip
----
---
