GitHub user BJangir opened a pull request:
https://github.com/apache/carbondata/pull/2508
[CARBONDATA-2744]Streaming lock is not released even Batch processingâ¦
Issue Detail :- if Streaming Application is running , DDLs like finish
streaming ,close streaming are blocked.
ideally DDLs like finish streaming ,close streaming should be blocked if
Batch Processing is running. if Batch processing is not happening then DDL's
should be allowed from JDBCServer/Beeline.
Root Cause :- Streaming lock is taken on application start and it is
released onQueryTerminate event of CarbonStreamingQueryListener ,this event
will be called when stop() is called on StreamingQuery which means streaming
Lock will be released on Either Streaming Query should be terminated Or
complete Streaming Application should be terminated till then all stream lock
DDL's are blocked.
Solution :- on AddBatch take streaming lock and once Batch is processed
,release the streaming lock.
Note:-
a. If close streaming is called OR streaming Table is updated with
'streaming'='false' and on Trigger time to AddBatch, addBatch will throw
Exception and StreamingQuery should be start again.
b. if DDLs like finish streaming ,close streaming started 1st and addBatch
started 2nd. addBatch will throw "can not acquire lock" Exception and
StreamingQuery should be start again.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
NO
- [ ] Any backward compatibility impacted?
NO
- [ ] Document update required?
NO
- [ ] Testing done
Manually verified below scenarios
a. Call StreamingFinished when addBatch is Done.
b. Whether New Batch works after Streaming Finish DDL success. it creates
new Streaming Segment
c. New Batch while Streaming Finnish DDL was running
d. Call Streaming Finish when Add Batch was Running
Verified above scenarios for other DDL' also (close streaming,set SET
TBLPROPERTIES('streaming'='false')
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
NO
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BJangir/incubator-carbondata streaming_lock
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2508.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2508
----
commit 2ff88bad23a74ea9b0167574955c4e14c65ca755
Author: BJangir <babulaljangir111@...>
Date: 2018-07-15T16:14:02Z
[CARBONDATA-2744]Streaming lock is not released even Batch processing is
not happening
----
---
