Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2706#discussion_r218474623
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/CarbonFactDataHandlerColumnar.java
---
@@ -239,6 +239,7 @@ public void addDataToStore(CarbonRow row) throws
CarbonDataWriterException {
* @return false if any varchar column page cannot add one more
value(2MB)
*/
private boolean isVarcharColumnFull(CarbonRow row) {
+ //TODO: test and remove this as now UnsafeSortDataRows can exceed 2MB
--- End diff --
@xuchuanyin @kevinjmh @ravipesala @kumarvishal09 : As per discussion let
us handle this with configurable page size [from 1 MB to 2GB(snappy max)] and
split the complex child pages here only and add validation for each column
based on row,
This will be analyzed more and I will open a discussion in community and
separate PR will be raised for this.
---
