Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2706#discussion_r218403711
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/CarbonFactDataHandlerColumnar.java
---
@@ -239,6 +239,7 @@ public void addDataToStore(CarbonRow row) throws
CarbonDataWriterException {
* @return false if any varchar column page cannot add one more
value(2MB)
*/
private boolean isVarcharColumnFull(CarbonRow row) {
+ //TODO: test and remove this as now UnsafeSortDataRows can exceed 2MB
--- End diff --
Original implementation use `2MB` to ensure next varchar column value can
be filled safely, because size of value of single column won't exceed size of a
row.
If UnsafeSortDataRows can exceed 2MB(growing dynamically), then we cannot
check whether we have enough space for next value because we are not sure how
many space next value will take. So the column page size check should be run
before adding row to `dataRows`
---
