[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2177 ---
[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2177#discussion_r183204296 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java --- @@ -144,12 +144,12 @@ private boolean hasDataMapSchema; /** - * The boolean field which points if the data written for UnManaged Table - * or Managed Table. The difference between managed and unManaged table is - * unManaged Table will not contain any Metadata folder and subsequently + * The boolean field which points if the data written for Non Transactional Table + * or Transactional Table. The difference between Transactional and non Transactional table is + * non Transactional Table will not contain any Metadata folder and subsequently --- End diff -- Done ---
[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2177#discussion_r183204292
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
---
@@ -425,9 +427,11 @@ class NewDataFrameLoaderRDD[K, V](
TableProcessingOperations.deleteLocalDataLoadFolderLocation(model,
false, false)
// in case of failure the same operation will be re-tried several
times.
// So print the data load statistics only in case of non failure
case
-if (SegmentStatus.LOAD_FAILURE !=
loadMetadataDetails.getSegmentStatus) {
- CarbonTimeStatisticsFactory.getLoadStatisticsInstance
-.printStatisticsInfo(CarbonTablePath.DEPRECATED_PATITION_ID)
+if (!carbonLoadModel.isCarbonNonTransactionalTable) {
--- End diff --
Done
---
[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2177#discussion_r183204286
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
---
@@ -381,8 +381,10 @@ class NewDataFrameLoaderRDD[K, V](
carbonLoadModel.getTableName + CarbonCommonConstants.UNDERSCORE +
theSplit.index
try {
-
loadMetadataDetails.setPartitionCount(CarbonTablePath.DEPRECATED_PATITION_ID)
-loadMetadataDetails.setSegmentStatus(SegmentStatus.SUCCESS)
+if (!carbonLoadModel.isCarbonNonTransactionalTable) {
--- End diff --
Done
---
[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2177#discussion_r183204294 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java --- @@ -144,12 +144,12 @@ private boolean hasDataMapSchema; /** - * The boolean field which points if the data written for UnManaged Table - * or Managed Table. The difference between managed and unManaged table is - * unManaged Table will not contain any Metadata folder and subsequently + * The boolean field which points if the data written for Non Transactional Table + * or Transactional Table. The difference between Transactional and non Transactional table is + * non Transactional Table will not contain any Metadata folder and subsequently * no TableStatus or Schema files. */ - private boolean isUnManagedTable; + private boolean isNonTransactionalTable; --- End diff -- Done ---
[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2177#discussion_r183085758 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java --- @@ -144,12 +144,12 @@ private boolean hasDataMapSchema; /** - * The boolean field which points if the data written for UnManaged Table - * or Managed Table. The difference between managed and unManaged table is - * unManaged Table will not contain any Metadata folder and subsequently + * The boolean field which points if the data written for Non Transactional Table + * or Transactional Table. The difference between Transactional and non Transactional table is + * non Transactional Table will not contain any Metadata folder and subsequently --- End diff -- Mention that `transactional table means carbon will provide transactional support when user doing data management like data loading, whether it is success or failure, data will be in consistent state` ---
[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2177#discussion_r183085076 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java --- @@ -144,12 +144,12 @@ private boolean hasDataMapSchema; /** - * The boolean field which points if the data written for UnManaged Table - * or Managed Table. The difference between managed and unManaged table is - * unManaged Table will not contain any Metadata folder and subsequently + * The boolean field which points if the data written for Non Transactional Table + * or Transactional Table. The difference between Transactional and non Transactional table is + * non Transactional Table will not contain any Metadata folder and subsequently * no TableStatus or Schema files. */ - private boolean isUnManagedTable; + private boolean isNonTransactionalTable; --- End diff -- change to `isTransactional` ---
[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2177#discussion_r183074326
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/util/CarbonLoaderUtil.java
---
@@ -169,8 +169,28 @@ public static boolean
recordNewLoadMetadata(LoadMetadataDetails newMetaEntry,
* @throws IOException
*/
public static boolean recordNewLoadMetadata(LoadMetadataDetails
newMetaEntry,
- CarbonLoadModel loadModel, boolean loadStartEntry, boolean
insertOverwrite, String uuid)
+ final CarbonLoadModel loadModel, boolean loadStartEntry, boolean
insertOverwrite, String uuid)
throws IOException {
+if (loadModel.isCarbonNonTransactionalTable()) {
--- End diff --
Done
---
[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2177#discussion_r183043670
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/util/CarbonLoaderUtil.java
---
@@ -169,8 +169,28 @@ public static boolean
recordNewLoadMetadata(LoadMetadataDetails newMetaEntry,
* @throws IOException
*/
public static boolean recordNewLoadMetadata(LoadMetadataDetails
newMetaEntry,
- CarbonLoadModel loadModel, boolean loadStartEntry, boolean
insertOverwrite, String uuid)
+ final CarbonLoadModel loadModel, boolean loadStartEntry, boolean
insertOverwrite, String uuid)
throws IOException {
+if (loadModel.isCarbonNonTransactionalTable()) {
--- End diff --
Function name says updation of metadata, so should not delete data in that
---
[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2177#discussion_r182969475
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
---
@@ -425,9 +427,11 @@ class NewDataFrameLoaderRDD[K, V](
TableProcessingOperations.deleteLocalDataLoadFolderLocation(model,
false, false)
// in case of failure the same operation will be re-tried several
times.
// So print the data load statistics only in case of non failure
case
-if (SegmentStatus.LOAD_FAILURE !=
loadMetadataDetails.getSegmentStatus) {
- CarbonTimeStatisticsFactory.getLoadStatisticsInstance
-.printStatisticsInfo(CarbonTablePath.DEPRECATED_PATITION_ID)
+if (!carbonLoadModel.isCarbonNonTransactionalTable) {
--- End diff --
this check should not be present, logic should be same for transactional
and non-transactional tables;
---
[GitHub] carbondata pull request #2177: [CARBONDATA-2360][Non Transactional Table]Ins...
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2177#discussion_r182968973
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
---
@@ -381,8 +381,10 @@ class NewDataFrameLoaderRDD[K, V](
carbonLoadModel.getTableName + CarbonCommonConstants.UNDERSCORE +
theSplit.index
try {
-
loadMetadataDetails.setPartitionCount(CarbonTablePath.DEPRECATED_PATITION_ID)
-loadMetadataDetails.setSegmentStatus(SegmentStatus.SUCCESS)
+if (!carbonLoadModel.isCarbonNonTransactionalTable) {
--- End diff --
this check should not be present, only writing to tablestatus needs to be
avoided. All the remaining flow should be same.
---
