[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3957: [CARBONDATA-4011] drop index table is failed with table not found exception
CarbonDataQA1 commented on pull request #3957: URL: https://github.com/apache/carbondata/pull/3957#issuecomment-699092913 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2480/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3957: [CARBONDATA-4011] drop index table is failed with table not found exception
CarbonDataQA1 commented on pull request #3957: URL: https://github.com/apache/carbondata/pull/3957#issuecomment-699092326 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4224/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on pull request #3957: [CARBONDATA-4011] drop index table is failed with table not found exception
maheshrajus commented on pull request #3957: URL: https://github.com/apache/carbondata/pull/3957#issuecomment-699034567 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698971452 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4221/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698970183 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2478/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-698959902 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4223/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-698958049 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2479/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
marchpure commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698882681 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (CARBONDATA-4009) PartialQuery not hitting mv
[ https://issues.apache.org/jira/browse/CARBONDATA-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akash R Nilugal resolved CARBONDATA-4009. - Fix Version/s: 2.1.0 Resolution: Fixed > PartialQuery not hitting mv > --- > > Key: CARBONDATA-4009 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4009 > Project: CarbonData > Issue Type: Bug >Reporter: Indhumathi Muthumurugesh >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] asfgit closed pull request #3955: [CARBONDATA-4009] Fix PartialQueries not hitting mv
asfgit closed pull request #3955: URL: https://github.com/apache/carbondata/pull/3955 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on pull request #3952: [CARBONDATA-4006] Fix for currentUser as NULL in getcount method during index server fallback mode
vikramahuja1001 commented on pull request #3952: URL: https://github.com/apache/carbondata/pull/3952#issuecomment-698943022 @kunal642 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat closed pull request #3958: [WIP] support presto 333
ajantha-bhat closed pull request #3958: URL: https://github.com/apache/carbondata/pull/3958 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3956: [CARBONDATA-4010] Added documentation for set/unset of longStringColumns.
CarbonDataQA1 commented on pull request #3956: URL: https://github.com/apache/carbondata/pull/3956#issuecomment-698813517 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-698817357 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494964396
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -18,34 +18,18 @@
package org.apache.carbondata.spark.testsuite.iud
import java.text.SimpleDateFormat
-import java.util
-import java.util.concurrent.{Callable, ExecutorService, Executors, Future}
-
-import scala.collection.JavaConverters._
-
-import org.apache.hadoop.fs.Path
-import org.apache.spark.sql.test.util.QueryTest
-import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.{DataFrame, SaveMode}
-import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.datastore.block.SegmentProperties
-import org.apache.carbondata.core.datastore.page.ColumnPage
import org.apache.carbondata.core.exception.ConcurrentOperationException
-import org.apache.carbondata.core.features.TableOperation
-import org.apache.carbondata.core.index.dev.cgindex.{CoarseGrainIndex,
CoarseGrainIndexFactory}
-import org.apache.carbondata.core.index.dev.{IndexBuilder, IndexWriter}
-import org.apache.carbondata.core.index.{IndexInputSplit, IndexMeta, Segment}
-import org.apache.carbondata.core.indexstore.PartitionSpec
-import org.apache.carbondata.core.metadata.schema.table.{CarbonTable,
IndexSchema}
-import org.apache.carbondata.core.scan.filter.intf.ExpressionType
import org.apache.carbondata.core.util.CarbonProperties
-import org.apache.carbondata.events.Event
+import org.apache.spark.sql.test.util.QueryTest
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{DataFrame, Row, SaveMode}
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/AsyncExecutorUtils.scala
##
@@ -0,0 +1,200 @@
+
+
+
+package org.apache.carbondata.spark.testsuite.iud
Review comment:
done
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala
##
@@ -65,7 +65,7 @@ private[sql] case class CarbonProjectForDeleteCommand(
s"Unsupported delete operation on table containing mixed format
segments")
}
-if (SegmentStatusManager.isLoadInProgressInTable(carbonTable)) {
+if (SegmentStatusManager.isInsertOverwriteInProgress(carbonTable)) {
throw new ConcurrentOperationException(carbonTable, "loading", "data
delete")
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -122,22 +139,23 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
sql("drop materialized view if exists mv4")
sql("drop materialized view if exists mv5")
}
+
dropMVs
sql(
"create materialized view mv1 as " +
- "select timeseries(projectjoindate,'second'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'second')")
+"select timeseries(projectjoindate,'second'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'second')")
sql(
"create materialized view mv2 as " +
- "select timeseries(projectjoindate,'hour'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'hour')")
+"select timeseries(projectjoindate,'hour'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'hour')")
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -170,17 +146,19 @@ class TestInsertAndOtherCommandConcurrent extends
QueryTest with BeforeAndAfterA
}
test("alter rename table should fail if insert overwrite is in progress") {
-val future = runSqlAsync("insert overwrite table orders select * from
orders_overwrite")
+sql("drop table if exists other_orders")
+val future = AsyncExecutorUtils.runSqlAsync("insert overwrite table orders
select * from orders_overwrite")
val ex = intercept[ConcurrentOperationException] {
- sql("alter table orders rename to other")
+ sql("alter table orders rename to other_orders")
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -92,54 +77,46 @@ class TestInsertAndOtherCommandConcurrent extends QueryTest
with BeforeAndAfterA
private def createTable(tableName: String, schema: StructType): Unit = {
val schemaString = schema.fields.map(x => x.name + " " +
x.dataType.typeName).mkString(", ")
sql(s"CREATE TABLE
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3957: [WIP] drop index table is failed with table not found exception
CarbonDataQA1 commented on pull request #3957: URL: https://github.com/apache/carbondata/pull/3957#issuecomment-698821503 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li closed pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li closed pull request #3947: URL: https://github.com/apache/carbondata/pull/3947 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3955: [CARBONDATA-4009] Fix SubQuery not hitting mv
akashrn5 commented on a change in pull request #3955:
URL: https://github.com/apache/carbondata/pull/3955#discussion_r494792185
##
File path:
mv/plan/src/main/scala/org/apache/carbondata/mv/plans/modular/Modularizer.scala
##
@@ -66,6 +68,22 @@ abstract class Modularizer[TreeType <: TreeNode[TreeType]] {
makeupAliasMappings(mplans)
}
+ private def transformPlan(plan: LogicalPlan) = {
Review comment:
please change the method name to something meaningful according to
functionality
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3959: [CARBONDATA-4010] Doc changes for long strings as bad record.
CarbonDataQA1 commented on pull request #3959: URL: https://github.com/apache/carbondata/pull/3959#issuecomment-698869602 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
marchpure commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494819647
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatus.java
##
@@ -31,6 +31,12 @@
@SerializedName("Success")
SUCCESS("Success"),
+ /**
Review comment:
revert this change
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatusManager.java
##
@@ -918,17 +893,40 @@ public static Boolean isCompactionInProgress(CarbonTable
carbonTable) {
return compactionInProgress;
}
+ /**
+ * Return true if insert or insert overwrite is in progress for specified
table
+ */
+ public static Boolean isInsertInProgress(CarbonTable carbonTable) {
+if (carbonTable == null) {
+ return false;
+}
+boolean loadInProgress = false;
+String metaPath = carbonTable.getMetadataPath();
+LoadMetadataDetails[] listOfLoadFolderDetailsArray =
SegmentStatusManager.readLoadMetadata(metaPath);
+if (listOfLoadFolderDetailsArray.length != 0) {
+ for (LoadMetadataDetails loadDetail :listOfLoadFolderDetailsArray) {
+SegmentStatus segmentStatus = loadDetail.getSegmentStatus();
+if (segmentStatus == SegmentStatus.INSERT_IN_PROGRESS
+|| segmentStatus == SegmentStatus.INSERT_OVERWRITE_IN_PROGRESS) {
+ loadInProgress =
+ isLoadInProgress(carbonTable.getAbsoluteTableIdentifier(),
+ loadDetail.getLoadName());
+}
+ }
+}
+return loadInProgress;
+ }
+
/**
* Return true if insert overwrite is in progress for specified table
*/
- public static Boolean isOverwriteInProgressInTable(CarbonTable carbonTable) {
+ public static boolean isInsertOverwriteInProgress(CarbonTable carbonTable) {
if (carbonTable == null) {
return false;
}
boolean loadInProgress = false;
String metaPath = carbonTable.getMetadataPath();
-LoadMetadataDetails[] listOfLoadFolderDetailsArray =
-SegmentStatusManager.readLoadMetadata(metaPath);
+LoadMetadataDetails[] listOfLoadFolderDetailsArray =
SegmentStatusManager.readLoadMetadata(metaPath);
Review comment:
revert this change
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatusManager.java
##
@@ -918,17 +893,40 @@ public static Boolean isCompactionInProgress(CarbonTable
carbonTable) {
return compactionInProgress;
}
+ /**
+ * Return true if insert or insert overwrite is in progress for specified
table
+ */
+ public static Boolean isInsertInProgress(CarbonTable carbonTable) {
Review comment:
revert this change
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/DeleteFromTableEventListener.scala
##
@@ -53,6 +53,12 @@ class DeleteFromTableEventListener extends
OperationEventListener with Logging {
carbonTable
.getDatabaseName
}.${ carbonTable.getTableName }]")
+} else if (!carbonTable.getIndexesMap.isEmpty) {
Review comment:
change to
"if (carbonTable.isIndexTable || !carbonTable.getIndexesMap.isEmpty)"
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/UpdateTablePreEventListener.scala
##
@@ -49,7 +49,6 @@ class UpdateTablePreEventListener extends
OperationEventListener with Logging {
carbonTable
.getDatabaseName
}.${ carbonTable.getTableName }]. Drop all indexes and retry")
Review comment:
revert this change
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -68,8 +53,8 @@ class TestInsertAndOtherCommandConcurrent extends QueryTest
with BeforeAndAfterA
.mode(SaveMode.Overwrite)
.save()
-sql(s"insert into orders select * from temp_table")
-sql(s"insert into orders_overwrite select * from temp_table")
+sql(s"insert into orders select * from temp_table") // load_0 success
Review comment:
revert this change
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
##
@@ -18,21 +18,18 @@ package org.apache.carbondata.spark.testsuite.iud
import java.io.File
-import org.apache.spark.sql.hive.CarbonRelation
-import org.apache.spark.sql.test.SparkTestQueryExecutor
-import org.apache.spark.sql.test.util.QueryTest
-import org.apache.spark.sql.{CarbonEnv, Row, SaveMode}
-import org.scalatest.BeforeAndAfterAll
-
import
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.index.Segment
import
[GitHub] [carbondata] Kejian-Li removed a comment on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li removed a comment on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-698910832 > Consider a scenario that the user want to insert data and then update the table, the dml commands are sent one by one in time series, but the 2nd(update) cmd is sent by another driver and start to execute while the 1st(insert) is still running, therefore the concurrency scenario described in this PR appears. > If we allow update table who has segment with status 'INSERT_INPROGRESS', the 2nd cmd(update) will execute successfully and the data inserted by the 1st cmd would not updated by the 2nd cmd. > In this case, the result may not be what the user expected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3955: [CARBONDATA-4009] Fix PartialQueries not hitting mv
akashrn5 commented on pull request #3955: URL: https://github.com/apache/carbondata/pull/3955#issuecomment-698900334 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3957: [WIP] drop index table is failed with table not found exception
Indhumathi27 commented on a change in pull request #3957:
URL: https://github.com/apache/carbondata/pull/3957#discussion_r494856214
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/DropTableTest.scala
##
@@ -74,4 +74,17 @@ class DropTableTest extends QueryTest with BeforeAndAfterAll
{
assert(exception.getMessage.contains("Index with name indtestdrop does not
exist"))
sql("drop table if exists testDrop")
}
+
+ test("test drop index command after refresh the index for empty index
table") {
+sql("drop table if exists testDropIndex")
+sql("create table testDropIndex (a string, b string, c string) STORED AS
carbondata")
+sql("create index helloIndex on table testDropIndex (c) AS 'carbondata'
properties" +
+"('table_blocksize'='1')")
+sql("show indexes on testDropIndex").show()
Review comment:
please remove this line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #3955: [CARBONDATA-4009] Fix PartialQueries not hitting mv
Indhumathi27 commented on pull request #3955: URL: https://github.com/apache/carbondata/pull/3955#issuecomment-698828029 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3952: [CARBONDATA-4006] Fix for currentUser as NULL in getcount method during index server fallback mode
CarbonDataQA1 commented on pull request #3952: URL: https://github.com/apache/carbondata/pull/3952#issuecomment-698160254 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-698519065 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
ShreelekhyaG commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-698763056 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-698910832 > Consider a scenario that the user want to insert data and then update the table, the dml commands are sent one by one in time series, but the 2nd(update) cmd is sent by another driver and start to execute while the 1st(insert) is still running, therefore the concurrency scenario described in this PR appears. > If we allow update table who has segment with status 'INSERT_INPROGRESS', the 2nd cmd(update) will execute successfully and the data inserted by the 1st cmd would not updated by the 2nd cmd. > In this case, the result may not be what the user expected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698133539 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on pull request #3952: [CARBONDATA-4006] Fix for currentUser as NULL in getcount method during index server fallback mode
vikramahuja1001 commented on pull request #3952: URL: https://github.com/apache/carbondata/pull/3952#issuecomment-698117428 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3953: [CARBONDATA-4008]Fixed IN filter on date column is returning 0 results when 'carbon.push.rowfilters.for.vector' is true
VenuReddy2103 commented on a change in pull request #3953:
URL: https://github.com/apache/carbondata/pull/3953#discussion_r494050860
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/filterexpr/TestInFilter.scala
##
@@ -165,8 +168,27 @@ class TestInFilter extends QueryTest with
BeforeAndAfterAll{
Seq(Row(4, 1.00, 2.00, 3.00)))
}
- override def afterAll(): Unit = {
+ test("test infilter with date, timestamp columns") {
+sql("create table test_table(i int, dt date, ts timestamp) stored as
carbondata")
+sql("insert into test_table select 1, '2020-03-30', '2020-03-30 10:00:00'")
Review comment:
Have already added drop table in afterEach()
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecutorImpl.java
##
@@ -138,6 +145,8 @@ public
RowLevelFilterExecutorImpl(List dimColEvalua
this.exp = exp;
this.tableIdentifier = tableIdentifier;
this.complexDimensionInfoMap = complexDimensionInfoMap;
+this.dateDictionaryGenerator =
+
DirectDictionaryKeyGeneratorFactory.getDirectDictionaryGenerator(DataTypes.DATE);
Review comment:
Yeah not happening for timestamp. But have added testcase for timestamp
as well.
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecutorImpl.java
##
@@ -138,6 +145,8 @@ public
RowLevelFilterExecutorImpl(List dimColEvalua
this.exp = exp;
this.tableIdentifier = tableIdentifier;
this.complexDimensionInfoMap = complexDimensionInfoMap;
+this.dateDictionaryGenerator =
+
DirectDictionaryKeyGeneratorFactory.getDirectDictionaryGenerator(DataTypes.DATE);
Review comment:
Yeah not happening for timestamp. But have added testcase for infilter
with timestamp as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on pull request #3953: [CARBONDATA-4008]Fixed IN filter on date column is returning 0 results when 'carbon.push.rowfilters.for.vector' is true
VenuReddy2103 commented on pull request #3953: URL: https://github.com/apache/carbondata/pull/3953#issuecomment-698100642 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat closed pull request #3954: [TEMP_NEW] support presto 330
ajantha-bhat closed pull request #3954: URL: https://github.com/apache/carbondata/pull/3954 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3955: [WIP] Fix SubQuery not hitting mv
CarbonDataQA1 commented on pull request #3955: URL: https://github.com/apache/carbondata/pull/3955#issuecomment-698525365 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3953: [CARBONDATA-4008]Fixed IN filter on date column is returning 0 results when 'carbon.push.rowfilters.for.vector' is true
CarbonDataQA1 commented on pull request #3953: URL: https://github.com/apache/carbondata/pull/3953#issuecomment-698136460 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3662: [TEMP] support prestosql 330 in carbon
ajantha-bhat commented on pull request #3662: URL: https://github.com/apache/carbondata/pull/3662#issuecomment-698291139 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-698284426 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] kunal642 commented on a change in pull request #3953: [CARBONDATA-4008]Fixed IN filter on date column is returning 0 results when 'carbon.push.rowfilters.for.vector' is true
kunal642 commented on a change in pull request #3953:
URL: https://github.com/apache/carbondata/pull/3953#discussion_r494046607
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecutorImpl.java
##
@@ -138,6 +145,8 @@ public
RowLevelFilterExecutorImpl(List dimColEvalua
this.exp = exp;
this.tableIdentifier = tableIdentifier;
this.complexDimensionInfoMap = complexDimensionInfoMap;
+this.dateDictionaryGenerator =
+
DirectDictionaryKeyGeneratorFactory.getDirectDictionaryGenerator(DataTypes.DATE);
Review comment:
this issue is not happening with timestamp column?
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/filterexpr/TestInFilter.scala
##
@@ -165,8 +168,27 @@ class TestInFilter extends QueryTest with
BeforeAndAfterAll{
Seq(Row(4, 1.00, 2.00, 3.00)))
}
- override def afterAll(): Unit = {
+ test("test infilter with date, timestamp columns") {
+sql("create table test_table(i int, dt date, ts timestamp) stored as
carbondata")
+sql("insert into test_table select 1, '2020-03-30', '2020-03-30 10:00:00'")
Review comment:
drop table before create to avoid failure if previous test fails
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3953: [CARBONDATA-4008]Fixed IN filter on date column is returning 0 results when 'carbon.push.rowfilters.for.vector' is true
QiangCai commented on a change in pull request #3953: URL: https://github.com/apache/carbondata/pull/3953#discussion_r494090761 ## File path: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecutorImpl.java ## @@ -106,6 +108,11 @@ */ boolean isNaturalSorted; + /** + * date direct dictionary generator + */ + private DirectDictionaryGenerator dateDictionaryGenerator; + public RowLevelFilterExecutorImpl(List dimColEvaluatorInfoList, Review comment: better to use IncludeFilterExecutorImpl as much as possible. ## File path: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecutorImpl.java ## @@ -106,6 +108,11 @@ */ boolean isNaturalSorted; + /** + * date direct dictionary generator + */ + private DirectDictionaryGenerator dateDictionaryGenerator; + public RowLevelFilterExecutorImpl(List dimColEvaluatorInfoList, Review comment: better to use IncludeFilterExecutorImpl for in expression as much as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494972678
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -33,17 +34,30 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
override def beforeAll(): Unit = {
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"dd-MM-")
-drop()
+dropTable()
sql("CREATE TABLE maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
-"deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+ "deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+sql(s"""
+ | CREATE INDEX maintable_index_test
+ | ON TABLE maintable (designation)
+ | AS '${classOf[WaitingIndexFactory].getName}'
+""".stripMargin)
+
sql(s"""LOAD DATA local inpath '$resourcesPath/data_big.csv' INTO TABLE
maintable OPTIONS
|('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+
+sql("CREATE TABLE temp_maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
Review comment:
this is temp_maintable here, this temporary table is used to insert into
maintable in order to make "insert into or insert overwrite" operation in the
following.
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -33,17 +34,30 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
override def beforeAll(): Unit = {
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"dd-MM-")
-drop()
+dropTable()
sql("CREATE TABLE maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
-"deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+ "deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
Review comment:
copy that
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -167,18 +182,24 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
test("insert and create materialized view in progress") {
sql("drop materialized view if exists mv1")
-val query = s"LOAD DATA local inpath '$resourcesPath/data_big.csv' INTO
TABLE maintable " +
-s"OPTIONS('DELIMITER'= ',')"
-val executorService = Executors.newFixedThreadPool(4)
-executorService.submit(new QueryTask(query))
-intercept[UnsupportedOperationException] {
- sql(
-"create materialized view mv1 as " +
+
+val future = AsyncExecutorUtils.runSqlAsync("insert overwrite table
maintable select * from temp_maintable")
+val ex = intercept[UnsupportedOperationException] {
+ sql("create materialized view mv1 as " +
"select timeseries(projectjoindate,'year'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'year')")
-}.getMessage
- .contains("Cannot create mv materialized view table when insert is in
progress on parent table: maintable")
-executorService.shutdown()
-executorService.awaitTermination(2, TimeUnit.HOURS)
+}
+assert(future.get.contains("PASS"))
+assert(ex.getMessage.contains("Cannot create mv when insert overwrite is
in progress on table default_maintable"))
+sql("drop materialized view if exists mv1")
+ }
+
+ test("create materialized view should success when parent table is insert in
progress") {
+sql("drop materialized view if exists mv1")
+
+val future = AsyncExecutorUtils.runSqlAsync("insert into table maintable
select * from temp_maintable")
+sql("create materialized view mv1 as " +
Review comment:
roger that
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
marchpure commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494970696
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -33,17 +34,30 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
override def beforeAll(): Unit = {
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"dd-MM-")
-drop()
+dropTable()
sql("CREATE TABLE maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
-"deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+ "deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+sql(s"""
+ | CREATE INDEX maintable_index_test
+ | ON TABLE maintable (designation)
+ | AS '${classOf[WaitingIndexFactory].getName}'
+""".stripMargin)
+
sql(s"""LOAD DATA local inpath '$resourcesPath/data_big.csv' INTO TABLE
maintable OPTIONS
|('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+
+sql("CREATE TABLE temp_maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
+ "deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+
+sql(s"""LOAD DATA local inpath '$resourcesPath/data_big.csv' INTO TABLE
temp_maintable OPTIONS
+ |('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
}
- def drop(): Unit = {
+ def dropTable(): Unit = {
sql("drop table if exists products")
sql("drop table IF EXISTS main_table")
sql("drop table IF EXISTS maintable")
+sql("drop table IF EXISTS temp_maintable")
Review comment:
tablename 'temp_maintable' is strange.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
marchpure commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494968395
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -167,18 +182,24 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
test("insert and create materialized view in progress") {
sql("drop materialized view if exists mv1")
-val query = s"LOAD DATA local inpath '$resourcesPath/data_big.csv' INTO
TABLE maintable " +
-s"OPTIONS('DELIMITER'= ',')"
-val executorService = Executors.newFixedThreadPool(4)
-executorService.submit(new QueryTask(query))
-intercept[UnsupportedOperationException] {
- sql(
-"create materialized view mv1 as " +
+
+val future = AsyncExecutorUtils.runSqlAsync("insert overwrite table
maintable select * from temp_maintable")
+val ex = intercept[UnsupportedOperationException] {
+ sql("create materialized view mv1 as " +
"select timeseries(projectjoindate,'year'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'year')")
-}.getMessage
- .contains("Cannot create mv materialized view table when insert is in
progress on parent table: maintable")
-executorService.shutdown()
-executorService.awaitTermination(2, TimeUnit.HOURS)
+}
+assert(future.get.contains("PASS"))
+assert(ex.getMessage.contains("Cannot create mv when insert overwrite is
in progress on table default_maintable"))
+sql("drop materialized view if exists mv1")
+ }
+
+ test("create materialized view should success when parent table is insert in
progress") {
+sql("drop materialized view if exists mv1")
+
+val future = AsyncExecutorUtils.runSqlAsync("insert into table maintable
select * from temp_maintable")
+sql("create materialized view mv1 as " +
Review comment:
Need to check result of 'create materialized view'
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -33,17 +34,30 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
override def beforeAll(): Unit = {
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"dd-MM-")
-drop()
+dropTable()
sql("CREATE TABLE maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
-"deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+ "deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
Review comment:
take care about the format
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -33,17 +34,30 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
override def beforeAll(): Unit = {
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"dd-MM-")
-drop()
+dropTable()
sql("CREATE TABLE maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
-"deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+ "deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+sql(s"""
+ | CREATE INDEX maintable_index_test
+ | ON TABLE maintable (designation)
+ | AS '${classOf[WaitingIndexFactory].getName}'
+""".stripMargin)
+
sql(s"""LOAD DATA local inpath '$resourcesPath/data_big.csv' INTO TABLE
maintable OPTIONS
|('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+
+sql("CREATE TABLE temp_maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
Review comment:
create table again?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li removed a comment on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li removed a comment on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-698910832 > Consider a scenario that the user want to insert data and then update the table, the dml commands are sent one by one in time series, but the 2nd(update) cmd is sent by another driver and start to execute while the 1st(insert) is still running, therefore the concurrency scenario described in this PR appears. > If we allow update table who has segment with status 'INSERT_INPROGRESS', the 2nd cmd(update) will execute successfully and the data inserted by the 1st cmd would not updated by the 2nd cmd. > In this case, the result may not be what the user expected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li closed pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li closed pull request #3947: URL: https://github.com/apache/carbondata/pull/3947 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494966092
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatusManager.java
##
@@ -918,17 +893,40 @@ public static Boolean isCompactionInProgress(CarbonTable
carbonTable) {
return compactionInProgress;
}
+ /**
+ * Return true if insert or insert overwrite is in progress for specified
table
+ */
+ public static Boolean isInsertInProgress(CarbonTable carbonTable) {
+if (carbonTable == null) {
+ return false;
+}
+boolean loadInProgress = false;
+String metaPath = carbonTable.getMetadataPath();
+LoadMetadataDetails[] listOfLoadFolderDetailsArray =
SegmentStatusManager.readLoadMetadata(metaPath);
+if (listOfLoadFolderDetailsArray.length != 0) {
+ for (LoadMetadataDetails loadDetail :listOfLoadFolderDetailsArray) {
+SegmentStatus segmentStatus = loadDetail.getSegmentStatus();
+if (segmentStatus == SegmentStatus.INSERT_IN_PROGRESS
+|| segmentStatus == SegmentStatus.INSERT_OVERWRITE_IN_PROGRESS) {
+ loadInProgress =
+ isLoadInProgress(carbonTable.getAbsoluteTableIdentifier(),
+ loadDetail.getLoadName());
+}
+ }
+}
+return loadInProgress;
+ }
+
/**
* Return true if insert overwrite is in progress for specified table
*/
- public static Boolean isOverwriteInProgressInTable(CarbonTable carbonTable) {
+ public static boolean isInsertOverwriteInProgress(CarbonTable carbonTable) {
if (carbonTable == null) {
return false;
}
boolean loadInProgress = false;
String metaPath = carbonTable.getMetadataPath();
-LoadMetadataDetails[] listOfLoadFolderDetailsArray =
-SegmentStatusManager.readLoadMetadata(metaPath);
+LoadMetadataDetails[] listOfLoadFolderDetailsArray =
SegmentStatusManager.readLoadMetadata(metaPath);
Review comment:
done
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatusManager.java
##
@@ -918,17 +893,40 @@ public static Boolean isCompactionInProgress(CarbonTable
carbonTable) {
return compactionInProgress;
}
+ /**
+ * Return true if insert or insert overwrite is in progress for specified
table
+ */
+ public static Boolean isInsertInProgress(CarbonTable carbonTable) {
Review comment:
done
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatus.java
##
@@ -31,6 +31,12 @@
@SerializedName("Success")
SUCCESS("Success"),
+ /**
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-698910832 > Consider a scenario that the user want to insert data and then update the table, the dml commands are sent one by one in time series, but the 2nd(update) cmd is sent by another driver and start to execute while the 1st(insert) is still running, therefore the concurrency scenario described in this PR appears. > If we allow update table who has segment with status 'INSERT_INPROGRESS', the 2nd cmd(update) will execute successfully and the data inserted by the 1st cmd would not updated by the 2nd cmd. > In this case, the result may not be what the user expected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (CARBONDATA-4013) NullPointerException when use ArrowCarbonReader to read carbondata created using orc ,parquet and avro files in SDK
[ https://issues.apache.org/jira/browse/CARBONDATA-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Bhat updated CARBONDATA-4013: Summary: NullPointerException when use ArrowCarbonReader to read carbondata created using orc ,parquet and avro files in SDK (was: NullPointerException when use ArrowCarbonReader to read carbondata created using orc ,parquet and avro files) > NullPointerException when use ArrowCarbonReader to read carbondata created > using orc ,parquet and avro files in SDK > --- > > Key: CARBONDATA-4013 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4013 > Project: CarbonData > Issue Type: Bug > Components: other >Affects Versions: 2.1.0 > Environment: Spark 2.4.5 compiled jars >Reporter: Chetan Bhat >Priority: Major > > when use ArrowCarbonReader to read carbondata created using orc files- > java.lang.NullPointerException > at > org.apache.carbondata.sdk.file.arrow.ArrowUtils.toArrowSchema(ArrowUtils.java:109) > at > org.apache.carbondata.sdk.file.arrow.ArrowConverter.(ArrowConverter.java:45) > at > org.apache.carbondata.sdk.file.ArrowCarbonReader.readArrowBatch(ArrowCarbonReader.java:54) > at > com.apache.spark.LoadFromFiles.testORCFileLoadWithComplexSchemaArrowReader(LoadFromFiles.java:1401) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) > at > com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) > at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) > > when use ArrowCarbonReader to read carbondata created using parquet and avro > files > java.lang.ClassCastException: java.lang.String cannot be cast to > [Ljava.lang.Object; > at > org.apache.carbondata.sdk.file.arrow.StructWriter.setValue(ArrowFieldWriter.java:374) > at > org.apache.carbondata.sdk.file.arrow.ArrowFieldWriter.write(ArrowFieldWriter.java:60) > at > org.apache.carbondata.sdk.file.arrow.StructWriter.setValue(ArrowFieldWriter.java:377) > at > org.apache.carbondata.sdk.file.arrow.ArrowFieldWriter.write(ArrowFieldWriter.java:60) > at > org.apache.carbondata.sdk.file.arrow.ArrowWriter.write(ArrowWriter.java:56) > at > org.apache.carbondata.sdk.file.arrow.ArrowConverter.addToArrowBuffer(ArrowConverter.java:63) > at > org.apache.carbondata.sdk.file.ArrowCarbonReader.readArrowBatch(ArrowCarbonReader.java:56) > at > com.apache.spark.LoadFromFiles.testParquetLoadAndCarbonArrowReader(LoadFromFiles.java:1775) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4013) NullPointerException when use ArrowCarbonReader to read carbondata created using orc ,parquet and avro files in SDK
[ https://issues.apache.org/jira/browse/CARBONDATA-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Bhat updated CARBONDATA-4013: Description: when use ArrowCarbonReader to read carbondata created using orc files in SDK- java.lang.NullPointerException at org.apache.carbondata.sdk.file.arrow.ArrowUtils.toArrowSchema(ArrowUtils.java:109) at org.apache.carbondata.sdk.file.arrow.ArrowConverter.(ArrowConverter.java:45) at org.apache.carbondata.sdk.file.ArrowCarbonReader.readArrowBatch(ArrowCarbonReader.java:54) at com.apache.spark.LoadFromFiles.testORCFileLoadWithComplexSchemaArrowReader(LoadFromFiles.java:1401) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) when use ArrowCarbonReader to read carbondata created using parquet and avro files in SDK java.lang.ClassCastException: java.lang.String cannot be cast to [Ljava.lang.Object; at org.apache.carbondata.sdk.file.arrow.StructWriter.setValue(ArrowFieldWriter.java:374) at org.apache.carbondata.sdk.file.arrow.ArrowFieldWriter.write(ArrowFieldWriter.java:60) at org.apache.carbondata.sdk.file.arrow.StructWriter.setValue(ArrowFieldWriter.java:377) at org.apache.carbondata.sdk.file.arrow.ArrowFieldWriter.write(ArrowFieldWriter.java:60) at org.apache.carbondata.sdk.file.arrow.ArrowWriter.write(ArrowWriter.java:56) at org.apache.carbondata.sdk.file.arrow.ArrowConverter.addToArrowBuffer(ArrowConverter.java:63) at org.apache.carbondata.sdk.file.ArrowCarbonReader.readArrowBatch(ArrowCarbonReader.java:56) at com.apache.spark.LoadFromFiles.testParquetLoadAndCarbonArrowReader(LoadFromFiles.java:1775) was: when use ArrowCarbonReader to read carbondata created using orc files- java.lang.NullPointerException at org.apache.carbondata.sdk.file.arrow.ArrowUtils.toArrowSchema(ArrowUtils.java:109) at org.apache.carbondata.sdk.file.arrow.ArrowConverter.(ArrowConverter.java:45) at org.apache.carbondata.sdk.file.ArrowCarbonReader.readArrowBatch(ArrowCarbonReader.java:54) at com.apache.spark.LoadFromFiles.testORCFileLoadWithComplexSchemaArrowReader(LoadFromFiles.java:1401) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494965993
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/DeleteFromTableEventListener.scala
##
@@ -53,6 +53,12 @@ class DeleteFromTableEventListener extends
OperationEventListener with Logging {
carbonTable
.getDatabaseName
}.${ carbonTable.getTableName }]")
+} else if (!carbonTable.getIndexesMap.isEmpty) {
Review comment:
I think that the original code has clearer error information
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494965168
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/UpdateTablePreEventListener.scala
##
@@ -49,7 +49,6 @@ class UpdateTablePreEventListener extends
OperationEventListener with Logging {
carbonTable
.getDatabaseName
}.${ carbonTable.getTableName }]. Drop all indexes and retry")
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494964752
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -170,17 +146,19 @@ class TestInsertAndOtherCommandConcurrent extends
QueryTest with BeforeAndAfterA
}
test("alter rename table should fail if insert overwrite is in progress") {
-val future = runSqlAsync("insert overwrite table orders select * from
orders_overwrite")
+sql("drop table if exists other_orders")
+val future = AsyncExecutorUtils.runSqlAsync("insert overwrite table orders
select * from orders_overwrite")
val ex = intercept[ConcurrentOperationException] {
- sql("alter table orders rename to other")
+ sql("alter table orders rename to other_orders")
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -92,54 +77,46 @@ class TestInsertAndOtherCommandConcurrent extends QueryTest
with BeforeAndAfterA
private def createTable(tableName: String, schema: StructType): Unit = {
val schemaString = schema.fields.map(x => x.name + " " +
x.dataType.typeName).mkString(", ")
sql(s"CREATE TABLE $tableName ($schemaString) stored as carbondata
tblproperties" +
-
s"('sort_scope'='local_sort','sort_columns'='o_country,o_name,o_phonetype,o_serialname,"
+
-s"o_comment')")
- }
-
- override def afterAll {
-executorService.shutdownNow()
-dropTable()
+
s"('sort_scope'='local_sort','sort_columns'='o_country,o_name,o_phonetype,o_serialname,"
+
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -68,8 +53,8 @@ class TestInsertAndOtherCommandConcurrent extends QueryTest
with BeforeAndAfterA
.mode(SaveMode.Overwrite)
.save()
-sql(s"insert into orders select * from temp_table")
-sql(s"insert into orders_overwrite select * from temp_table")
+sql(s"insert into orders select * from temp_table") // load_0 success
+sql(s"insert into orders_overwrite select * from temp_table") // load_0
success
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -68,8 +53,8 @@ class TestInsertAndOtherCommandConcurrent extends QueryTest
with BeforeAndAfterA
.mode(SaveMode.Overwrite)
.save()
-sql(s"insert into orders select * from temp_table")
-sql(s"insert into orders_overwrite select * from temp_table")
+sql(s"insert into orders select * from temp_table") // load_0 success
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
##
@@ -18,21 +18,18 @@ package org.apache.carbondata.spark.testsuite.iud
import java.io.File
-import org.apache.spark.sql.hive.CarbonRelation
-import org.apache.spark.sql.test.SparkTestQueryExecutor
-import org.apache.spark.sql.test.util.QueryTest
-import org.apache.spark.sql.{CarbonEnv, Row, SaveMode}
-import org.scalatest.BeforeAndAfterAll
-
import
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.index.Segment
import org.apache.carbondata.core.datastore.filesystem.{CarbonFile,
CarbonFileFilter}
import org.apache.carbondata.core.datastore.impl.FileFactory
import org.apache.carbondata.core.mutate.CarbonUpdateUtil
-import org.apache.carbondata.core.util.CarbonProperties
-import org.apache.carbondata.core.util.CarbonUtil
import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.core.util.{CarbonProperties, CarbonUtil}
+import org.apache.spark.sql.hive.CarbonRelation
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494964644
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -122,22 +139,23 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
sql("drop materialized view if exists mv4")
sql("drop materialized view if exists mv5")
}
+
dropMVs
sql(
"create materialized view mv1 as " +
- "select timeseries(projectjoindate,'second'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'second')")
+"select timeseries(projectjoindate,'second'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'second')")
sql(
"create materialized view mv2 as " +
- "select timeseries(projectjoindate,'hour'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'hour')")
+"select timeseries(projectjoindate,'hour'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'hour')")
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (CARBONDATA-4013) NullPointerException when use ArrowCarbonReader to read carbondata created using orc ,parquet and avro files
Chetan Bhat created CARBONDATA-4013: --- Summary: NullPointerException when use ArrowCarbonReader to read carbondata created using orc ,parquet and avro files Key: CARBONDATA-4013 URL: https://issues.apache.org/jira/browse/CARBONDATA-4013 Project: CarbonData Issue Type: Bug Components: other Affects Versions: 2.1.0 Environment: Spark 2.4.5 compiled jars Reporter: Chetan Bhat when use ArrowCarbonReader to read carbondata created using orc files- java.lang.NullPointerException at org.apache.carbondata.sdk.file.arrow.ArrowUtils.toArrowSchema(ArrowUtils.java:109) at org.apache.carbondata.sdk.file.arrow.ArrowConverter.(ArrowConverter.java:45) at org.apache.carbondata.sdk.file.ArrowCarbonReader.readArrowBatch(ArrowCarbonReader.java:54) at com.apache.spark.LoadFromFiles.testORCFileLoadWithComplexSchemaArrowReader(LoadFromFiles.java:1401) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) when use ArrowCarbonReader to read carbondata created using parquet and avro files java.lang.ClassCastException: java.lang.String cannot be cast to [Ljava.lang.Object; at org.apache.carbondata.sdk.file.arrow.StructWriter.setValue(ArrowFieldWriter.java:374) at org.apache.carbondata.sdk.file.arrow.ArrowFieldWriter.write(ArrowFieldWriter.java:60) at org.apache.carbondata.sdk.file.arrow.StructWriter.setValue(ArrowFieldWriter.java:377) at org.apache.carbondata.sdk.file.arrow.ArrowFieldWriter.write(ArrowFieldWriter.java:60) at org.apache.carbondata.sdk.file.arrow.ArrowWriter.write(ArrowWriter.java:56) at org.apache.carbondata.sdk.file.arrow.ArrowConverter.addToArrowBuffer(ArrowConverter.java:63) at org.apache.carbondata.sdk.file.ArrowCarbonReader.readArrowBatch(ArrowCarbonReader.java:56) at com.apache.spark.LoadFromFiles.testParquetLoadAndCarbonArrowReader(LoadFromFiles.java:1775) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494964396
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -18,34 +18,18 @@
package org.apache.carbondata.spark.testsuite.iud
import java.text.SimpleDateFormat
-import java.util
-import java.util.concurrent.{Callable, ExecutorService, Executors, Future}
-
-import scala.collection.JavaConverters._
-
-import org.apache.hadoop.fs.Path
-import org.apache.spark.sql.test.util.QueryTest
-import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.{DataFrame, SaveMode}
-import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.datastore.block.SegmentProperties
-import org.apache.carbondata.core.datastore.page.ColumnPage
import org.apache.carbondata.core.exception.ConcurrentOperationException
-import org.apache.carbondata.core.features.TableOperation
-import org.apache.carbondata.core.index.dev.cgindex.{CoarseGrainIndex,
CoarseGrainIndexFactory}
-import org.apache.carbondata.core.index.dev.{IndexBuilder, IndexWriter}
-import org.apache.carbondata.core.index.{IndexInputSplit, IndexMeta, Segment}
-import org.apache.carbondata.core.indexstore.PartitionSpec
-import org.apache.carbondata.core.metadata.schema.table.{CarbonTable,
IndexSchema}
-import org.apache.carbondata.core.scan.filter.intf.ExpressionType
import org.apache.carbondata.core.util.CarbonProperties
-import org.apache.carbondata.events.Event
+import org.apache.spark.sql.test.util.QueryTest
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{DataFrame, Row, SaveMode}
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/AsyncExecutorUtils.scala
##
@@ -0,0 +1,200 @@
+
+
+
+package org.apache.carbondata.spark.testsuite.iud
Review comment:
done
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala
##
@@ -65,7 +65,7 @@ private[sql] case class CarbonProjectForDeleteCommand(
s"Unsupported delete operation on table containing mixed format
segments")
}
-if (SegmentStatusManager.isLoadInProgressInTable(carbonTable)) {
+if (SegmentStatusManager.isInsertOverwriteInProgress(carbonTable)) {
throw new ConcurrentOperationException(carbonTable, "loading", "data
delete")
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3957: [CARBONDATA-4011] drop index table is failed with table not found exception
CarbonDataQA1 commented on pull request #3957: URL: https://github.com/apache/carbondata/pull/3957#issuecomment-698901000 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4216/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3955: [CARBONDATA-4009] Fix PartialQueries not hitting mv
akashrn5 commented on pull request #3955: URL: https://github.com/apache/carbondata/pull/3955#issuecomment-698900334 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3957: [CARBONDATA-4011] drop index table is failed with table not found exception
CarbonDataQA1 commented on pull request #3957: URL: https://github.com/apache/carbondata/pull/3957#issuecomment-698900072 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2473/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698885408 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2476/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698883873 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4219/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
marchpure commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698882681 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698878002 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2475/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3955: [CARBONDATA-4009] Fix PartialQueries not hitting mv
CarbonDataQA1 commented on pull request #3955: URL: https://github.com/apache/carbondata/pull/3955#issuecomment-69883 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4215/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3955: [CARBONDATA-4009] Fix PartialQueries not hitting mv
CarbonDataQA1 commented on pull request #3955: URL: https://github.com/apache/carbondata/pull/3955#issuecomment-698877603 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2472/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698876449 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4218/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698874146 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2474/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-698872630 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4217/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3959: [CARBONDATA-4010] Doc changes for long strings as bad record.
CarbonDataQA1 commented on pull request #3959: URL: https://github.com/apache/carbondata/pull/3959#issuecomment-698870185 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4214/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3959: [CARBONDATA-4010] Doc changes for long strings as bad record.
CarbonDataQA1 commented on pull request #3959: URL: https://github.com/apache/carbondata/pull/3959#issuecomment-698869602 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2471/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3956: [CARBONDATA-4010] Added documentation for set/unset of longStringColumns.
CarbonDataQA1 commented on pull request #3956: URL: https://github.com/apache/carbondata/pull/3956#issuecomment-698865549 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4212/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (CARBONDATA-4012) Documentations issues.
Prasanna Ravichandran created CARBONDATA-4012: - Summary: Documentations issues. Key: CARBONDATA-4012 URL: https://issues.apache.org/jira/browse/CARBONDATA-4012 Project: CarbonData Issue Type: Bug Reporter: Prasanna Ravichandran Support Array and Struct of all primitive type reading from presto from Carbon tables. This feature details have to be added in the below opensource link: [https://github.com/apache/carbondata/blob/master/docs/prestosql-guide.md] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4012) Documentations issues.
[ https://issues.apache.org/jira/browse/CARBONDATA-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Ravichandran updated CARBONDATA-4012: -- Description: Support Array and Struct of all primitive type reading on presto from Spark Carbon tables. This feature details have to be added in the below opensource link: [https://github.com/apache/carbondata/blob/master/docs/prestosql-guide.md] was: Support Array and Struct of all primitive type reading from presto from Carbon tables. This feature details have to be added in the below opensource link: [https://github.com/apache/carbondata/blob/master/docs/prestosql-guide.md] > Documentations issues. > -- > > Key: CARBONDATA-4012 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4012 > Project: CarbonData > Issue Type: Bug >Reporter: Prasanna Ravichandran >Priority: Minor > > Support Array and Struct of all primitive type reading on presto from Spark > Carbon tables. This feature details have to be added in the below opensource > link: > [https://github.com/apache/carbondata/blob/master/docs/prestosql-guide.md] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4011) drop index table is failed with table not found exception
Mahesh Raju Somalaraju created CARBONDATA-4011: -- Summary: drop index table is failed with table not found exception Key: CARBONDATA-4011 URL: https://issues.apache.org/jira/browse/CARBONDATA-4011 Project: CarbonData Issue Type: Bug Reporter: Mahesh Raju Somalaraju h1. [drop index table is failed with table not found exception|https://github.com/apache/carbondata/pull/3957#] passing index table name instead of passing main table name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3956: [CARBONDATA-4010] Added documentation for set/unset of longStringColumns.
CarbonDataQA1 commented on pull request #3956: URL: https://github.com/apache/carbondata/pull/3956#issuecomment-698834926 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2469/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3955: [CARBONDATA-4009] Fix PartialQueries not hitting mv
CarbonDataQA1 commented on pull request #3955: URL: https://github.com/apache/carbondata/pull/3955#issuecomment-698828553 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2468/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #3955: [CARBONDATA-4009] Fix PartialQueries not hitting mv
Indhumathi27 commented on pull request #3955: URL: https://github.com/apache/carbondata/pull/3955#issuecomment-698828029 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3955: [CARBONDATA-4009] Fix PartialQueries not hitting mv
CarbonDataQA1 commented on pull request #3955: URL: https://github.com/apache/carbondata/pull/3955#issuecomment-698827074 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4211/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3957: [WIP] drop index table is failed with table not found exception
CarbonDataQA1 commented on pull request #3957: URL: https://github.com/apache/carbondata/pull/3957#issuecomment-698825430 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2467/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3957: [WIP] drop index table is failed with table not found exception
CarbonDataQA1 commented on pull request #3957: URL: https://github.com/apache/carbondata/pull/3957#issuecomment-698821503 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4210/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3957: [WIP] drop index table is failed with table not found exception
Indhumathi27 commented on a change in pull request #3957:
URL: https://github.com/apache/carbondata/pull/3957#discussion_r494856214
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/DropTableTest.scala
##
@@ -74,4 +74,17 @@ class DropTableTest extends QueryTest with BeforeAndAfterAll
{
assert(exception.getMessage.contains("Index with name indtestdrop does not
exist"))
sql("drop table if exists testDrop")
}
+
+ test("test drop index command after refresh the index for empty index
table") {
+sql("drop table if exists testDropIndex")
+sql("create table testDropIndex (a string, b string, c string) STORED AS
carbondata")
+sql("create index helloIndex on table testDropIndex (c) AS 'carbondata'
properties" +
+"('table_blocksize'='1')")
+sql("show indexes on testDropIndex").show()
Review comment:
please remove this line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-698818556 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4213/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-698818435 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4208/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-698818078 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2465/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-698817357 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2470/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3956: [CARBONDATA-4010] Added documentation for set/unset of longStringColumns.
CarbonDataQA1 commented on pull request #3956: URL: https://github.com/apache/carbondata/pull/3956#issuecomment-698813517 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2464/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] nihal0107 opened a new pull request #3959: [CARBONDATA-4010] Doc changes for long strings as bad record.
nihal0107 opened a new pull request #3959: URL: https://github.com/apache/carbondata/pull/3959 ### Why is this PR needed? Added documentation change for the handling of long strings(length greater than 32000) as bad record. ### What changes were proposed in this PR? Added documentation change for the handling of long strings(length greater than 32000) as bad record. ### Does this PR introduce any user interface change? - No ### Is any new testcase added? - No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
marchpure commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494842615
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala
##
@@ -65,7 +65,7 @@ private[sql] case class CarbonProjectForDeleteCommand(
s"Unsupported delete operation on table containing mixed format
segments")
}
-if (SegmentStatusManager.isLoadInProgressInTable(carbonTable)) {
+if (SegmentStatusManager.isInsertOverwriteInProgress(carbonTable)) {
throw new ConcurrentOperationException(carbonTable, "loading", "data
delete")
Review comment:
loading->insert overwrite
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -18,34 +18,18 @@
package org.apache.carbondata.spark.testsuite.iud
import java.text.SimpleDateFormat
-import java.util
-import java.util.concurrent.{Callable, ExecutorService, Executors, Future}
-
-import scala.collection.JavaConverters._
-
-import org.apache.hadoop.fs.Path
-import org.apache.spark.sql.test.util.QueryTest
-import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.{DataFrame, SaveMode}
-import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.datastore.block.SegmentProperties
-import org.apache.carbondata.core.datastore.page.ColumnPage
import org.apache.carbondata.core.exception.ConcurrentOperationException
-import org.apache.carbondata.core.features.TableOperation
-import org.apache.carbondata.core.index.dev.cgindex.{CoarseGrainIndex,
CoarseGrainIndexFactory}
-import org.apache.carbondata.core.index.dev.{IndexBuilder, IndexWriter}
-import org.apache.carbondata.core.index.{IndexInputSplit, IndexMeta, Segment}
-import org.apache.carbondata.core.indexstore.PartitionSpec
-import org.apache.carbondata.core.metadata.schema.table.{CarbonTable,
IndexSchema}
-import org.apache.carbondata.core.scan.filter.intf.ExpressionType
import org.apache.carbondata.core.util.CarbonProperties
-import org.apache.carbondata.events.Event
+import org.apache.spark.sql.test.util.QueryTest
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{DataFrame, Row, SaveMode}
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
Review comment:
revert this change
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/AsyncExecutorUtils.scala
##
@@ -0,0 +1,200 @@
+
+
+
+package org.apache.carbondata.spark.testsuite.iud
Review comment:
remove into util package
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat opened a new pull request #3958: [WIP] support presto 333
ajantha-bhat opened a new pull request #3958: URL: https://github.com/apache/carbondata/pull/3958 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat closed pull request #3958: [WIP] support presto 333
ajantha-bhat closed pull request #3958: URL: https://github.com/apache/carbondata/pull/3958 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
marchpure commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r494819647
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatus.java
##
@@ -31,6 +31,12 @@
@SerializedName("Success")
SUCCESS("Success"),
+ /**
Review comment:
revert this change
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatusManager.java
##
@@ -918,17 +893,40 @@ public static Boolean isCompactionInProgress(CarbonTable
carbonTable) {
return compactionInProgress;
}
+ /**
+ * Return true if insert or insert overwrite is in progress for specified
table
+ */
+ public static Boolean isInsertInProgress(CarbonTable carbonTable) {
+if (carbonTable == null) {
+ return false;
+}
+boolean loadInProgress = false;
+String metaPath = carbonTable.getMetadataPath();
+LoadMetadataDetails[] listOfLoadFolderDetailsArray =
SegmentStatusManager.readLoadMetadata(metaPath);
+if (listOfLoadFolderDetailsArray.length != 0) {
+ for (LoadMetadataDetails loadDetail :listOfLoadFolderDetailsArray) {
+SegmentStatus segmentStatus = loadDetail.getSegmentStatus();
+if (segmentStatus == SegmentStatus.INSERT_IN_PROGRESS
+|| segmentStatus == SegmentStatus.INSERT_OVERWRITE_IN_PROGRESS) {
+ loadInProgress =
+ isLoadInProgress(carbonTable.getAbsoluteTableIdentifier(),
+ loadDetail.getLoadName());
+}
+ }
+}
+return loadInProgress;
+ }
+
/**
* Return true if insert overwrite is in progress for specified table
*/
- public static Boolean isOverwriteInProgressInTable(CarbonTable carbonTable) {
+ public static boolean isInsertOverwriteInProgress(CarbonTable carbonTable) {
if (carbonTable == null) {
return false;
}
boolean loadInProgress = false;
String metaPath = carbonTable.getMetadataPath();
-LoadMetadataDetails[] listOfLoadFolderDetailsArray =
-SegmentStatusManager.readLoadMetadata(metaPath);
+LoadMetadataDetails[] listOfLoadFolderDetailsArray =
SegmentStatusManager.readLoadMetadata(metaPath);
Review comment:
revert this change
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatusManager.java
##
@@ -918,17 +893,40 @@ public static Boolean isCompactionInProgress(CarbonTable
carbonTable) {
return compactionInProgress;
}
+ /**
+ * Return true if insert or insert overwrite is in progress for specified
table
+ */
+ public static Boolean isInsertInProgress(CarbonTable carbonTable) {
Review comment:
revert this change
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/DeleteFromTableEventListener.scala
##
@@ -53,6 +53,12 @@ class DeleteFromTableEventListener extends
OperationEventListener with Logging {
carbonTable
.getDatabaseName
}.${ carbonTable.getTableName }]")
+} else if (!carbonTable.getIndexesMap.isEmpty) {
Review comment:
change to
"if (carbonTable.isIndexTable || !carbonTable.getIndexesMap.isEmpty)"
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/UpdateTablePreEventListener.scala
##
@@ -49,7 +49,6 @@ class UpdateTablePreEventListener extends
OperationEventListener with Logging {
carbonTable
.getDatabaseName
}.${ carbonTable.getTableName }]. Drop all indexes and retry")
Review comment:
revert this change
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -68,8 +53,8 @@ class TestInsertAndOtherCommandConcurrent extends QueryTest
with BeforeAndAfterA
.mode(SaveMode.Overwrite)
.save()
-sql(s"insert into orders select * from temp_table")
-sql(s"insert into orders_overwrite select * from temp_table")
+sql(s"insert into orders select * from temp_table") // load_0 success
Review comment:
revert this change
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
##
@@ -18,21 +18,18 @@ package org.apache.carbondata.spark.testsuite.iud
import java.io.File
-import org.apache.spark.sql.hive.CarbonRelation
-import org.apache.spark.sql.test.SparkTestQueryExecutor
-import org.apache.spark.sql.test.util.QueryTest
-import org.apache.spark.sql.{CarbonEnv, Row, SaveMode}
-import org.scalatest.BeforeAndAfterAll
-
import
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.index.Segment
import
[jira] [Updated] (CARBONDATA-4009) PartialQuery not hitting mv
[ https://issues.apache.org/jira/browse/CARBONDATA-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi Muthumurugesh updated CARBONDATA-4009: - Summary: PartialQuery not hitting mv (was: SubQuery not hitting mv) > PartialQuery not hitting mv > --- > > Key: CARBONDATA-4009 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4009 > Project: CarbonData > Issue Type: Bug >Reporter: Indhumathi Muthumurugesh >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] maheshrajus opened a new pull request #3957: [WIP] drop index table is failed with table not found exception
maheshrajus opened a new pull request #3957: URL: https://github.com/apache/carbondata/pull/3957 ### Why is this PR needed? drop index table is failed with table not found exception ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3955: [CARBONDATA-4009] Fix SubQuery not hitting mv
akashrn5 commented on a change in pull request #3955:
URL: https://github.com/apache/carbondata/pull/3955#discussion_r494792185
##
File path:
mv/plan/src/main/scala/org/apache/carbondata/mv/plans/modular/Modularizer.scala
##
@@ -66,6 +68,22 @@ abstract class Modularizer[TreeType <: TreeNode[TreeType]] {
makeupAliasMappings(mplans)
}
+ private def transformPlan(plan: LogicalPlan) = {
Review comment:
please change the method name to something meaningful according to
functionality
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
ShreelekhyaG commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-698763056 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 opened a new pull request #3956: [CARBONDATA-4010] Added documentation for set/unset of longStringColumns.
Karan980 opened a new pull request #3956: URL: https://github.com/apache/carbondata/pull/3956 ### Why is this PR needed? Added documentation for set/unset of longStringColumns. ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No ### Is any new testcase added? - No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (CARBONDATA-4010) "Alter table set tblproperties should support long string columns" and bad record handling of long string data for string columns need to be updated in https://githu
Chetan Bhat created CARBONDATA-4010: --- Summary: "Alter table set tblproperties should support long string columns" and bad record handling of long string data for string columns need to be updated in https://github.com/apache/carbondata/blob/master/docs Key: CARBONDATA-4010 URL: https://issues.apache.org/jira/browse/CARBONDATA-4010 Project: CarbonData Issue Type: Bug Components: docs Affects Versions: 2.1.0 Environment: https://github.com/apache/carbondata/blob/master/docs Reporter: Chetan Bhat "Alter table set tblproperties should support long string columns" and bad record handling of long string data for string columns need to be updated in https://github.com/apache/carbondata/blob/master/docs -- This message was sent by Atlassian Jira (v8.3.4#803005)
