[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
CarbonDataQA1 commented on pull request #3846: URL: https://github.com/apache/carbondata/pull/3846#issuecomment-662011240 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3458/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
CarbonDataQA1 commented on pull request #3846: URL: https://github.com/apache/carbondata/pull/3846#issuecomment-662012100 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1716/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 opened a new pull request #3857: [CARBONDATA-3914] Fixed issue on reading data from carbon table through hive beeline when no data is present in table.
Karan980 opened a new pull request #3857: URL: https://github.com/apache/carbondata/pull/3857 **Why is this PR needed?** Reading data from empty carbontable through hive beeline was giving "Unable read Carbon Schema" exception, when no data is present in table. **What changes were proposed in this PR?** **Does this PR introduce any user interface change?** No **Is any new testcase added?** No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3857: [CARBONDATA-3914] Fixed issue on reading data from carbon table through hive beeline when no data is present in table.
CarbonDataQA1 commented on pull request #3857: URL: https://github.com/apache/carbondata/pull/3857#issuecomment-662073114 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (CARBONDATA-3902) Query on partition table gives incorrect results after Delete records using CDC
[
https://issues.apache.org/jira/browse/CARBONDATA-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akash R Nilugal resolved CARBONDATA-3902.
-
Fix Version/s: 2.1.0
Resolution: Fixed
> Query on partition table gives incorrect results after Delete records using
> CDC
> ---
>
> Key: CARBONDATA-3902
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3902
> Project: CarbonData
> Issue Type: Bug
>Reporter: Indhumathi Muthumurugesh
>Priority: Major
> Fix For: 2.1.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Steps to Reproduce Issue :
> {code:java}
> import scala.collection.JavaConverters.
> import java.sql.Date
> import org.apache.spark.sql._
> import org.apache.spark.sql.CarbonSession._
> import org.apache.spark.sql.catalyst.TableIdentifier
> import
> org.apache.spark.sql.execution.command.mutation.merge.{CarbonMergeDataSetCommand,
> DeleteAction, InsertAction, InsertInHistoryTableAction, MergeDataSetMatches,
> MergeMatch, UpdateAction, WhenMatched, WhenNotMatched,
> WhenNotMatchedAndExistsOnlyOnTarget}
> import org.apache.spark.sql.functions._
> import org.apache.spark.sql.test.util.QueryTest
> import org.apache.spark.sql.types.{BooleanType, DateType, IntegerType,
> StringType, StructField, StructType}
> import spark.implicits.
> sql("drop table if exists target").show()
> val initframe = spark.createDataFrame(Seq(
> Row("a", "0"),
> Row("b", "1"),
> Row("c", "2"),
> Row("d", "3")
> ).asJava, StructType(Seq(StructField("key", StringType), StructField("value",
> StringType
> initframe.write
> .format("carbondata")
> .option("tableName", "target")
> .option("partitionColumns", "value")
> .mode(SaveMode.Overwrite)
> .save()
> val target = spark.read.format("carbondata").option("tableName",
> "target").load()
> var ccd =
> spark.createDataFrame(Seq(
> Row("a", "10", false, 0),
> Row("a", null, true, 1),
> Row("b", null, true, 2),
> Row("c", null, true, 3),
> Row("c", "20", false, 4),
> Row("c", "200", false, 5),
> Row("e", "100", false, 6)
> ).asJava,
> StructType(Seq(StructField("key", StringType),
> StructField("newValue", StringType),
> StructField("deleted", BooleanType), StructField("time", IntegerType
> ccd.createOrReplaceTempView("changes")
> ccd = sql("SELECT key, latest.newValue as newValue, latest.deleted as deleted
> FROM ( SELECT key, max(struct(time, newValue, deleted)) as latest FROM
> changes GROUP BY key)")
> val updateMap = Map("key" -> "B.key", "value" ->
> "B.newValue").asInstanceOf[Map[Any, Any]]
> val insertMap = Map("key" -> "B.key", "value" ->
> "B.newValue").asInstanceOf[Map[Any, Any]]
> target.as("A").merge(ccd.as("B"), "A.key=B.key").
> whenMatched("B.deleted=true").
> delete().execute(){code}
>
> After this delete operation, partition 0, 1 and 2 should have deleted from
> it.
> Actual:
> {color:#067d17}select * from target order by key;{color}
> {color:#067d17}+---+-+
> |key|value|
> +---+-+
> |a |0 |
> |b |1 |
> |c |2 |
> |d |3 |
> +---+-+{color}
> {color:#067d17}Expected:{color}
> {color:#067d17}+---+-+
> |key|value|
> +---+-+
> |d |3 |
> +---+-+{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [carbondata] ajantha-bhat edited a comment on pull request #3778: [CARBONDATA-3916] Support array with SI
ajantha-bhat edited a comment on pull request #3778: URL: https://github.com/apache/carbondata/pull/3778#issuecomment-662251770 a) I feel we should handle the SI support for all array of primitive not just string. b) Better to discuss the row level SI solution in community once ? @QiangCai , @kunal642 @Indhumathi27 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3849: [CARBONDATA-3913] Table level dateformat, timestampformat support
QiangCai commented on pull request #3849: URL: https://github.com/apache/carbondata/pull/3849#issuecomment-662199441 please check all usage of CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT/CarbonCommonConstants.CARBON_DATE_FORMAT and consider whether it need get table properties or not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] brijoobopanna commented on pull request #3774: [CARBONDATA-3833] Make geoID visible
brijoobopanna commented on pull request #3774: URL: https://github.com/apache/carbondata/pull/3774#issuecomment-662238464 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3778: [CARBONDATA-3916] Support array with SI
ajantha-bhat commented on a change in pull request #3778:
URL: https://github.com/apache/carbondata/pull/3778#discussion_r458547213
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestSIWithComplexArrayType.scala
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.spark.testsuite.secondaryindex
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterEach
+
+import
org.apache.carbondata.spark.testsuite.secondaryindex.TestSecondaryIndexUtils.isFilterPushedDownToSI
+
+class TestSIWithComplexArrayType extends QueryTest with BeforeAndAfterEach {
+
+ override def beforeEach(): Unit = {
+sql("drop table if exists complextable")
+ }
+
+ override def afterEach(): Unit = {
+sql("drop index if exists index_1 on complextable")
+sql("drop table if exists complextable")
+ }
+
+ test("test array on secondary index") {
Review comment:
a) I feel we should handle the SI support for all array of primitive not
just string.
b) Better to discuss the row level SI solution in community once ?
c) Also if multiple SI created for one primitive and one complex column.
query gives 0 rows. Need to handle it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on pull request #3855: [CARBONDATA-3863], after using index service clean the temp data
vikramahuja1001 commented on pull request #3855: URL: https://github.com/apache/carbondata/pull/3855#issuecomment-662251240 @MarvinLitt , please add proper description as to why this PR is needed and what changes are proposed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
akashrn5 commented on pull request #3846: URL: https://github.com/apache/carbondata/pull/3846#issuecomment-662235965 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] jackylk commented on pull request #3770: [CARBONDATA-3829] Support pagination in SDK reader
jackylk commented on pull request #3770: URL: https://github.com/apache/carbondata/pull/3770#issuecomment-662189893 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
asfgit closed pull request #3846: URL: https://github.com/apache/carbondata/pull/3846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3778: [CARBONDATA-3916] Support array with SI
ajantha-bhat commented on pull request #3778: URL: https://github.com/apache/carbondata/pull/3778#issuecomment-662251770 a) I feel we should handle the SI support for all array not just string. b) Better to discuss the row level SI solution in community once ? @QiangCai , @kunal642 @Indhumathi27 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3850: [CARBONDATA-3907]Reuse firePreLoadEvents and firePostLoadEvents methods from CommonLoadUtils to trigger LoadTablePreExecutionEvent and L
akashrn5 commented on pull request #3850: URL: https://github.com/apache/carbondata/pull/3850#issuecomment-661668892 @VenuReddy2103 please make PR heading short like below `Refactor to use CommonLoadUtils API's firePreLoadEvents and firePostLoadEvents to trigger Load pre and post events` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3785: [CARBONDATA-3843] Fix merge index issue in streaming table
asfgit closed pull request #3785: URL: https://github.com/apache/carbondata/pull/3785 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 opened a new pull request #3856: [WIP]cdc improvement
akashrn5 opened a new pull request #3856: URL: https://github.com/apache/carbondata/pull/3856 avro write ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3834: [CARBONDATA-3865] Implementation of delete/update feature in carbondata SDK.
CarbonDataQA1 commented on pull request #3834: URL: https://github.com/apache/carbondata/pull/3834#issuecomment-661705591 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1711/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
akashrn5 commented on a change in pull request #3846:
URL: https://github.com/apache/carbondata/pull/3846#discussion_r457857926
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/merge/CarbonMergeDataSetCommand.scala
##
@@ -175,6 +178,21 @@ case class CarbonMergeDataSetCommand(
LOGGER.error("writing of update status file failed")
throw new CarbonMergeDataSetException("writing of update status file
failed")
}
+ if (carbonTable.isHivePartitionTable) {
+// If load count is 0 and if merge action contains delete operation,
update
+// tableUpdateStatus file name in loadMeta entry
+if (count == 0 && hasDelAction && !tuple._1.isEmpty) {
+ val loadMetaDataDetails =
SegmentStatusManager.readTableStatusFile(CarbonTablePath
+.getTableStatusFilePath(carbonTable.getTablePath))
+ CarbonUpdateUtil.updateTableMetadataStatus(loadMetaDataDetails.map(l
=>
+new Segment(l.getMergedLoadName,
Review comment:
rename variable `l` to `loadMetadataDetails`
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/merge/MergeTestCase.scala
##
@@ -725,6 +725,45 @@ class MergeTestCase extends QueryTest with
BeforeAndAfterAll {
checkAnswer(sql("select * from target order by key"), Seq(Row("c", "200"),
Row("d", "3"), Row("e", "100")))
}
+ test("check the ccd delete with partition") {
+sql("drop table if exists target")
+
+val initframe = sqlContext.sparkSession.createDataFrame(Seq(
+ Row("a", "0"),
+ Row("b", "1"),
+ Row("c", "2"),
+ Row("d", "3")
+).asJava, StructType(Seq(StructField("key", StringType),
StructField("value", StringType
+
+initframe.write
+ .format("carbondata")
+ .option("tableName", "target")
+ .option("partitionColumns", "value")
+ .mode(SaveMode.Overwrite)
+ .save()
+val target = sqlContext.read.format("carbondata").option("tableName",
"target").load()
+var ccd =
+ sqlContext.sparkSession.createDataFrame(Seq(
+Row("a", null, true, 1),
+Row("b", null, true, 2),
+Row("c", null, true, 3),
+Row("e", "100", false, 6)
+ ).asJava,
+StructType(Seq(StructField("key", StringType),
+ StructField("newValue", StringType),
+ StructField("deleted", BooleanType), StructField("time",
IntegerType
+ccd.createOrReplaceTempView("changes")
+
+ccd = sql("SELECT key, latest.newValue as newValue, latest.deleted as
deleted FROM ( SELECT key, max(struct(time, newValue, deleted)) as latest FROM
changes GROUP BY key)")
+
+target.as("A").merge(ccd.as("B"), "A.key=B.key").
+ whenMatched("B.deleted=true").delete().execute()
+
+assert(getDeleteDeltaFileCount("target", "0") == 0)
Review comment:
if delete operation is successful, then it should contain delete delta
file, please check and add proper assert for delete delta file count
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] xubo245 commented on pull request #3770: [CARBONDATA-3829] Support pagination in SDK reader
xubo245 commented on pull request #3770: URL: https://github.com/apache/carbondata/pull/3770#issuecomment-661711709 need one more LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3834: [CARBONDATA-3865] Implementation of delete/update feature in carbondata SDK.
CarbonDataQA1 commented on pull request #3834: URL: https://github.com/apache/carbondata/pull/3834#issuecomment-661706074 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3453/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3850: [CARBONDATA-3907]Refactor to use CommonLoadUtils API's firePreLoadEvents and firePostLoadEvents to trigger Load pre and post events
akashrn5 commented on pull request #3850: URL: https://github.com/apache/carbondata/pull/3850#issuecomment-661791291 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
akashrn5 commented on a change in pull request #3846:
URL: https://github.com/apache/carbondata/pull/3846#discussion_r458018601
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/merge/MergeTestCase.scala
##
@@ -725,6 +725,47 @@ class MergeTestCase extends QueryTest with
BeforeAndAfterAll {
checkAnswer(sql("select * from target order by key"), Seq(Row("c", "200"),
Row("d", "3"), Row("e", "100")))
}
+ test("check the ccd delete with partition") {
Review comment:
```suggestion
test("check the cdc delete with partition") {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
akashrn5 commented on a change in pull request #3846:
URL: https://github.com/apache/carbondata/pull/3846#discussion_r458018787
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/merge/MergeTestCase.scala
##
@@ -725,6 +725,47 @@ class MergeTestCase extends QueryTest with
BeforeAndAfterAll {
checkAnswer(sql("select * from target order by key"), Seq(Row("c", "200"),
Row("d", "3"), Row("e", "100")))
}
+ test("check the ccd delete with partition") {
+sql("drop table if exists target")
+
+val initframe = sqlContext.sparkSession.createDataFrame(Seq(
+ Row("a", "0"),
+ Row("a1", "0"),
+ Row("b", "1"),
+ Row("c", "2"),
+ Row("d", "3")
+).asJava, StructType(Seq(StructField("key", StringType),
StructField("value", StringType
+
+initframe.write
+ .format("carbondata")
+ .option("tableName", "target")
+ .option("partitionColumns", "value")
+ .mode(SaveMode.Overwrite)
+ .save()
+val target = sqlContext.read.format("carbondata").option("tableName",
"target").load()
+var ccd =
Review comment:
```suggestion
var cdc =
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
akashrn5 commented on a change in pull request #3846:
URL: https://github.com/apache/carbondata/pull/3846#discussion_r458018898
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/merge/MergeTestCase.scala
##
@@ -725,6 +725,47 @@ class MergeTestCase extends QueryTest with
BeforeAndAfterAll {
checkAnswer(sql("select * from target order by key"), Seq(Row("c", "200"),
Row("d", "3"), Row("e", "100")))
}
+ test("check the ccd delete with partition") {
+sql("drop table if exists target")
+
+val initframe = sqlContext.sparkSession.createDataFrame(Seq(
+ Row("a", "0"),
+ Row("a1", "0"),
+ Row("b", "1"),
+ Row("c", "2"),
+ Row("d", "3")
+).asJava, StructType(Seq(StructField("key", StringType),
StructField("value", StringType
+
+initframe.write
+ .format("carbondata")
+ .option("tableName", "target")
+ .option("partitionColumns", "value")
+ .mode(SaveMode.Overwrite)
+ .save()
+val target = sqlContext.read.format("carbondata").option("tableName",
"target").load()
+var ccd =
+ sqlContext.sparkSession.createDataFrame(Seq(
+Row("a", null, true, 1),
+Row("a1", null, false, 1),
+Row("b", null, true, 2),
+Row("c", null, true, 3),
+Row("e", "100", false, 6)
+ ).asJava,
+StructType(Seq(StructField("key", StringType),
+ StructField("newValue", StringType),
+ StructField("deleted", BooleanType), StructField("time",
IntegerType
+ccd.createOrReplaceTempView("changes")
+
+ccd = sql("SELECT key, latest.newValue as newValue, latest.deleted as
deleted FROM ( SELECT key, max(struct(time, newValue, deleted)) as latest FROM
changes GROUP BY key)")
Review comment:
same as above
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3841: [CARBONDATA-3899] Drop materialized view when executed concurrently from 4 concurrent client fails in all 4 clients.
VenuReddy2103 commented on a change in pull request #3841:
URL: https://github.com/apache/carbondata/pull/3841#discussion_r458023893
##
File path:
integration/spark/src/main/scala/org/apache/carbondata/view/MVManagerInSpark.scala
##
@@ -48,17 +48,14 @@ class MVManagerInSpark(session: SparkSession) extends
MVManager {
object MVManagerInSpark {
- private val MANAGER_MAP_BY_SESSION =
-new util.HashMap[SparkSession, MVManagerInSpark]()
+ private var viewManager: MVManagerInSpark = null
+ // returns single MVManager instance for all the current sessions.
Review comment:
Probably a MVManagerInSpark instance was maintained per SparkSession to
support multi-tenency. Each user may have an independent and isolated set of
databases/tables/conf etc sharing same sparkcontext. I believe, we still need
this isolation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
akashrn5 commented on a change in pull request #3846:
URL: https://github.com/apache/carbondata/pull/3846#discussion_r458019493
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/merge/MergeTestCase.scala
##
@@ -869,10 +910,15 @@ class MergeTestCase extends QueryTest with
BeforeAndAfterAll {
CarbonProperties.getInstance().addProperty("carbon.enable.auto.load.merge",
"false")
}
- private def getDeleteDeltaFileCount(tableName: String, segment: String): Int
= {
+ private def getDeleteDeltaFileCount(tableName: String,
+ segment: String,
Review comment:
no need to change the method signature, you are already having the
carbonTable object, check from it whether its partition table or not.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3841: [CARBONDATA-3899] Drop materialized view when executed concurrently from 4 concurrent client fails in all 4 clients.
VenuReddy2103 commented on a change in pull request #3841:
URL: https://github.com/apache/carbondata/pull/3841#discussion_r458023893
##
File path:
integration/spark/src/main/scala/org/apache/carbondata/view/MVManagerInSpark.scala
##
@@ -48,17 +48,14 @@ class MVManagerInSpark(session: SparkSession) extends
MVManager {
object MVManagerInSpark {
- private val MANAGER_MAP_BY_SESSION =
-new util.HashMap[SparkSession, MVManagerInSpark]()
+ private var viewManager: MVManagerInSpark = null
+ // returns single MVManager instance for all the current sessions.
Review comment:
Probably MVManagerInSpark instance is maintained per SparkSession to
support multi-tenency. Each user may have an independent and isolated set of
databases/tables/conf etc sharing same sparkcontext. I think, we still need
this isolation. right ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #3841: [CARBONDATA-3899] Drop materialized view when executed concurrently from 4 concurrent client fails in all 4 clients.
ShreelekhyaG commented on a change in pull request #3841:
URL: https://github.com/apache/carbondata/pull/3841#discussion_r458068490
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/rewrite/MVCreateTestCase.scala
##
@@ -955,9 +955,9 @@ class MVCreateTestCase extends QueryTest with
BeforeAndAfterAll {
sql(" insert into mvtable1 select 'n4',12,12")
sql("update mvtable1 set(name) = ('updatedName')").show()
checkAnswer(sql("select count(*) from mvtable1 where name =
'updatedName'"),Seq(Row(4)))
+sql(s"drop materialized view MV11")
Review comment:
Here, `drop materialized view` command is given after table drop and
without `ifexists` check it fails.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
akashrn5 commented on a change in pull request #3846:
URL: https://github.com/apache/carbondata/pull/3846#discussion_r458017848
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/merge/CarbonMergeDataSetCommand.scala
##
@@ -175,6 +178,21 @@ case class CarbonMergeDataSetCommand(
LOGGER.error("writing of update status file failed")
throw new CarbonMergeDataSetException("writing of update status file
failed")
}
+ if (carbonTable.isHivePartitionTable) {
+// If load count is 0 and if merge action contains delete operation,
update
+// tableUpdateStatus file name in loadMeta entry
+if (count == 0 && hasDelAction && !tuple._1.isEmpty) {
+ val loadMetaDataDetails =
SegmentStatusManager.readTableStatusFile(CarbonTablePath
+.getTableStatusFilePath(carbonTable.getTablePath))
+
CarbonUpdateUtil.updateTableMetadataStatus(loadMetaDataDetails.map(loadMetadataDetails
=>
Review comment:
```suggestion
CarbonUpdateUtil.updateTableMetadataStatus(loadMetaDataDetails.map(loadMetadataDetail
=>
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3856: [WIP]cdc improvement
CarbonDataQA1 commented on pull request #3856: URL: https://github.com/apache/carbondata/pull/3856#issuecomment-661812319 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3454/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3856: [WIP]cdc improvement
CarbonDataQA1 commented on pull request #3856: URL: https://github.com/apache/carbondata/pull/3856#issuecomment-661812041 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1712/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #3849: [CARBONDATA-3913] Table level timestampformat support
ShreelekhyaG commented on a change in pull request #3849:
URL: https://github.com/apache/carbondata/pull/3849#discussion_r458050377
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestLoadDataWithDiffTimestampFormat.scala
##
@@ -122,6 +122,35 @@ class TestLoadDataWithDiffTimestampFormat extends
QueryTest with BeforeAndAfterA
}
+ test("test load data with timestamp format set at different levels") {
+sql("DROP TABLE IF EXISTS t3")
+sql(
+ """
+ CREATE TABLE IF NOT EXISTS t3
+ (ID Int, date date, starttime Timestamp, country String,
+ name String, phonetype String, serialname String, salary Int)
+ STORED AS carbondata
TBLPROPERTIES('table_timestampformat'='-MM-dd')
Review comment:
Added new test case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #3841: [CARBONDATA-3899] Drop materialized view when executed concurrently from 4 concurrent client fails in all 4 clients.
ShreelekhyaG commented on a change in pull request #3841:
URL: https://github.com/apache/carbondata/pull/3841#discussion_r458068693
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/MVTest.scala
##
@@ -178,6 +180,13 @@ class MVTest extends QueryTest with BeforeAndAfterAll {
}
}
+ test("test drop mv must fail if not exists") {
+val ex = intercept[MalformedMVCommandException] {
+ sql("drop materialized view MV11")
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
CarbonDataQA1 commented on pull request #3846: URL: https://github.com/apache/carbondata/pull/3846#issuecomment-661748809 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3449/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3846: [CARBONDATA-3902] Fix CDC delete data issue on partition table
CarbonDataQA1 commented on pull request #3846: URL: https://github.com/apache/carbondata/pull/3846#issuecomment-661749375 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1707/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #3774: [CARBONDATA-3833] Make geoID visible
ShreelekhyaG commented on a change in pull request #3774: URL: https://github.com/apache/carbondata/pull/3774#discussion_r458007728 ## File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala ## @@ -482,6 +490,7 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String], null } var createOrderColumns = table.getCreateOrderColumn.asScala + .filterNot(_.getColumnSchema.isSpatialColumn) Review comment: even now `data` and `dataFields` args of the methods don't match when insert with original schema. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3856: [WIP]cdc improvement
CarbonDataQA1 commented on pull request #3856: URL: https://github.com/apache/carbondata/pull/3856#issuecomment-661764814 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3451/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3856: [WIP]cdc improvement
CarbonDataQA1 commented on pull request #3856: URL: https://github.com/apache/carbondata/pull/3856#issuecomment-661766963 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1709/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3778: [CARBONDATA-3916] Support array with SI
CarbonDataQA1 commented on pull request #3778: URL: https://github.com/apache/carbondata/pull/3778#issuecomment-661763876 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3841: [CARBONDATA-3899] Drop materialized view when executed concurrently from 4 concurrent client fails in all 4 clients.
VenuReddy2103 commented on a change in pull request #3841:
URL: https://github.com/apache/carbondata/pull/3841#discussion_r458011089
##
File path:
integration/spark/src/main/scala/org/apache/carbondata/view/MVManagerInSpark.scala
##
@@ -48,17 +48,14 @@ class MVManagerInSpark(session: SparkSession) extends
MVManager {
object MVManagerInSpark {
- private val MANAGER_MAP_BY_SESSION =
-new util.HashMap[SparkSession, MVManagerInSpark]()
+ private var viewManager: MVManagerInSpark = null
+ // returns single MVManager instance for all the current sessions.
def get(session: SparkSession): MVManagerInSpark = {
-var viewManager = MANAGER_MAP_BY_SESSION.get(session)
if (viewManager == null) {
- MANAGER_MAP_BY_SESSION.synchronized {
-viewManager = MANAGER_MAP_BY_SESSION.get(session)
+ this.synchronized {
if (viewManager == null) {
viewManager = new MVManagerInSpark(session)
- MANAGER_MAP_BY_SESSION.put(session, viewManager)
session.sparkContext.addSparkListener(new SparkListener {
Review comment:
If we have multiple sessions, only one session(i.e., the one which
created MVManagerInSpark instance) is added to listeners. Others are not added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3831: [CARBONDATA-3846]Data load issue for boolean column configured as BUCKET_COLUMNS
asfgit closed pull request #3831: URL: https://github.com/apache/carbondata/pull/3831 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (CARBONDATA-3846) Dataload fails for boolean column configured as BUCKET_COLUMNS
[
https://issues.apache.org/jira/browse/CARBONDATA-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajantha Bhat resolved CARBONDATA-3846.
--
Fix Version/s: 2.1.0
Resolution: Fixed
> Dataload fails for boolean column configured as BUCKET_COLUMNS
> --
>
> Key: CARBONDATA-3846
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3846
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 2.0.0
> Environment: Spark 2.3.2, Spark 2.4.5
>Reporter: Chetan Bhat
>Priority: Minor
> Fix For: 2.1.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> *Steps-*
> 0: jdbc:hive2://10.20.255.171:23040/default> create table if not exists
> all_data_types1(*bool_1 boolean*,bool_2 boolean,chinese string,Number
> int,smallNumber smallint,BigNumber bigint,LargeDecimal double,smalldecimal
> float,customdecimal decimal(38,15),words string,smallwords char(8),varwords
> varchar(20),time timestamp,day date,emptyNumber int,emptysmallNumber
> smallint,emptyBigNumber bigint,emptyLargeDecimal double,emptysmalldecimal
> float,emptycustomdecimal decimal(38,38),emptywords string,emptysmallwords
> char(8),emptyvarwords varchar(20)) stored as carbondata TBLPROPERTIES
> ('BUCKET_NUMBER'='1', '*BUCKET_COLUMNS'='bool_1*');
> +--+-+
> |Result|
> +--+-+
> +--+-+
> No rows selected (0.939 seconds)
> 0: jdbc:hive2://10.20.255.171:23040/default> LOAD DATA INPATH
> 'hdfs://hacluster/chetan/datafile_0.csv' into table all_data_types1
> OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='bool_1 ,bool_2
> ,chinese ,Number ,smallNumber ,BigNumber ,LargeDecimal ,smalldecimal
> ,customdecimal,words ,smallwords ,varwords ,time ,day ,emptyNumber
> ,emptysmallNumber ,emptyBigNumber ,emptyLargeDecimal
> ,emptysmalldecimal,emptycustomdecimal ,emptywords ,emptysmallwords
> ,emptyvarwords');
> *Error: java.lang.Exception: DataLoad failure: (state=,code=0)*
>
> *Log-*
> java.lang.Exception: DataLoad failure:
> at
> org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.loadCarbonData(CarbonDataRDDFactory.scala:560)
> at
> org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand.loadData(CarbonLoadDataCommand.scala:207)
> at
> org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand.processData(CarbonLoadDataCommand.scala:168)
> at
> org.apache.spark.sql.execution.command.AtomicRunnableCommand$$anonfun$run$3.apply(package.scala:148)
> at
> org.apache.spark.sql.execution.command.AtomicRunnableCommand$$anonfun$run$3.apply(package.scala:145)
> at
> org.apache.spark.sql.execution.command.Auditable$class.runWithAudit(package.scala:104)
> at
> org.apache.spark.sql.execution.command.AtomicRunnableCommand.runWithAudit(package.scala:141)
> at
> org.apache.spark.sql.execution.command.AtomicRunnableCommand.run(package.scala:145)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
> at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
> at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
> at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3259)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
> at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258)
> at org.apache.spark.sql.Dataset.(Dataset.scala:190)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
> at
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:232)
> at
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:175)
> at
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:185)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
[GitHub] [carbondata] akashrn5 commented on pull request #3850: [CARBONDATA-3907]Refactor to use CommonLoadUtils API's firePreLoadEvents and firePostLoadEvents to trigger Load pre and post events
akashrn5 commented on pull request #3850: URL: https://github.com/apache/carbondata/pull/3850#issuecomment-661876221 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3850: [CARBONDATA-3907]Refactor to use CommonLoadUtils API's firePreLoadEvents and firePostLoadEvents to trigger Load pre and post events
asfgit closed pull request #3850: URL: https://github.com/apache/carbondata/pull/3850 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3849: [CARBONDATA-3913] Table level dateformat, timestampformat support
CarbonDataQA1 commented on pull request #3849: URL: https://github.com/apache/carbondata/pull/3849#issuecomment-661896594 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3456/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3850: [CARBONDATA-3907]Refactor to use CommonLoadUtils API's firePreLoadEvents and firePostLoadEvents to trigger Load pre and post events
CarbonDataQA1 commented on pull request #3850: URL: https://github.com/apache/carbondata/pull/3850#issuecomment-661862560 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3455/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3850: [CARBONDATA-3907]Refactor to use CommonLoadUtils API's firePreLoadEvents and firePostLoadEvents to trigger Load pre and post events
CarbonDataQA1 commented on pull request #3850: URL: https://github.com/apache/carbondata/pull/3850#issuecomment-661863139 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1713/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3850: [CARBONDATA-3907]Refactor to use CommonLoadUtils API's firePreLoadEvents and firePostLoadEvents to trigger Load pre and
VenuReddy2103 commented on a change in pull request #3850: URL: https://github.com/apache/carbondata/pull/3850#discussion_r458109264 ## File path: processing/src/main/java/org/apache/carbondata/processing/loading/events/LoadEvents.java ## @@ -51,12 +51,6 @@ public LoadTablePreExecutionEvent(CarbonTableIdentifier carbonTableIdentifier, this.isOverWriteTable = isOverWriteTable; } Review comment: Have checked and found a listener(MVLoadPostEventListener) for it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (CARBONDATA-3907) Reuse firePreLoadEvents and firePostLoadEvents methods from CommonLoadUtils to trigger LoadTablePreExecutionEvent and LoadTablePostExecutionEvent respectively in al
[ https://issues.apache.org/jira/browse/CARBONDATA-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akash R Nilugal resolved CARBONDATA-3907. - Resolution: Fixed > Reuse firePreLoadEvents and firePostLoadEvents methods from CommonLoadUtils > to trigger LoadTablePreExecutionEvent and LoadTablePostExecutionEvent > respectively in alter table add segment flow > -- > > Key: CARBONDATA-3907 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3907 > Project: CarbonData > Issue Type: Improvement > Components: spark-integration >Affects Versions: 2.0.0 >Reporter: Venugopal Reddy K >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > *[Issue]* > Currently we have 2 different ways of firing LoadTablePreExecutionEvent and > LoadTablePostExecutionEvent. We can reuse firePreLoadEvents and > firePostLoadEvents methods from CommonLoadUtils to trigger > LoadTablePreExecutionEvent and LoadTablePostExecutionEvent respectively in > alter table add segment flow as well. > *[Suggestion]* > Reuse firePreLoadEvents and firePostLoadEvents methods from CommonLoadUtils > to trigger LoadTablePreExecutionEvent and LoadTablePostExecutionEvent > respectively in alter table add segment flow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #3841: [CARBONDATA-3899] Drop materialized view when executed concurrently from 4 concurrent client fails in all 4 clients.
ShreelekhyaG commented on a change in pull request #3841:
URL: https://github.com/apache/carbondata/pull/3841#discussion_r458094901
##
File path: core/src/main/java/org/apache/carbondata/core/view/MVManager.java
##
@@ -143,9 +143,9 @@ public void createSchema(String databaseName, MVSchema
viewSchema)
/**
* Drops the mv schema from storage
*
- * @param viewName index name
+ * @param viewName mv name
*/
- public void deleteSchema(String databaseName, String viewName) throws
IOException {
+ public synchronized void deleteSchema(String databaseName, String viewName)
throws IOException {
Review comment:
added so that another thread could skip some steps. as this occurs only
few times when multiple threads run, removing it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3849: [CARBONDATA-3913] Table level dateformat, timestampformat support
CarbonDataQA1 commented on pull request #3849: URL: https://github.com/apache/carbondata/pull/3849#issuecomment-661896331 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1714/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3841: [CARBONDATA-3899] Drop materialized view when executed concurrently from 4 concurrent client fails in all 4 clients.
CarbonDataQA1 commented on pull request #3841: URL: https://github.com/apache/carbondata/pull/3841#issuecomment-661917823 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1715/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3841: [CARBONDATA-3899] Drop materialized view when executed concurrently from 4 concurrent client fails in all 4 clients.
CarbonDataQA1 commented on pull request #3841: URL: https://github.com/apache/carbondata/pull/3841#issuecomment-661915225 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3457/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
