[jira] [Updated] (CARBONDATA-4111) Filter query having invalid results after add segment to table having SI with Indexserver
[
https://issues.apache.org/jira/browse/CARBONDATA-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SHREELEKHYA GAMPA updated CARBONDATA-4111:
--
Description:
queries to execute:
create table maintable_sdk(a string, b int, c string) stored as carbondata;
insert into maintable_sdk select 'k',1,'k';
insert into maintable_sdk select 'l',2,'l';
CREATE INDEX maintable_si_sdk on table maintable_sdk (c) as 'carbondata';
alter table maintable_sdk add segment
options('path'='hdfs://hacluster/sdkfiles/newsegment/', 'format'='carbon');
spark-sql> select *from maintable_sdk where c='m';
2021-01-27 12:10:54,326 | WARN | IPC Client (653337757) connection to
linux-30/10.19.90.30:22900 from car...@hadoop.com | Unexpected error reading
responses on connection Thread[IPC Client (653337757) connection to
linux-30/10.19.90.30:22900 from car...@hadoop.com,5,main] |
org.apache.hadoop.ipc.Client.run(Client.java:1113)
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.carbondata.core.indexstore.SegmentWrapperContainer.()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:135)
at
org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:58)
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:284)
at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:77)
at
org.apache.hadoop.ipc.RpcWritable$WritableWrapper.readFrom(RpcWritable.java:85)
at org.apache.hadoop.ipc.RpcWritable$Buffer.getValue(RpcWritable.java:187)
at org.apache.hadoop.ipc.RpcWritable$Buffer.newInstance(RpcWritable.java:183)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1223)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1107)
Caused by: java.lang.NoSuchMethodException:
org.apache.carbondata.core.indexstore.SegmentWrapperContainer.()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
... 8 more
2021-01-27 12:10:54,330 | WARN | main | Distributed Segment Pruning failed,
initiating embedded pruning |
org.apache.spark.sql.secondaryindex.joins.BroadCastSIFilterPushJoin$.getFilteredSegments(BroadCastSIFilterPushJoin.scala:349)
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy59.getPrunedSegments(Unknown Source)
at
org.apache.spark.sql.secondaryindex.joins.BroadCastSIFilterPushJoin$.getFilteredSegments(BroadCastSIFilterPushJoin.scala:341)
at
org.apache.spark.sql.secondaryindex.joins.BroadCastSIFilterPushJoin$.getFilteredSegments(BroadCastSIFilterPushJoin.scala:426)
at
org.apache.spark.sql.secondaryindex.joins.BroadCastSIFilterPushJoin.partitions$lzycompute(BroadCastSIFilterPushJoin.scala:80)
at
org.apache.spark.sql.secondaryindex.joins.BroadCastSIFilterPushJoin.partitions(BroadCastSIFilterPushJoin.scala:78)
at
org.apache.spark.sql.secondaryindex.joins.BroadCastSIFilterPushJoin.inputCopy$lzycompute(BroadCastSIFilterPushJoin.scala:94)
at
org.apache.spark.sql.secondaryindex.joins.BroadCastSIFilterPushJoin.inputCopy(BroadCastSIFilterPushJoin.scala:93)
at
org.apache.spark.sql.secondaryindex.joins.BroadCastSIFilterPushJoin.doExecute(BroadCastSIFilterPushJoin.scala:132)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:177)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:173)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:201)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:198)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:173)
at
org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:293)
at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:342)
at
org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:372)
at
org.apache.spark.sql.execution.QueryExecution.hiveResultString(QueryExecution.scala:127)
at
org.apache.spark.sql.hive.thriftserver.SparkSQLDriver$$anonfun$run$1.apply(SparkSQLDriver.scala:66)
at
org.apache.spark.sql.hive.thriftserver.SparkSQLDriver$$anonfun$run$1.apply(SparkSQLDriver.scala:66)
at
org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1$$anonfun$apply$1.apply(SQLExecution.scala:95)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:144)
at
org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:86)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:789)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:63)
at
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4072: [CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation
CarbonDataQA2 commented on pull request #4072: URL: https://github.com/apache/carbondata/pull/4072#issuecomment-768075845 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5357/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4072: [CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation
CarbonDataQA2 commented on pull request #4072: URL: https://github.com/apache/carbondata/pull/4072#issuecomment-768075252 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3597/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4076: [CARBONDATA-4107] Added related MV tables Map to fact table and added lock while touchMDTFile
akashrn5 commented on a change in pull request #4076:
URL: https://github.com/apache/carbondata/pull/4076#discussion_r565042150
##
File path:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##
@@ -1222,6 +1222,11 @@ private CarbonCommonConstants() {
public static final String CARBON_ENABLE_MV_DEFAULT = "true";
+ /**
+ * Related mv table's map for a fact table
+ */
+ public static final String RELATED_MV_TABLES_MAP = "relatedMVTablesMap";
Review comment:
since you are always making lower case in usage, define in lowercase
itself and avoid converting
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/view/CarbonDropMVCommand.scala
##
@@ -38,7 +38,8 @@ case class CarbonDropMVCommand(
databaseNameOption: Option[String],
name: String,
ifExistsSet: Boolean,
-forceDrop: Boolean = false)
+forceDrop: Boolean = false,
+lockAcquiredOnFactTable: String = null)
Review comment:
```suggestion
isLockAcquiredOnFactTable: String = null)
```
please change other places also
##
File path:
integration/spark/src/main/scala/org/apache/carbondata/view/MVHelper.scala
##
@@ -372,6 +380,89 @@ object MVHelper {
updatedName
}
+ /**
+ * Add or modify the MV database and table name to fact table's related mv
tables Map
+ */
+ def addOrModifyMVTablesMap(session: SparkSession,
Review comment:
please add the scenarios where this method is called and how handled
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (CARBONDATA-4111) Filter query having invalid results after add segment to table having SI with Indexserver
[
https://issues.apache.org/jira/browse/CARBONDATA-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SHREELEKHYA GAMPA updated CARBONDATA-4111:
--
Summary: Filter query having invalid results after add segment to table
having SI with Indexserver (was: Filter query having invalid results when add
segment to SI with Indexserver)
> Filter query having invalid results after add segment to table having SI with
> Indexserver
> -
>
> Key: CARBONDATA-4111
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4111
> Project: CarbonData
> Issue Type: Bug
>Reporter: SHREELEKHYA GAMPA
>Priority: Minor
> Attachments: addseg_si_is.png
>
>
> queries to execute:
> create table maintable_sdk(a string, b int, c string) stored as carbondata;
> insert into maintable_sdk select 'k',1,'k';
> insert into maintable_sdk select 'l',2,'l';
> CREATE INDEX maintable_si_sdk on table maintable_sdk (c) as 'carbondata';
> alter table maintable_sdk add segment
> options('path'='hdfs://hacluster/sdkfiles/newsegment/', 'format'='carbon');
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CARBONDATA-4111) Filter query having invalid results when add segment to SI with Indexserver
SHREELEKHYA GAMPA created CARBONDATA-4111:
-
Summary: Filter query having invalid results when add segment to
SI with Indexserver
Key: CARBONDATA-4111
URL: https://issues.apache.org/jira/browse/CARBONDATA-4111
Project: CarbonData
Issue Type: Bug
Reporter: SHREELEKHYA GAMPA
Attachments: addseg_si_is.png
queries to execute:
create table maintable_sdk(a string, b int, c string) stored as carbondata;
insert into maintable_sdk select 'k',1,'k';
insert into maintable_sdk select 'l',2,'l';
CREATE INDEX maintable_si_sdk on table maintable_sdk (c) as 'carbondata';
alter table maintable_sdk add segment
options('path'='hdfs://hacluster/sdkfiles/newsegment/', 'format'='carbon');
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4111) Filter query having invalid results when add segment to SI with Indexserver
[
https://issues.apache.org/jira/browse/CARBONDATA-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SHREELEKHYA GAMPA updated CARBONDATA-4111:
--
Attachment: addseg_si_is.png
> Filter query having invalid results when add segment to SI with Indexserver
> ---
>
> Key: CARBONDATA-4111
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4111
> Project: CarbonData
> Issue Type: Bug
>Reporter: SHREELEKHYA GAMPA
>Priority: Minor
> Attachments: addseg_si_is.png
>
>
> queries to execute:
> create table maintable_sdk(a string, b int, c string) stored as carbondata;
> insert into maintable_sdk select 'k',1,'k';
> insert into maintable_sdk select 'l',2,'l';
> CREATE INDEX maintable_si_sdk on table maintable_sdk (c) as 'carbondata';
> alter table maintable_sdk add segment
> options('path'='hdfs://hacluster/sdkfiles/newsegment/', 'format'='carbon');
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-4097) Direct filling of column vector is not allowed for a alter table, because it uses RestructureBasedCollector. However ColumnVectors were initialized as ColumnVectorW
[ https://issues.apache.org/jira/browse/CARBONDATA-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-4097. -- Fix Version/s: (was: 2.0.1) 2.1.1 Resolution: Fixed > Direct filling of column vector is not allowed for a alter table, because it > uses RestructureBasedCollector. However ColumnVectors were initialized as > ColumnVectorWrapperDirect even for alter table. > -- > > Key: CARBONDATA-4097 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4097 > Project: CarbonData > Issue Type: Bug > Components: core >Affects Versions: 2.1.0 >Reporter: Karan >Priority: Major > Fix For: 2.1.1 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > ColumnVector for alter tables should not be initialized as > ColumnVectorWrapperDirect because direct filling is not allowed for alter > table. It should be initialized as ColumnVectorWrapper. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] asfgit closed pull request #4062: [CARBONDATA-4097] ColumnVectors should not be initialized as ColumnVectorWrapperDirect for alter tables.
asfgit closed pull request #4062: URL: https://github.com/apache/carbondata/pull/4062 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #4062: [CARBONDATA-4097] ColumnVectors should not be initialized as ColumnVectorWrapperDirect for alter tables.
kunal642 commented on pull request #4062: URL: https://github.com/apache/carbondata/pull/4062#issuecomment-768022820 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4074: [CARBONDATA-4109] Improve carbondata coverage for presto-integration code
CarbonDataQA2 commented on pull request #4074: URL: https://github.com/apache/carbondata/pull/4074#issuecomment-767869836 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3596/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4074: [CARBONDATA-4109] Improve carbondata coverage for presto-integration code
CarbonDataQA2 commented on pull request #4074: URL: https://github.com/apache/carbondata/pull/4074#issuecomment-767869528 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5356/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on pull request #4074: [CARBONDATA-4109] Improve carbondata coverage for presto-integration code
akkio-97 commented on pull request #4074: URL: https://github.com/apache/carbondata/pull/4074#issuecomment-767821108 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4074: [CARBONDATA-4109] Improve carbondata coverage for presto-integration code
CarbonDataQA2 commented on pull request #4074: URL: https://github.com/apache/carbondata/pull/4074#issuecomment-767816579 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3595/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4074: [CARBONDATA-4109] Improve carbondata coverage for presto-integration code
CarbonDataQA2 commented on pull request #4074: URL: https://github.com/apache/carbondata/pull/4074#issuecomment-767815975 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5355/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
