[GitHub] [carbondata] CarbonDataQA1 commented on issue #3603: [CARBONDATA-3679] Optimize local sort performance
CarbonDataQA1 commented on issue #3603: [CARBONDATA-3679] Optimize local sort performance URL: https://github.com/apache/carbondata/pull/3603#issuecomment-586860718 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2017/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command URL: https://github.com/apache/carbondata/pull/3612#issuecomment-586843876 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/313/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3603: [CARBONDATA-3679] Optimize local sort performance
CarbonDataQA1 commented on issue #3603: [CARBONDATA-3679] Optimize local sort performance URL: https://github.com/apache/carbondata/pull/3603#issuecomment-586843945 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/314/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] kevinjmh commented on issue #3603: [CARBONDATA-3679] Optimize local sort performance
kevinjmh commented on issue #3603: [CARBONDATA-3679] Optimize local sort performance URL: https://github.com/apache/carbondata/pull/3603#issuecomment-586837885 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate
Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r380002851
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/command/CreateMaterializedViewCommand.scala
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.mv.extension.command
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.sql._
+import org.apache.spark.sql.execution.command._
+
+import
org.apache.carbondata.common.exceptions.sql.MalformedMaterializedViewException
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datamap.{DataMapProvider,
DataMapStoreManager}
+import org.apache.carbondata.core.datamap.status.DataMapStatusManager
+import
org.apache.carbondata.core.metadata.schema.datamap.{DataMapClassProvider,
DataMapProperty}
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.datamap.DataMapManager
+import org.apache.carbondata.events._
+
+/**
+ * Create Materialized View Command implementation
+ * It will create the MV table, load the MV table (if deferred rebuild is
false),
+ * and register the MV schema in [[DataMapStoreManager]]
+ */
+case class CreateMaterializedViewCommand(
+mvName: String,
+properties: Map[String, String],
+queryString: Option[String],
+ifNotExistsSet: Boolean = false,
+deferredRebuild: Boolean = false)
+ extends AtomicRunnableCommand {
+
+ private val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+ private var dataMapProvider: DataMapProvider = _
+ private var dataMapSchema: DataMapSchema = _
+
+ override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
+
+setAuditInfo(Map("mvName" -> mvName) ++ properties)
+
+val mutableMap = mutable.Map[String, String](properties.toSeq: _*)
+mutableMap.put(DataMapProperty.DEFERRED_REBUILD, deferredRebuild.toString)
+
+dataMapSchema = new DataMapSchema(mvName, DataMapClassProvider.MV.name())
+dataMapSchema.setProperties(mutableMap.asJava)
+dataMapProvider = DataMapManager.get.getDataMapProvider(null,
dataMapSchema, sparkSession)
+if (DataMapStoreManager.getInstance().getAllDataMapSchemas.asScala
+
.exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) {
+ if (!ifNotExistsSet) {
+throw new MalformedMaterializedViewException(
+ s"Materialized view with name ${dataMapSchema.getDataMapName}
already exists")
+ } else {
+return Seq.empty
+ }
+}
+
+val systemFolderLocation: String =
CarbonProperties.getInstance().getSystemFolderLocation
+val operationContext: OperationContext = new OperationContext()
+val preExecEvent = CreateDataMapPreExecutionEvent(sparkSession,
systemFolderLocation, null)
+OperationListenerBus.getInstance().fireEvent(preExecEvent,
operationContext)
+
+dataMapProvider.initMeta(queryString.orNull)
Review comment:
It is disabled inside initMeta
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate
Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r380002484
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala
##
@@ -106,7 +106,8 @@ case class CarbonCreateDataMapCommand(
.exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) {
if (!ifNotExistsSet) {
throw new MalformedDataMapCommandException(
-"DataMap with name " + dataMapSchema.getDataMapName + " already
exists in storage")
+"Materialized view with name " + dataMapSchema.getDataMapName +
Review comment:
I will revert this change. This PR only modifies MV related command, DataMap
and index related command modification will be done in #3614
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command URL: https://github.com/apache/carbondata/pull/3612#discussion_r38879 ## File path: docs/datamap/mv-datamap-guide.md ## @@ -105,7 +105,7 @@ EXPLAIN SELECT a, sum(b) from maintable group by a; property is inherited from parent table, which allows user to provide different tableproperties for child table * MV creation with limit or union all ctas queries is unsupported - * MV datamap does not support Streaming + * MV does not support Streaming Review comment: Yes, I will change doc later in another PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command URL: https://github.com/apache/carbondata/pull/3612#discussion_r37974 ## File path: docs/datamap/mv-datamap-guide.md ## @@ -105,7 +105,7 @@ EXPLAIN SELECT a, sum(b) from maintable group by a; property is inherited from parent table, which allows user to provide different tableproperties for child table * MV creation with limit or union all ctas queries is unsupported - * MV datamap does not support Streaming + * MV does not support Streaming Review comment: I think complete document has to be changed. Using mv instead of datamap This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694]
Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r37285
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala
##
@@ -106,7 +106,8 @@ case class CarbonCreateDataMapCommand(
.exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) {
if (!ifNotExistsSet) {
throw new MalformedDataMapCommandException(
-"DataMap with name " + dataMapSchema.getDataMapName + " already
exists in storage")
+"Materialized view with name " + dataMapSchema.getDataMapName +
Review comment:
Since you have added a command to create mv, is this check in createdatamap
is required?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694]
Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379997174
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/command/CreateMaterializedViewCommand.scala
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.mv.extension.command
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.sql._
+import org.apache.spark.sql.execution.command._
+
+import
org.apache.carbondata.common.exceptions.sql.MalformedMaterializedViewException
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datamap.{DataMapProvider,
DataMapStoreManager}
+import org.apache.carbondata.core.datamap.status.DataMapStatusManager
+import
org.apache.carbondata.core.metadata.schema.datamap.{DataMapClassProvider,
DataMapProperty}
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.datamap.DataMapManager
+import org.apache.carbondata.events._
+
+/**
+ * Create Materialized View Command implementation
+ * It will create the MV table, load the MV table (if deferred rebuild is
false),
+ * and register the MV schema in [[DataMapStoreManager]]
+ */
+case class CreateMaterializedViewCommand(
+mvName: String,
+properties: Map[String, String],
+queryString: Option[String],
+ifNotExistsSet: Boolean = false,
+deferredRebuild: Boolean = false)
+ extends AtomicRunnableCommand {
+
+ private val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+ private var dataMapProvider: DataMapProvider = _
+ private var dataMapSchema: DataMapSchema = _
+
+ override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
+
+setAuditInfo(Map("mvName" -> mvName) ++ properties)
+
+val mutableMap = mutable.Map[String, String](properties.toSeq: _*)
+mutableMap.put(DataMapProperty.DEFERRED_REBUILD, deferredRebuild.toString)
+
+dataMapSchema = new DataMapSchema(mvName, DataMapClassProvider.MV.name())
+dataMapSchema.setProperties(mutableMap.asJava)
+dataMapProvider = DataMapManager.get.getDataMapProvider(null,
dataMapSchema, sparkSession)
+if (DataMapStoreManager.getInstance().getAllDataMapSchemas.asScala
+
.exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) {
+ if (!ifNotExistsSet) {
+throw new MalformedMaterializedViewException(
+ s"Materialized view with name ${dataMapSchema.getDataMapName}
already exists")
+ } else {
+return Seq.empty
+ }
+}
+
+val systemFolderLocation: String =
CarbonProperties.getInstance().getSystemFolderLocation
+val operationContext: OperationContext = new OperationContext()
+val preExecEvent = CreateDataMapPreExecutionEvent(sparkSession,
systemFolderLocation, null)
+OperationListenerBus.getInstance().fireEvent(preExecEvent,
operationContext)
+
+dataMapProvider.initMeta(queryString.orNull)
Review comment:
Lazy datamaps has to be disabled after creation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694]
Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379997083
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/command/CreateMaterializedViewCommand.scala
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.mv.extension.command
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.sql._
+import org.apache.spark.sql.execution.command._
+
+import
org.apache.carbondata.common.exceptions.sql.MalformedMaterializedViewException
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datamap.{DataMapProvider,
DataMapStoreManager}
+import org.apache.carbondata.core.datamap.status.DataMapStatusManager
+import
org.apache.carbondata.core.metadata.schema.datamap.{DataMapClassProvider,
DataMapProperty}
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.datamap.DataMapManager
+import org.apache.carbondata.events._
+
+/**
+ * Create Materialized View Command implementation
+ * It will create the MV table, load the MV table (if deferred rebuild is
false),
+ * and register the MV schema in [[DataMapStoreManager]]
+ */
+case class CreateMaterializedViewCommand(
+mvName: String,
+properties: Map[String, String],
+queryString: Option[String],
+ifNotExistsSet: Boolean = false,
+deferredRebuild: Boolean = false)
+ extends AtomicRunnableCommand {
+
+ private val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+ private var dataMapProvider: DataMapProvider = _
+ private var dataMapSchema: DataMapSchema = _
+
+ override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
+
+setAuditInfo(Map("mvName" -> mvName) ++ properties)
+
+val mutableMap = mutable.Map[String, String](properties.toSeq: _*)
+mutableMap.put(DataMapProperty.DEFERRED_REBUILD, deferredRebuild.toString)
+
+dataMapSchema = new DataMapSchema(mvName, DataMapClassProvider.MV.name())
+dataMapSchema.setProperties(mutableMap.asJava)
+dataMapProvider = DataMapManager.get.getDataMapProvider(null,
dataMapSchema, sparkSession)
+if (DataMapStoreManager.getInstance().getAllDataMapSchemas.asScala
+
.exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) {
+ if (!ifNotExistsSet) {
+throw new MalformedMaterializedViewException(
+ s"Materialized view with name ${dataMapSchema.getDataMapName}
already exists")
+ } else {
+return Seq.empty
+ }
+}
+
+val systemFolderLocation: String =
CarbonProperties.getInstance().getSystemFolderLocation
+val operationContext: OperationContext = new OperationContext()
+val preExecEvent = CreateDataMapPreExecutionEvent(sparkSession,
systemFolderLocation, null)
+OperationListenerBus.getInstance().fireEvent(preExecEvent,
operationContext)
+
Review comment:
Lazy datamaps has to be disabled after creation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694]
Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379997083
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/command/CreateMaterializedViewCommand.scala
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.mv.extension.command
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.sql._
+import org.apache.spark.sql.execution.command._
+
+import
org.apache.carbondata.common.exceptions.sql.MalformedMaterializedViewException
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datamap.{DataMapProvider,
DataMapStoreManager}
+import org.apache.carbondata.core.datamap.status.DataMapStatusManager
+import
org.apache.carbondata.core.metadata.schema.datamap.{DataMapClassProvider,
DataMapProperty}
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.datamap.DataMapManager
+import org.apache.carbondata.events._
+
+/**
+ * Create Materialized View Command implementation
+ * It will create the MV table, load the MV table (if deferred rebuild is
false),
+ * and register the MV schema in [[DataMapStoreManager]]
+ */
+case class CreateMaterializedViewCommand(
+mvName: String,
+properties: Map[String, String],
+queryString: Option[String],
+ifNotExistsSet: Boolean = false,
+deferredRebuild: Boolean = false)
+ extends AtomicRunnableCommand {
+
+ private val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+ private var dataMapProvider: DataMapProvider = _
+ private var dataMapSchema: DataMapSchema = _
+
+ override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
+
+setAuditInfo(Map("mvName" -> mvName) ++ properties)
+
+val mutableMap = mutable.Map[String, String](properties.toSeq: _*)
+mutableMap.put(DataMapProperty.DEFERRED_REBUILD, deferredRebuild.toString)
+
+dataMapSchema = new DataMapSchema(mvName, DataMapClassProvider.MV.name())
+dataMapSchema.setProperties(mutableMap.asJava)
+dataMapProvider = DataMapManager.get.getDataMapProvider(null,
dataMapSchema, sparkSession)
+if (DataMapStoreManager.getInstance().getAllDataMapSchemas.asScala
+
.exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) {
+ if (!ifNotExistsSet) {
+throw new MalformedMaterializedViewException(
+ s"Materialized view with name ${dataMapSchema.getDataMapName}
already exists")
+ } else {
+return Seq.empty
+ }
+}
+
+val systemFolderLocation: String =
CarbonProperties.getInstance().getSystemFolderLocation
+val operationContext: OperationContext = new OperationContext()
+val preExecEvent = CreateDataMapPreExecutionEvent(sparkSession,
systemFolderLocation, null)
+OperationListenerBus.getInstance().fireEvent(preExecEvent,
operationContext)
+
Review comment:
Lazy datamaps has to be disabled after creation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3617: [CARBONDATA-3695] Integrating deep learning framework PyTorch
ajantha-bhat commented on issue #3617: [CARBONDATA-3695] Integrating deep learning framework PyTorch URL: https://github.com/apache/carbondata/pull/3617#issuecomment-586823841 Good. Can we add one documentation "carbondata AI guide" that can explain which all AI engines we support and points to simple example code for each AI engine? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on issue #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi… URL: https://github.com/apache/carbondata/pull/3620#issuecomment-586821309 good finding, it can avoid unnecessary creation of `TableBlockIndexUniqueIdentifierWrapper` if the pruned blocklet is zero size. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379988354
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/blockprune/BlockPruneQueryTestCase.scala
##
@@ -18,16 +18,30 @@ package org.apache.carbondata.spark.testsuite.blockprune
import java.io.DataOutputStream
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.spark.sql.Row
import org.scalatest.BeforeAndAfterAll
import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.util.CarbonProperties
import org.apache.spark.sql.test.util.QueryTest
/**
* This class contains test cases for block prune query
*/
class BlockPruneQueryTestCase extends QueryTest with BeforeAndAfterAll {
val outputPath = s"$resourcesPath/block_prune_test.csv"
+ val MULTI_THREAD_ENABLE_FILES_COUNT = "1";
Review comment:
Still it won't prune multi-thread as other conditions may not satisfy
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379988354
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/blockprune/BlockPruneQueryTestCase.scala
##
@@ -18,16 +18,30 @@ package org.apache.carbondata.spark.testsuite.blockprune
import java.io.DataOutputStream
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.spark.sql.Row
import org.scalatest.BeforeAndAfterAll
import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.util.CarbonProperties
import org.apache.spark.sql.test.util.QueryTest
/**
* This class contains test cases for block prune query
*/
class BlockPruneQueryTestCase extends QueryTest with BeforeAndAfterAll {
val outputPath = s"$resourcesPath/block_prune_test.csv"
+ val MULTI_THREAD_ENABLE_FILES_COUNT = "1";
Review comment:
Still it won't prune multi-thread as other conditions may not satisfy
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379988045
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/blockprune/BlockPruneQueryTestCase.scala
##
@@ -18,16 +18,30 @@ package org.apache.carbondata.spark.testsuite.blockprune
import java.io.DataOutputStream
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.spark.sql.Row
import org.scalatest.BeforeAndAfterAll
import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.util.CarbonProperties
import org.apache.spark.sql.test.util.QueryTest
/**
* This class contains test cases for block prune query
*/
class BlockPruneQueryTestCase extends QueryTest with BeforeAndAfterAll {
val outputPath = s"$resourcesPath/block_prune_test.csv"
+ val MULTI_THREAD_ENABLE_FILES_COUNT = "1";
Review comment:
use camel case for variable names
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379988045
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/blockprune/BlockPruneQueryTestCase.scala
##
@@ -18,16 +18,30 @@ package org.apache.carbondata.spark.testsuite.blockprune
import java.io.DataOutputStream
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.spark.sql.Row
import org.scalatest.BeforeAndAfterAll
import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.util.CarbonProperties
import org.apache.spark.sql.test.util.QueryTest
/**
* This class contains test cases for block prune query
*/
class BlockPruneQueryTestCase extends QueryTest with BeforeAndAfterAll {
val outputPath = s"$resourcesPath/block_prune_test.csv"
+ val MULTI_THREAD_ENABLE_FILES_COUNT = "1";
Review comment:
use small case for variable names
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379987838
##
File path:
core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java
##
@@ -138,8 +138,12 @@ public CarbonTable getTable() {
}
}
int numOfThreadsForPruning = CarbonProperties.getNumOfThreadsForPruning();
+int carbonDriverPruningMultiThreadEnableFilesCount =
Review comment:
Need to add validation for carbon property, if someone configures negative
value. Need to use the default value
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379987630
##
File path:
core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java
##
@@ -138,8 +138,12 @@ public CarbonTable getTable() {
}
}
int numOfThreadsForPruning = CarbonProperties.getNumOfThreadsForPruning();
+int carbonDriverPruningMultiThreadEnableFilesCount =
+Integer.parseInt(CarbonProperties.getInstance().getProperty(
+
CarbonCommonConstants.CARBON_DRIVER_PRUNING_MULTI_THREAD_ENABLE_FILES_COUNT,
Review comment:
Need update the document for the new property added
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3621: [HOTFIX] Support both listfile() and listfile(maxCount) in InsertStag…
ajantha-bhat commented on a change in pull request #3621: [HOTFIX] Support both
listfile() and listfile(maxCount) in InsertStag…
URL: https://github.com/apache/carbondata/pull/3621#discussion_r379986679
##
File path:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##
@@ -1593,6 +1593,14 @@ private CarbonCommonConstants() {
public static final String SUPPORT_DIRECT_QUERY_ON_DATAMAP_DEFAULTVALUE =
"false";
+ // Whether the filename of stage files is in order of time
+ @CarbonProperty
Review comment:
Need to update document also for any new carbon property added
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3621: [HOTFIX] Support both listfile() and listfile(maxCount) in InsertStag…
ajantha-bhat commented on a change in pull request #3621: [HOTFIX] Support both
listfile() and listfile(maxCount) in InsertStag…
URL: https://github.com/apache/carbondata/pull/3621#discussion_r379985844
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertFromStageCommand.scala
##
@@ -502,13 +503,23 @@ case class CarbonInsertFromStageCommand(
): Array[(CarbonFile, CarbonFile)] = {
val dir = FileFactory.getCarbonFile(loadDetailsDir, hadoopConf)
if (dir.exists()) {
- // Only HDFS/OBS/S3 server side can guarantee the files got from
iterator are sorted
- // based on file name so that we can use iterator to get the A and
A.success together
- // without loop all files which can improve performance compared with
list all files.
- // One file and another with '.success', so we need *2 as total and this
value is just
- // an approximate value. For local files, as can it can we not guarantee
the order, we
- // just list all.
- val allFiles = dir.listFiles(false, batchSize * 2)
+ // It is possible that the filename of stage files is not in order of
time,
+ // A switch is used here to judge whether to list files with specify
batch size
+ val CARBON_STAGE_FILENAME_IS_IN_ORDER_OF_TIME =
CarbonProperties.getInstance().getProperty(
Review comment:
change variable name to `isSortedFileNames`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3621: [HOTFIX] Support both listfile() and listfile(maxCount) in InsertStag…
ajantha-bhat commented on a change in pull request #3621: [HOTFIX] Support both
listfile() and listfile(maxCount) in InsertStag…
URL: https://github.com/apache/carbondata/pull/3621#discussion_r379985673
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertFromStageCommand.scala
##
@@ -502,13 +503,23 @@ case class CarbonInsertFromStageCommand(
): Array[(CarbonFile, CarbonFile)] = {
val dir = FileFactory.getCarbonFile(loadDetailsDir, hadoopConf)
if (dir.exists()) {
- // Only HDFS/OBS/S3 server side can guarantee the files got from
iterator are sorted
- // based on file name so that we can use iterator to get the A and
A.success together
- // without loop all files which can improve performance compared with
list all files.
- // One file and another with '.success', so we need *2 as total and this
value is just
- // an approximate value. For local files, as can it can we not guarantee
the order, we
- // just list all.
- val allFiles = dir.listFiles(false, batchSize * 2)
+ // It is possible that the filename of stage files is not in order of
time,
+ // A switch is used here to judge whether to list files with specify
batch size
+ val CARBON_STAGE_FILENAME_IS_IN_ORDER_OF_TIME =
CarbonProperties.getInstance().getProperty(
Review comment:
Also need validation, toBoolean may fail if user sets a non-boolean value
say 123 by mistake.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3621: [HOTFIX] Support both listfile() and listfile(maxCount) in InsertStag…
ajantha-bhat commented on a change in pull request #3621: [HOTFIX] Support both
listfile() and listfile(maxCount) in InsertStag…
URL: https://github.com/apache/carbondata/pull/3621#discussion_r379985373
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertFromStageCommand.scala
##
@@ -502,13 +503,23 @@ case class CarbonInsertFromStageCommand(
): Array[(CarbonFile, CarbonFile)] = {
val dir = FileFactory.getCarbonFile(loadDetailsDir, hadoopConf)
if (dir.exists()) {
- // Only HDFS/OBS/S3 server side can guarantee the files got from
iterator are sorted
- // based on file name so that we can use iterator to get the A and
A.success together
- // without loop all files which can improve performance compared with
list all files.
- // One file and another with '.success', so we need *2 as total and this
value is just
- // an approximate value. For local files, as can it can we not guarantee
the order, we
- // just list all.
- val allFiles = dir.listFiles(false, batchSize * 2)
+ // It is possible that the filename of stage files is not in order of
time,
Review comment:
Can you explain in which scenario it won't be ordered of time ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3622: [CARBONDATA-3702] Clean temp index files in parallel in merge index flow
ajantha-bhat commented on a change in pull request #3622: [CARBONDATA-3702]
Clean temp index files in parallel in merge index flow
URL: https://github.com/apache/carbondata/pull/3622#discussion_r379984005
##
File path:
integration/spark-common/src/main/scala/org/apache/spark/rdd/CarbonMergeFilesRDD.scala
##
@@ -191,6 +193,27 @@ object CarbonMergeFilesRDD {
mergeIndexSize
}
+ /**
+ * delete the file with retry
+ */
+ def deleteFileWithRetry(filePath: String, _retryTimes: Integer): Unit = {
+var retryTimes = _retryTimes
+while (!deleteFile(filePath) && retryTimes > 0) {
Review comment:
why not use `deleteAllCarbonFilesOfDir` ?
Because line 175: you have to loop again to delete the directories. May be
slow on s3
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command URL: https://github.com/apache/carbondata/pull/3612#issuecomment-586813087 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2015/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3603: [CARBONDATA-3679] Optimize local sort performance
CarbonDataQA1 commented on issue #3603: [CARBONDATA-3679] Optimize local sort performance URL: https://github.com/apache/carbondata/pull/3603#issuecomment-586805122 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2014/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command URL: https://github.com/apache/carbondata/pull/3612#issuecomment-586798625 Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/312/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate
Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379966344
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/MVExtension.scala
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.mv.extension
+
+import org.apache.spark.sql.{SparkSession, SparkSessionExtensions, SQLConf}
+import org.apache.spark.sql.catalyst.parser.ParserInterface
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+
+import org.apache.carbondata.mv.rewrite.MVUdf
+import org.apache.carbondata.mv.timeseries.TimeSeriesFunction
+
+// Materialized View extension for Apache Spark
Review comment:
fixed, thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate
Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379966232
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/MVDataMapProvider.scala
##
@@ -207,3 +207,7 @@ class MVDataMapProvider(
override def supportRebuild(): Boolean = true
}
+
+object MVDataMapProvider {
Review comment:
removed now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3603: [CARBONDATA-3679] Optimize local sort performance
CarbonDataQA1 commented on issue #3603: [CARBONDATA-3679] Optimize local sort performance URL: https://github.com/apache/carbondata/pull/3603#issuecomment-586794541 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/311/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] jackylk edited a comment on issue #3623: [CARBONDATA-3692] Support NoneCompression during loading data.
jackylk edited a comment on issue #3623: [CARBONDATA-3692] Support NoneCompression during loading data. URL: https://github.com/apache/carbondata/pull/3623#issuecomment-586716370 please push your changes to original PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3623: [CARBONDATA-3692] Support NoneCompression during loading data.
jackylk commented on issue #3623: [CARBONDATA-3692] Support NoneCompression during loading data. URL: https://github.com/apache/carbondata/pull/3623#issuecomment-586716370 please push to original PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
akashrn5 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command URL: https://github.com/apache/carbondata/pull/3612#discussion_r379887593 ## File path: datamap/mv/core/src/main/spark2.3/org/apache/carbondata/mv/extension/MVRules.scala ## @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.mv.extension + +import org.apache.spark.sql.SparkSession +import org.apache.spark.sql.catalyst.catalog.SessionCatalog +import org.apache.spark.sql.catalyst.optimizer.Optimizer +import org.apache.spark.sql.hive.CarbonMVRules + +class MVRules( Review comment: ok, didn't observe that, you are right This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
akashrn5 commented on a change in pull request #3612: [CARBONDATA-3694]
Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379887248
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/MVExtension.scala
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.mv.extension
+
+import org.apache.spark.sql.{SparkSession, SparkSessionExtensions, SQLConf}
+import org.apache.spark.sql.catalyst.parser.ParserInterface
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+
+import org.apache.carbondata.mv.rewrite.MVUdf
+import org.apache.carbondata.mv.timeseries.TimeSeriesFunction
+
+// Materialized View extension for Apache Spark
Review comment:
comments are duplicated, please remove the non java-style one
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
akashrn5 commented on a change in pull request #3612: [CARBONDATA-3694]
Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379887165
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/MVDataMapProvider.scala
##
@@ -207,3 +207,7 @@ class MVDataMapProvider(
override def supportRebuild(): Boolean = true
}
+
+object MVDataMapProvider {
Review comment:
`object MVDataMapProvider` is still present, its not used, please remove
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
