[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/190


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/190#discussion_r80220071
  
--- Diff: 
integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala
 ---
@@ -68,9 +70,50 @@ object CarbonOptimizer {
  */
 class ResolveCarbonFunctions(relations: Seq[CarbonDecoderRelation])
   extends Rule[LogicalPlan] with PredicateHelper {
-
+  val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
   def apply(plan: LogicalPlan): LogicalPlan = {
-transformCarbonPlan(plan, relations)
+if (relations.nonEmpty && !isOptimized(plan)) {
+  LOGGER.info("Starting to optimize plan")
+  val startTime = System.currentTimeMillis
+  val result = transformCarbonPlan(plan, relations)
+  LOGGER.info("Time taken to optimize plan: " + ( 
System.currentTimeMillis - startTime))
--- End diff --

add this to STATISTICS logger type


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/190#discussion_r80220168
  
--- Diff: 
integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala
 ---
@@ -397,20 +452,38 @@ class ResolveCarbonFunctions(relations: 
Seq[CarbonDecoderRelation])
 case others => others
   }
 
+}
+
+val tStartTime = System.currentTimeMillis
--- End diff --

this variable not used


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/190#discussion_r80219989
  
--- Diff: 
integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala
 ---
@@ -68,9 +70,50 @@ object CarbonOptimizer {
  */
 class ResolveCarbonFunctions(relations: Seq[CarbonDecoderRelation])
   extends Rule[LogicalPlan] with PredicateHelper {
-
+  val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
   def apply(plan: LogicalPlan): LogicalPlan = {
-transformCarbonPlan(plan, relations)
+if (relations.nonEmpty && !isOptimized(plan)) {
+  LOGGER.info("Starting to optimize plan")
+  val startTime = System.currentTimeMillis
+  val result = transformCarbonPlan(plan, relations)
+  LOGGER.info("Time taken to optimize plan: " + ( 
System.currentTimeMillis - startTime))
--- End diff --

change the statement to "Time taken for Carbon Optimizer to optimize"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/190#discussion_r80219576
  
--- Diff: 
integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala
 ---
@@ -87,20 +130,32 @@ class ResolveCarbonFunctions(relations: 
Seq[CarbonDecoderRelation])
   return plan
 }
 var decoder = false
+val mapOfNodes = new java.util.HashMap[LogicalPlan, ExtraNodeInfo]
+fillNodeInfo(plan, mapOfNodes)
 val aliasMap = CarbonAliasDecoderRelation()
 // collect alias information before hand.
 collectInformationOnAttributes(plan, aliasMap)
-val transFormedPlan =
-  plan transformDown {
-case cd: CarbonDictionaryTempDecoder if cd.isOuter =>
-  decoder = true
-  cd
+
+def hasCarbonRelation(currentPlan: LogicalPlan): Boolean = {
+  val extraNodeInfo = mapOfNodes.get(currentPlan)
--- End diff --

Change the name of mapofNodes to mapOfNonCarbonPlanNodes


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...

[QiangCai]

GitHub user QiangCai opened a pull request:

https://github.com/apache/incubator-carbondata/pull/190

[CARBONDATA-268]Improve carbonoptimizer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/incubator-carbondata improveoptimizer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #190


commit 343a3d20fca9bee15a22063b4be8d73d21b81646
Author: c00318382 
Date:   2016-09-22T10:57:00Z

improve carbonoptimizer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---