[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert. URL: https://github.com/apache/carbondata/pull/3538#issuecomment-580603426 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1815/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries
Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532]
Support Query Rollup for MV TimeSeries Queries
URL: https://github.com/apache/carbondata/pull/3495#discussion_r373327070
##
File path:
datamap/mv/core/src/test/scala/org/apache/carbondata/mv/timeseries/TestMVTimeSeriesQueryRollUp.scala
##
@@ -0,0 +1,259 @@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.carbondata.mv.timeseries
+
+ import org.apache.spark.sql.test.util.QueryTest
+ import org.scalatest.BeforeAndAfterAll
+
+ import org.apache.carbondata.mv.rewrite.TestUtil
+
+ class TestMVTimeSeriesQueryRollUp extends QueryTest with BeforeAndAfterAll {
+
+override def beforeAll(): Unit = {
+ drop()
+ createTable()
+ loadData("maintable")
+}
+
+override def afterAll(): Unit = {
+ drop()
+}
+
+test("test timeseries query rollup with simple projection") {
+ val result = sql("select timeseries(projectjoindate,'day'),projectcode
from maintable")
+ sql("drop datamap if exists datamap1")
+ sql("drop datamap if exists datamap2")
+ sql(
+"create datamap datamap1 on table maintable using 'mv' as " +
+"select timeseries(projectjoindate,'second'),projectcode from
maintable")
+ sql(
+"create datamap datamap2 on table maintable using 'mv' as " +
+"select timeseries(projectjoindate,'hour'),projectcode from maintable")
+ val df = sql("select timeseries(projectjoindate,'day'),projectcode from
maintable")
+ assert(TestUtil.verifyMVDataMap(df.queryExecution.optimizedPlan,
"datamap2"))
+ checkAnswer(result,df)
+ sql("drop datamap if exists datamap1")
+ sql("drop datamap if exists datamap2")
+}
+
+test("test timeseries query rollup with simple projection with group by -
scenario-1") {
+ val result = sql("select timeseries(projectjoindate,'day'),projectcode
from maintable group by timeseries(projectjoindate,'day'),projectcode")
+ sql("drop datamap if exists datamap1")
+ sql("drop datamap if exists datamap2")
+ sql(
+"create datamap datamap1 on table maintable using 'mv' as " +
+"select timeseries(projectjoindate,'second'),projectcode from
maintable group by timeseries(projectjoindate,'second'),projectcode")
+ sql(
+"create datamap datamap2 on table maintable using 'mv' as " +
+"select timeseries(projectjoindate,'hour'),projectcode from maintable
group by timeseries(projectjoindate,'hour'),projectcode")
+ val df = sql("select timeseries(projectjoindate,'day'),projectcode from
maintable group by timeseries(projectjoindate,'day'),projectcode")
+ assert(TestUtil.verifyMVDataMap(df.queryExecution.optimizedPlan,
"datamap2"))
+ checkAnswer(result,df)
+ sql("drop datamap if exists datamap1")
+ sql("drop datamap if exists datamap2")
+}
+
+test("test timeseries query rollup with simple projection with group by -
scenario-2") {
+ val result = sql("select
timeseries(projectjoindate,'day'),sum(projectcode) from maintable group by
timeseries(projectjoindate,'day')")
+ sql("drop datamap if exists datamap1")
+ sql("drop datamap if exists datamap2")
+ sql(
+"create datamap datamap1 on table maintable using 'mv' as " +
+"select timeseries(projectjoindate,'second'),sum(projectcode) from
maintable group by timeseries(projectjoindate,'second')")
+ sql(
+"create datamap datamap2 on table maintable using 'mv' as " +
+"select timeseries(projectjoindate,'hour'),sum(projectcode) from
maintable group by timeseries(projectjoindate,'hour')")
+ val df =sql("select timeseries(projectjoindate,'day'),sum(projectcode)
from maintable group by timeseries(projectjoindate,'day')")
+ assert(TestUtil.verifyMVDataMap(df.queryExecution.optimizedPlan,
"datamap2"))
+ checkAnswer(result,df)
+ sql("drop datamap if exists datamap1")
+ sql("drop datamap if exists datamap2")
+}
+
+test("test timeseries query rollup with simple projection with filter") {
+ val result = sql("select timeseries(projectjoindate,'day'),projectcode
from maintable where pr
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries
Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries URL: https://github.com/apache/carbondata/pull/3495#discussion_r373326774 ## File path: datamap/mv/plan/src/main/scala/org/apache/carbondata/mv/plans/modular/ModularPlan.scala ## @@ -96,6 +96,24 @@ abstract class ModularPlan _rewritten } + private var _rolledup: Boolean = false Review comment: changed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries
Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532]
Support Query Rollup for MV TimeSeries Queries
URL: https://github.com/apache/carbondata/pull/3495#discussion_r373326873
##
File path:
datamap/mv/core/src/test/scala/org/apache/carbondata/mv/timeseries/TestMVTimeSeriesQueryRollUp.scala
##
@@ -0,0 +1,259 @@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.carbondata.mv.timeseries
+
+ import org.apache.spark.sql.test.util.QueryTest
+ import org.scalatest.BeforeAndAfterAll
+
+ import org.apache.carbondata.mv.rewrite.TestUtil
+
+ class TestMVTimeSeriesQueryRollUp extends QueryTest with BeforeAndAfterAll {
+
+override def beforeAll(): Unit = {
+ drop()
+ createTable()
+ loadData("maintable")
+}
+
+override def afterAll(): Unit = {
+ drop()
+}
+
+test("test timeseries query rollup with simple projection") {
+ val result = sql("select timeseries(projectjoindate,'day'),projectcode
from maintable")
+ sql("drop datamap if exists datamap1")
+ sql("drop datamap if exists datamap2")
+ sql(
+"create datamap datamap1 on table maintable using 'mv' as " +
+"select timeseries(projectjoindate,'second'),projectcode from
maintable")
+ sql(
+"create datamap datamap2 on table maintable using 'mv' as " +
+"select timeseries(projectjoindate,'hour'),projectcode from maintable")
+ val df = sql("select timeseries(projectjoindate,'day'),projectcode from
maintable")
+ assert(TestUtil.verifyMVDataMap(df.queryExecution.optimizedPlan,
"datamap2"))
+ checkAnswer(result,df)
+ sql("drop datamap if exists datamap1")
+ sql("drop datamap if exists datamap2")
+}
+
+test("test timeseries query rollup with simple projection with group by -
scenario-1") {
+ val result = sql("select timeseries(projectjoindate,'day'),projectcode
from maintable group by timeseries(projectjoindate,'day'),projectcode")
+ sql("drop datamap if exists datamap1")
+ sql("drop datamap if exists datamap2")
+ sql(
+"create datamap datamap1 on table maintable using 'mv' as " +
+"select timeseries(projectjoindate,'second'),projectcode from
maintable group by timeseries(projectjoindate,'second'),projectcode")
+ sql(
+"create datamap datamap2 on table maintable using 'mv' as " +
+"select timeseries(projectjoindate,'hour'),projectcode from maintable
group by timeseries(projectjoindate,'hour'),projectcode")
+ val df = sql("select timeseries(projectjoindate,'day'),projectcode from
maintable group by timeseries(projectjoindate,'day'),projectcode")
+ assert(TestUtil.verifyMVDataMap(df.queryExecution.optimizedPlan,
"datamap2"))
Review comment:
Added scenarios and verified. Please check
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries
Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries URL: https://github.com/apache/carbondata/pull/3495#discussion_r373326730 ## File path: datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Navigator.scala ## @@ -206,4 +312,59 @@ private[mv] class Navigator(catalog: SummaryDatasetCatalog, session: MVSession) } true } + + /** + * Replace the granularity in the plan Review comment: Added description. Please check This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries
Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532]
Support Query Rollup for MV TimeSeries Queries
URL: https://github.com/apache/carbondata/pull/3495#discussion_r373326636
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/datamap/MVHelper.scala
##
@@ -830,18 +831,186 @@ object MVHelper {
*/
def rewriteWithMVTable(rewrittenPlan: ModularPlan, rewrite: QueryRewrite):
ModularPlan = {
if (rewrittenPlan.find(_.rewritten).isDefined) {
- val updatedDataMapTablePlan = rewrittenPlan transform {
+ var updatedDataMapTablePlan = rewrittenPlan transform {
case s: Select =>
MVHelper.updateDataMap(s, rewrite)
case g: GroupBy =>
MVHelper.updateDataMap(g, rewrite)
}
+ if (rewrittenPlan.rolledUp) {
+// If the rewritten query is rolled up, then rewrite the query based
on the original modular
+// plan. Make a new outputList based on original modular plan and wrap
rewritten plan with
+// select & group-by nodes with new outputList.
+
+// For example:
+// Given User query:
+// SELECT timeseries(col,'day') from maintable group by
timeseries(col,'day')
+// If plan is rewritten as per 'hour' granularity of datamap1,
+// then rewritten query will be like,
+// SELECT datamap1_table.`UDF:timeseries_projectjoindate_hour` AS
`UDF:timeseries
+// (projectjoindate, hour)`
+// FROM
+// default.datamap1_table
+// GROUP BY datamap1_table.`UDF:timeseries_projectjoindate_hour`
+//
+// Now, rewrite the rewritten plan as per the 'day' granularity
+// SELECT timeseries(gen_subsumer_0.`UDF:timeseries(projectjoindate,
hour)`,'day' ) AS
+// `UDF:timeseries(projectjoindate, day)`
+// FROM
+// (SELECT datamap2_table.`UDF:timeseries_projectjoindate_hour` AS
`UDF:timeseries
+// (projectjoindate, hour)`
+// FROM
+//default.datamap2_table
+// GROUP BY datamap2_table.`UDF:timeseries_projectjoindate_hour`)
gen_subsumer_0
+// GROUP BY timeseries(gen_subsumer_0.`UDF:timeseries(projectjoindate,
hour)`,'day' )
+rewrite.modularPlan match {
+ case select: Select =>
+val outputList = select.outputList
+val rolledUpOutputList =
updatedDataMapTablePlan.asInstanceOf[Select].outputList
+var finalOutputList: Seq[NamedExpression] = Seq.empty
+val mapping = outputList zip rolledUpOutputList
+val newSubsme = rewrite.newSubsumerName()
+
+for ((s, d) <- mapping) {
+ var name: String = getAliasName(d)
+ s match {
+case a@Alias(scalaUdf: ScalaUDF, aliasName) =>
+ if (scalaUdf.function.isInstanceOf[TimeSeriesFunction]) {
+val newName = newSubsme + ".`" + name + "`"
+val transformedUdf = transformTimeSeriesUdf(scalaUdf,
newName)
+finalOutputList = finalOutputList.:+(Alias(transformedUdf,
aliasName)(a.exprId,
+ a.qualifier).asInstanceOf[NamedExpression])
+ }
+case Alias(attr: AttributeReference, _) =>
+ finalOutputList =
finalOutputList.:+(AttributeReference(name, attr
+.dataType)(
+exprId = attr.exprId,
+qualifier = Some(newSubsme)))
+case attr: AttributeReference =>
+ finalOutputList =
finalOutputList.:+(AttributeReference(name, attr
+.dataType)(
+exprId = attr.exprId,
+qualifier = Some(newSubsme)))
+ }
+}
+val tChildren = new collection.mutable.ArrayBuffer[ModularPlan]()
+val tAliasMap = new collection.mutable.HashMap[Int, String]()
+
+val sel_plan = select.copy(outputList = finalOutputList,
+ inputList = finalOutputList,
+ predicateList = Seq.empty)
+tChildren += sel_plan
+tAliasMap += (tChildren.indexOf(sel_plan) -> newSubsme)
+updatedDataMapTablePlan = select.copy(outputList = finalOutputList,
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries
Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532]
Support Query Rollup for MV TimeSeries Queries
URL: https://github.com/apache/carbondata/pull/3495#discussion_r373326593
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/datamap/MVHelper.scala
##
@@ -830,18 +831,186 @@ object MVHelper {
*/
def rewriteWithMVTable(rewrittenPlan: ModularPlan, rewrite: QueryRewrite):
ModularPlan = {
if (rewrittenPlan.find(_.rewritten).isDefined) {
- val updatedDataMapTablePlan = rewrittenPlan transform {
+ var updatedDataMapTablePlan = rewrittenPlan transform {
case s: Select =>
MVHelper.updateDataMap(s, rewrite)
case g: GroupBy =>
MVHelper.updateDataMap(g, rewrite)
}
+ if (rewrittenPlan.rolledUp) {
+// If the rewritten query is rolled up, then rewrite the query based
on the original modular
+// plan. Make a new outputList based on original modular plan and wrap
rewritten plan with
+// select & group-by nodes with new outputList.
+
+// For example:
+// Given User query:
+// SELECT timeseries(col,'day') from maintable group by
timeseries(col,'day')
+// If plan is rewritten as per 'hour' granularity of datamap1,
+// then rewritten query will be like,
+// SELECT datamap1_table.`UDF:timeseries_projectjoindate_hour` AS
`UDF:timeseries
+// (projectjoindate, hour)`
+// FROM
+// default.datamap1_table
+// GROUP BY datamap1_table.`UDF:timeseries_projectjoindate_hour`
+//
+// Now, rewrite the rewritten plan as per the 'day' granularity
+// SELECT timeseries(gen_subsumer_0.`UDF:timeseries(projectjoindate,
hour)`,'day' ) AS
+// `UDF:timeseries(projectjoindate, day)`
+// FROM
+// (SELECT datamap2_table.`UDF:timeseries_projectjoindate_hour` AS
`UDF:timeseries
+// (projectjoindate, hour)`
+// FROM
+//default.datamap2_table
+// GROUP BY datamap2_table.`UDF:timeseries_projectjoindate_hour`)
gen_subsumer_0
+// GROUP BY timeseries(gen_subsumer_0.`UDF:timeseries(projectjoindate,
hour)`,'day' )
+rewrite.modularPlan match {
+ case select: Select =>
+val outputList = select.outputList
+val rolledUpOutputList =
updatedDataMapTablePlan.asInstanceOf[Select].outputList
+var finalOutputList: Seq[NamedExpression] = Seq.empty
+val mapping = outputList zip rolledUpOutputList
+val newSubsme = rewrite.newSubsumerName()
+
+for ((s, d) <- mapping) {
+ var name: String = getAliasName(d)
+ s match {
+case a@Alias(scalaUdf: ScalaUDF, aliasName) =>
+ if (scalaUdf.function.isInstanceOf[TimeSeriesFunction]) {
+val newName = newSubsme + ".`" + name + "`"
+val transformedUdf = transformTimeSeriesUdf(scalaUdf,
newName)
+finalOutputList = finalOutputList.:+(Alias(transformedUdf,
aliasName)(a.exprId,
+ a.qualifier).asInstanceOf[NamedExpression])
+ }
+case Alias(attr: AttributeReference, _) =>
+ finalOutputList =
finalOutputList.:+(AttributeReference(name, attr
+.dataType)(
+exprId = attr.exprId,
+qualifier = Some(newSubsme)))
+case attr: AttributeReference =>
+ finalOutputList =
finalOutputList.:+(AttributeReference(name, attr
+.dataType)(
+exprId = attr.exprId,
+qualifier = Some(newSubsme)))
+ }
+}
+val tChildren = new collection.mutable.ArrayBuffer[ModularPlan]()
+val tAliasMap = new collection.mutable.HashMap[Int, String]()
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries
Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532]
Support Query Rollup for MV TimeSeries Queries
URL: https://github.com/apache/carbondata/pull/3495#discussion_r373326534
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/datamap/MVHelper.scala
##
@@ -830,18 +831,186 @@ object MVHelper {
*/
def rewriteWithMVTable(rewrittenPlan: ModularPlan, rewrite: QueryRewrite):
ModularPlan = {
if (rewrittenPlan.find(_.rewritten).isDefined) {
- val updatedDataMapTablePlan = rewrittenPlan transform {
+ var updatedDataMapTablePlan = rewrittenPlan transform {
case s: Select =>
MVHelper.updateDataMap(s, rewrite)
case g: GroupBy =>
MVHelper.updateDataMap(g, rewrite)
}
+ if (rewrittenPlan.rolledUp) {
+// If the rewritten query is rolled up, then rewrite the query based
on the original modular
+// plan. Make a new outputList based on original modular plan and wrap
rewritten plan with
+// select & group-by nodes with new outputList.
+
+// For example:
+// Given User query:
+// SELECT timeseries(col,'day') from maintable group by
timeseries(col,'day')
+// If plan is rewritten as per 'hour' granularity of datamap1,
+// then rewritten query will be like,
+// SELECT datamap1_table.`UDF:timeseries_projectjoindate_hour` AS
`UDF:timeseries
+// (projectjoindate, hour)`
+// FROM
+// default.datamap1_table
+// GROUP BY datamap1_table.`UDF:timeseries_projectjoindate_hour`
+//
+// Now, rewrite the rewritten plan as per the 'day' granularity
+// SELECT timeseries(gen_subsumer_0.`UDF:timeseries(projectjoindate,
hour)`,'day' ) AS
+// `UDF:timeseries(projectjoindate, day)`
+// FROM
+// (SELECT datamap2_table.`UDF:timeseries_projectjoindate_hour` AS
`UDF:timeseries
+// (projectjoindate, hour)`
+// FROM
+//default.datamap2_table
+// GROUP BY datamap2_table.`UDF:timeseries_projectjoindate_hour`)
gen_subsumer_0
+// GROUP BY timeseries(gen_subsumer_0.`UDF:timeseries(projectjoindate,
hour)`,'day' )
+rewrite.modularPlan match {
+ case select: Select =>
+val outputList = select.outputList
+val rolledUpOutputList =
updatedDataMapTablePlan.asInstanceOf[Select].outputList
+var finalOutputList: Seq[NamedExpression] = Seq.empty
+val mapping = outputList zip rolledUpOutputList
+val newSubsme = rewrite.newSubsumerName()
+
+for ((s, d) <- mapping) {
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries
Indhumathi27 commented on a change in pull request #3495: [CARBONDATA-3532]
Support Query Rollup for MV TimeSeries Queries
URL: https://github.com/apache/carbondata/pull/3495#discussion_r373326664
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Navigator.scala
##
@@ -66,7 +79,100 @@ private[mv] class Navigator(catalog:
SummaryDatasetCatalog, session: MVSession)
}
}
if (rewrittenPlan.fastEquals(plan)) {
- None
+ if (modularPlan.asScala.exists(p => p.sameResult(rewrittenPlan))) {
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
akashrn5 commented on a change in pull request #3541:
[CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in
query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373320221
##
File path:
datamap/mv/core/src/test/scala/org/apache/carbondata/mv/timeseries/TestMVTimeSeriesCreateDataMapCommand.scala
##
@@ -210,6 +210,27 @@ class TestMVTimeSeriesCreateDataMapCommand extends
QueryTest with BeforeAndAfter
}.getMessage.contains("MV Timeseries is only supported on Timestamp/Date
column")
}
+ test("test timeseries with case sensitive granularity") {
+sql("drop datamap if exists datamap1")
+sql("create datamap datamap1 on table maintable using 'mv'" +
+ " as select timeseries(projectjoindate,'Second'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'Second')")
+val df1 = sql("select timeseries(projectjoindate,'SECOND'),
sum(projectcode) from maintable group by timeseries(projectjoindate,'SECOND')")
+val df2 = sql("select timeseries(projectjoinDATE,'SECOnd'),
sum(projectcode) from maintable where projectcode=8 group by
timeseries(projectjoinDATE,'SECOnd')")
+TestUtil.verifyMVDataMap(df1.queryExecution.analyzed, "datamap1")
+TestUtil.verifyMVDataMap(df2.queryExecution.analyzed, "datamap1")
+sql("drop datamap if exists datamap1")
Review comment:
no need to again drop and create datamap, you can use the same with other
query of minute level granularity
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
akashrn5 commented on a change in pull request #3541:
[CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in
query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373319016
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Utils.scala
##
@@ -519,4 +521,73 @@ object Utils extends PredicateHelper {
}
}
+ /**
+ * Check's if timeseries udf function exists. If exists, compare literal
with case insensitive
+ * value
+ */
+ def isExpressionMatchesUDF(exp: Expression, exprList: Seq[Expression]):
Boolean = {
+exp match {
+ case Alias(s: ScalaUDF, _) if
s.function.isInstanceOf[TimeSeriesFunction] =>
+val children = s.children
+val finalExpr = exprList.filter(a1 =>
+ a1.isInstanceOf[Alias] &&
a1.asInstanceOf[Alias].child.isInstanceOf[ScalaUDF] &&
+ a1.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].function.
+isInstanceOf[TimeSeriesFunction])
+finalExpr.exists(f => {
+ val finalExp =
f.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].children
Review comment:
rename it, i think `childExprsOfTimeSeriesUDF` sounds better
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
akashrn5 commented on a change in pull request #3541:
[CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in
query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373319541
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Utils.scala
##
@@ -519,4 +521,73 @@ object Utils extends PredicateHelper {
}
}
+ /**
+ * Check's if timeseries udf function exists. If exists, compare literal
with case insensitive
+ * value
+ */
+ def isExpressionMatchesUDF(exp: Expression, exprList: Seq[Expression]):
Boolean = {
+exp match {
+ case Alias(s: ScalaUDF, _) if
s.function.isInstanceOf[TimeSeriesFunction] =>
+val children = s.children
+val finalExpr = exprList.filter(a1 =>
+ a1.isInstanceOf[Alias] &&
a1.asInstanceOf[Alias].child.isInstanceOf[ScalaUDF] &&
+ a1.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].function.
+isInstanceOf[TimeSeriesFunction])
+finalExpr.exists(f => {
+ val finalExp =
f.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].children
+ finalExp.head.semanticEquals(children.head) &&
+ finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+children.last.asInstanceOf[Literal].toString())
+})
+ case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+val children = s.children
+var expList: Seq[Expression] = Seq.empty
+exprList foreach {
+ case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+expList = expList.+:(s)
+ case Alias(s: ScalaUDF, _) if
s.function.isInstanceOf[TimeSeriesFunction] =>
+expList = expList.+:(s.asInstanceOf[Expression])
+ case _ =>
+}
+expList.exists(f => {
+ val finalExp = f.asInstanceOf[ScalaUDF].children
+ finalExp.head.semanticEquals(children.head) &&
+ finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+children.last.asInstanceOf[Literal].toString())
+})
+ case exp: Expression =>
+val newExp = exp.transform {
+ case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+getTransformedTimeSeriesUDF(s)
+ case other => other
+}
+val newExprList = exprList map { expr =>
Review comment:
rename to `tranformedExprListWithLowerCase`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
akashrn5 commented on a change in pull request #3541:
[CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in
query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373319253
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Utils.scala
##
@@ -519,4 +521,73 @@ object Utils extends PredicateHelper {
}
}
+ /**
+ * Check's if timeseries udf function exists. If exists, compare literal
with case insensitive
+ * value
+ */
+ def isExpressionMatchesUDF(exp: Expression, exprList: Seq[Expression]):
Boolean = {
+exp match {
+ case Alias(s: ScalaUDF, _) if
s.function.isInstanceOf[TimeSeriesFunction] =>
+val children = s.children
+val finalExpr = exprList.filter(a1 =>
+ a1.isInstanceOf[Alias] &&
a1.asInstanceOf[Alias].child.isInstanceOf[ScalaUDF] &&
+ a1.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].function.
+isInstanceOf[TimeSeriesFunction])
+finalExpr.exists(f => {
+ val finalExp =
f.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].children
+ finalExp.head.semanticEquals(children.head) &&
+ finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+children.last.asInstanceOf[Literal].toString())
+})
+ case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+val children = s.children
+var expList: Seq[Expression] = Seq.empty
Review comment:
rename to more meaningful one, here `expList` and `exprList` are both
confusing
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
akashrn5 commented on a change in pull request #3541:
[CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in
query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373319949
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Utils.scala
##
@@ -519,4 +521,73 @@ object Utils extends PredicateHelper {
}
}
+ /**
Review comment:
Here better to add a comment, which explains the structure of the expression
tree, first write the expression and better to draw a tree, of ScalaUDFExpr,
children as TimeSeriesFun and Literal , so it will hep to understand the below
logic in a better way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
akashrn5 commented on a change in pull request #3541:
[CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in
query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373319428
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Utils.scala
##
@@ -519,4 +521,73 @@ object Utils extends PredicateHelper {
}
}
+ /**
+ * Check's if timeseries udf function exists. If exists, compare literal
with case insensitive
+ * value
+ */
+ def isExpressionMatchesUDF(exp: Expression, exprList: Seq[Expression]):
Boolean = {
+exp match {
+ case Alias(s: ScalaUDF, _) if
s.function.isInstanceOf[TimeSeriesFunction] =>
+val children = s.children
+val finalExpr = exprList.filter(a1 =>
+ a1.isInstanceOf[Alias] &&
a1.asInstanceOf[Alias].child.isInstanceOf[ScalaUDF] &&
+ a1.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].function.
+isInstanceOf[TimeSeriesFunction])
+finalExpr.exists(f => {
+ val finalExp =
f.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].children
+ finalExp.head.semanticEquals(children.head) &&
+ finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+children.last.asInstanceOf[Literal].toString())
+})
+ case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+val children = s.children
+var expList: Seq[Expression] = Seq.empty
+exprList foreach {
+ case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+expList = expList.+:(s)
+ case Alias(s: ScalaUDF, _) if
s.function.isInstanceOf[TimeSeriesFunction] =>
+expList = expList.+:(s.asInstanceOf[Expression])
+ case _ =>
+}
+expList.exists(f => {
+ val finalExp = f.asInstanceOf[ScalaUDF].children
+ finalExp.head.semanticEquals(children.head) &&
+ finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+children.last.asInstanceOf[Literal].toString())
+})
+ case exp: Expression =>
+val newExp = exp.transform {
Review comment:
same as above
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
akashrn5 commented on a change in pull request #3541:
[CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in
query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373317237
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Utils.scala
##
@@ -519,4 +521,73 @@ object Utils extends PredicateHelper {
}
}
+ /**
+ * Check's if timeseries udf function exists. If exists, compare literal
with case insensitive
+ * value
+ */
+ def isExpressionMatchesUDF(exp: Expression, exprList: Seq[Expression]):
Boolean = {
+exp match {
+ case Alias(s: ScalaUDF, _) if
s.function.isInstanceOf[TimeSeriesFunction] =>
+val children = s.children
+val finalExpr = exprList.filter(a1 =>
+ a1.isInstanceOf[Alias] &&
a1.asInstanceOf[Alias].child.isInstanceOf[ScalaUDF] &&
+ a1.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].function.
+isInstanceOf[TimeSeriesFunction])
+finalExpr.exists(f => {
+ val finalExp =
f.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].children
+ finalExp.head.semanticEquals(children.head) &&
+ finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+children.last.asInstanceOf[Literal].toString())
+})
+ case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+val children = s.children
+var expList: Seq[Expression] = Seq.empty
+exprList foreach {
+ case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+expList = expList.+:(s)
+ case Alias(s: ScalaUDF, _) if
s.function.isInstanceOf[TimeSeriesFunction] =>
+expList = expList.+:(s.asInstanceOf[Expression])
+ case _ =>
+}
+expList.exists(f => {
+ val finalExp = f.asInstanceOf[ScalaUDF].children
+ finalExp.head.semanticEquals(children.head) &&
+ finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+children.last.asInstanceOf[Literal].toString())
+})
+ case exp: Expression =>
+val newExp = exp.transform {
+ case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+getTransformedTimeSeriesUDF(s)
+ case other => other
+}
+val newExprList = exprList map { expr =>
+ expr.transform {
+case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+ getTransformedTimeSeriesUDF(s)
+case other => other
+ }
+}
+newExprList.exists(_.semanticEquals(newExp))
+ case _ => false
+}
+ }
+
+ def getTransformedTimeSeriesUDF(s: ScalaUDF): Expression = {
+s.transform {
+ case l: Literal =>
+Literal(l.toString().toLowerCase, l.dataType)
+}
+ }
+
+ def isExpressionMatchesUDF(exp: Expression, expr: Expression): Boolean = {
Review comment:
add comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
akashrn5 commented on a change in pull request #3541:
[CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in
query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373316719
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/DefaultMatchMaker.scala
##
@@ -76,8 +82,16 @@ abstract class DefaultMatchPattern extends
MatchPattern[ModularPlan] {
qualifier = a.qualifier)
}.getOrElse(a)
case a: Expression =>
-aliasMapExp
+var attribute = aliasMapExp
.get(a)
+if (attribute.isEmpty) {
Review comment:
better to give an example in comment, that in which case the attribute is
empty
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3598: [WIP] Remove MDK and cardinality in write path
CarbonDataQA1 commented on issue #3598: [WIP] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598#issuecomment-580584017 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1814/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage URL: https://github.com/apache/carbondata/pull/3595#discussion_r373309450 ## File path: core/src/main/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImpl.java ## @@ -118,7 +118,7 @@ return org.apache.carbondata.format.Encoding.DIRECT_COMPRESS_VARCHAR; case BIT_PACKED: return org.apache.carbondata.format.Encoding.BIT_PACKED; - case DIRECT_DICTIONARY: + case DIRECT_DICTIONARY: Review comment: fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF
ajantha-bhat commented on issue #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF URL: https://github.com/apache/carbondata/pull/3436#issuecomment-580571302 LGTM Need to have one good document about how to use and how to decide configuration parameter and a link to test case. Can be done in another PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And a
ajantha-bhat commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373307776
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/column/ColumnSchema.java
##
@@ -578,6 +581,7 @@ public void readFields(DataInput in) throws IOException {
}
}
this.isLocalDictColumn = in.readBoolean();
+this.indexColumn = in.readBoolean();
Review comment:
I checked, it is for seriallization. Should not affect compatibility
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And a
ajantha-bhat commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373298768
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonRelation.scala
##
@@ -48,7 +48,8 @@ case class CarbonRelation(
}
val dimensionsAttr: Seq[AttributeReference] = {
-val sett = new
LinkedHashSet(carbonTable.getVisibleDimensions.asScala.asJava)
+val sett = new
LinkedHashSet(carbonTable.getVisibleDimensions.asScala.filterNot(_.isIndexColumn)
Review comment:
please use better name
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And a
ajantha-bhat commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373296382
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/column/ColumnSchema.java
##
@@ -578,6 +581,7 @@ public void readFields(DataInput in) throws IOException {
}
}
this.isLocalDictColumn = in.readBoolean();
+this.indexColumn = in.readBoolean();
Review comment:
Have you tested compatibility ? If the old version doesn't have this field.
in.readBoolean() may fail
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3598: [WIP] Remove MDK and cardinality in write path
CarbonDataQA1 commented on issue #3598: [WIP] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598#issuecomment-580452081 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1813/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF
CarbonDataQA1 commented on issue #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF URL: https://github.com/apache/carbondata/pull/3436#issuecomment-580422600 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1811/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3598: [WIP] Remove MDK and cardinality in write path
CarbonDataQA1 commented on issue #3598: [WIP] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598#issuecomment-580412489 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1812/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF
CarbonDataQA1 commented on issue #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF URL: https://github.com/apache/carbondata/pull/3436#issuecomment-580367981 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1810/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF URL: https://github.com/apache/carbondata/pull/3436#discussion_r373046713 ## File path: processing/src/main/java/org/apache/carbondata/processing/loading/steps/InputProcessorStepWithNoConverterImpl.java ## @@ -63,6 +64,8 @@ private Map dataFieldsWithComplexDataType; + private RowConverterImpl rowConverter; Review comment: Have skipped the converter step for schema columns. And did the index column value generation and field conversion through FieldConverter for index column(IndexFieldConverterImpl). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373044373
##
File path:
processing/src/main/java/org/apache/carbondata/processing/loading/converter/impl/RowConverterImpl.java
##
@@ -162,28 +167,96 @@ public DictionaryClient call() throws Exception {
return null;
}
+ private int getDataFieldIndexByName(String column) {
+for (int i = 0; i < fields.length; i++) {
+ if (fields[i].getColumn().getColName().equalsIgnoreCase(column)) {
+return i;
+ }
+}
+return -1;
+ }
+
+ private String generateNonSchemaColumnValue(DataField field, CarbonRow row) {
Review comment:
Agreed. Have defined a new FieldConverter for index
column(IndexFieldConverterImpl). And this code is removed from this file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373042570
##
File path:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
##
@@ -568,6 +569,11 @@ private int getBlockCount(List
blocklets) {
filter = filter == null ? new DataMapFilter(carbonTable, null) : filter;
ExplainCollector.setFilterStatement(
filter.getExpression() == null ? "none" :
filter.getExpression().getStatement());
+if (filter.isEmpty() && filter.getExpression() != null &&
Review comment:
Have refactored the code. This check is not required anymore. Have removed
it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373042454
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/filter/FilterExpressionProcessor.java
##
@@ -323,6 +323,11 @@ private FilterResolverIntf
createFilterResolverTree(Expression expressionTree,
case TRUE:
return getFilterResolverBasedOnExpressionType(ExpressionType.TRUE,
false,
expressionTree, tableIdentifier, expressionTree);
+ case POLYGON:
+if (expressionTree.getChildren().isEmpty()) {
Review comment:
Have refactored the code. This check is not required anymore. Have removed
it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373041437
##
File path:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
##
@@ -264,8 +305,116 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
s"Carbon Implicit column ${col.column} is not allowed in" +
s" column name while creating table")
}
+}
+ }
+ /**
+ * The method parses, validates and processes the index_handler property.
+ * @param tableProperties Table properties
+ * @param tableFields Sequence of table fields
+ * @return Sequence of index fields to add to table fields
+ */
+ private def processIndexProperty(tableProperties: mutable.Map[String,
String],
Review comment:
Moved to CarbonParserUtil similar to other parse util methods.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373038067
##
File path:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
##
@@ -264,8 +308,142 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
s"Carbon Implicit column ${col.column} is not allowed in" +
s" column name while creating table")
}
+}
+ }
+ /**
+ * The method parses, validates and processes the index_handler property.
+ * @param tableProperties Table properties
+ * @param tableFields Sequence of table fields
+ * @return Sequence of index fields to add to table fields
+ */
+ private def processIndexProperty(tableProperties: mutable.Map[String,
String],
+ tableFields: Seq[Field]): Seq[Field] = {
+val option = tableProperties.get(CarbonCommonConstants.INDEX_HANDLER)
+val fields = ListBuffer[Field]()
+if (option.isDefined) {
+ if (option.get.trim.isEmpty) {
+throw new MalformedCarbonCommandException(
+ s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is invalid.
" +
+s"Option value is empty.")
+ }
+ option.get.split(",").map(_.trim).foreach { handler =>
+/* Validate target column name */
+if (tableFields.exists(_.column.equalsIgnoreCase(handler))) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"handler: $handler must not match with any other column name in
the table")
+}
+val TYPE = s"${CarbonCommonConstants.INDEX_HANDLER}.$handler.type"
+val SOURCE_COLUMNS =
s"${CarbonCommonConstants.INDEX_HANDLER}.$handler.sourcecolumns"
+val SOURCE_COLUMN_TYPES
+ =
s"${CarbonCommonConstants.INDEX_HANDLER}.$handler.sourcecolumntypes"
+val HANDLER_CLASS =
s"${CarbonCommonConstants.INDEX_HANDLER}.$handler.class"
+val HANDLER_INSTANCE =
s"${CarbonCommonConstants.INDEX_HANDLER}.$handler.instance"
+
+val handlerType = tableProperties.get(TYPE)
+if (handlerType.isEmpty || handlerType.get.trim.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"$TYPE properties must be specified.")
+}
+val sourceColumnsOption = tableProperties.get(SOURCE_COLUMNS)
+if (sourceColumnsOption.isEmpty ||
sourceColumnsOption.get.trim.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"$SOURCE_COLUMNS property must be specified.")
+}
+val sourcesWithoutSpaces = sourceColumnsOption.get.replaceAll("\\s",
"")
+/* Validate source columns */
+val sources = sourcesWithoutSpaces.split(",")
+if (sources.distinct.length != sources.size) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"$SOURCE_COLUMNS property cannot have duplicate columns.")
+}
+val sourceTypes = StringBuilder.newBuilder
+sources.foreach { column =>
+ tableFields.find(_.column.equalsIgnoreCase(column)) match {
+case Some(field) =>
sourceTypes.append(field.dataType.get).append(",")
+case None =>
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"Source column: $column in property " +
+ s"$SOURCE_COLUMNS must be a column in the table.")
+ }
+}
+tableProperties.put(SOURCE_COLUMNS, sourcesWithoutSpaces)
+tableProperties.put(SOURCE_COLUMN_TYPES,
sourceTypes.dropRight(1).toString())
+val handlerClass = tableProperties.get(HANDLER_CLASS)
+val handlerClassName: String = handlerClass match {
+ case Some(className) => className.trim
+ case None =>
+/* use handler type to find the default implementation */
+ if
(handlerType.get.trim.equalsIgnoreCase(CarbonCommonConstants.GEOHASH)) {
+ /* Use GeoHash default implementation */
+ val className = classOf[GeoHashImpl].getName
+ tableProperties.put(HANDLER_CLASS, className)
+ className
+} else {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"Unsupported value: ${handlerType
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373040378
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/expression/PolygonExpression.java
##
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.expression;
+
+import java.util.List;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import
org.apache.carbondata.core.scan.expression.conditional.EqualToExpression;
+import
org.apache.carbondata.core.scan.expression.conditional.GreaterThanEqualToExpression;
+import
org.apache.carbondata.core.scan.expression.conditional.LessThanEqualToExpression;
+import
org.apache.carbondata.core.scan.expression.exception.FilterIllegalMemberException;
+import
org.apache.carbondata.core.scan.expression.exception.FilterUnsupportedException;
+import org.apache.carbondata.core.scan.expression.logical.OrExpression;
+import org.apache.carbondata.core.scan.expression.logical.RangeExpression;
+import org.apache.carbondata.core.scan.filter.intf.ExpressionType;
+import org.apache.carbondata.core.scan.filter.intf.RowIntf;
+import org.apache.carbondata.core.util.CustomIndex;
+
+/**
+ * InPolygon expression processor. It inputs the InPolygon string to the
GeoHash implementation's
+ * query method, gets the list of ranges of GeoHash IDs to filter as an
output. And then, multiple
+ * range expressions are build from those list of ranges.
+ */
+@InterfaceAudience.Internal
+public class PolygonExpression extends Expression {
Review comment:
It is moved to geo now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373038264
##
File path:
processing/src/main/java/org/apache/carbondata/processing/loading/converter/impl/RowConverterImpl.java
##
@@ -161,11 +166,59 @@ public DictionaryClient call() throws Exception {
return null;
}
+ private int getDataFieldIndexByName(String column) {
+for (int i = 0; i < fields.length; i++) {
+ if (fields[i].getColumn().getColName().equalsIgnoreCase(column)) {
+return i;
+ }
+}
+return -1;
+ }
+
+ private String generateNonSchemaColumnValue(DataField field, CarbonRow row) {
+Map properties =
configuration.getTableSpec().getCarbonTable()
+.getTableInfo().getFactTable().getTableProperties();
+String handler = properties.get(CarbonCommonConstants.INDEX_HANDLER
++ "." + field.getColumn().getColName() + ".instance");
+if (handler != null) {
+ try {
+// TODO Need to check to how to store the instance. This serialization
may be incorrect.
+ByteArrayInputStream bis = new
ByteArrayInputStream(Base64.getDecoder().decode(handler));
+ObjectInputStream in = new ObjectInputStream(bis);
+CustomIndex instance = (CustomIndex) in.readObject();
+String sourceColumns =
properties.get(CarbonCommonConstants.INDEX_HANDLER
++ "." + field.getColumn().getColName() + ".sourcecolumns");
+assert (sourceColumns != null);
+String[] sources = sourceColumns.split(",");
+int srcFieldIndex;
+List sourceValues = new ArrayList();
+for (String source : sources) {
+ srcFieldIndex = getDataFieldIndexByName(source);
+ assert (srcFieldIndex != -1);
+ sourceValues.add(row.getData()[srcFieldIndex]);
+}
+return instance.generate(sourceValues);
+ } catch (Exception e) {
+LOGGER.error("Failed to generate column value while processing
index_handler property."
++ e);
+throw new RuntimeException(e);
+ }
+}
+return null;
+ }
+
@Override
public CarbonRow convert(CarbonRow row) throws CarbonDataLoadingException {
logHolder.setLogged(false);
logHolder.clear();
+Boolean bNonSchemaPresent = false;
for (int i = 0; i < fieldConverters.length; i++) {
+ if (fields[i].getColumn().getSchemaOrdinal() == -1) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373039872
##
File path:
processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java
##
@@ -381,7 +381,9 @@ public static boolean isHeaderValid(String tableName,
String[] csvHeader,
List dimensions =
schema.getCarbonTable().getDimensionByTableName(factTableName);
for (CarbonDimension dimension : dimensions) {
- columnNames.add(dimension.getColName());
+ if (dimension.getSchemaOrdinal() != -1) {
Review comment:
Modified. Have defined a new property to identify a column as index column
instead of relying on schemaOrdinal value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373037031
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/expression/geo/PolygonExpression.java
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.expression.geo;
+
+import java.util.List;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.scan.expression.ColumnExpression;
+import org.apache.carbondata.core.scan.expression.Expression;
+import org.apache.carbondata.core.scan.expression.ExpressionResult;
+import org.apache.carbondata.core.scan.expression.LiteralExpression;
+import
org.apache.carbondata.core.scan.expression.conditional.GreaterThanEqualToExpression;
+import
org.apache.carbondata.core.scan.expression.conditional.LessThanEqualToExpression;
+import
org.apache.carbondata.core.scan.expression.exception.FilterIllegalMemberException;
+import
org.apache.carbondata.core.scan.expression.exception.FilterUnsupportedException;
+import org.apache.carbondata.core.scan.expression.logical.AndExpression;
+import org.apache.carbondata.core.scan.expression.logical.OrExpression;
+import org.apache.carbondata.core.scan.expression.logical.RangeExpression;
+import org.apache.carbondata.core.scan.expression.logical.TrueExpression;
+import org.apache.carbondata.core.scan.filter.intf.ExpressionType;
+import org.apache.carbondata.core.scan.filter.intf.RowIntf;
+import org.apache.carbondata.core.util.CustomIndex;
+
+/**
+ * InPolygon expression processor. It inputs the InPolygon string to the
GeoHash implementation's
+ * query method, gets the list of ranges of GeoHash IDs to filter as an
output. And then, multiple
+ * range expressions are build from those list of ranges.
+ */
+@InterfaceAudience.Internal
+public class PolygonExpression extends Expression {
+ private String polygon;
+ private String columnName;
+ private CustomIndex> handler;
+ private List ranges;
+
+ public PolygonExpression(String polygon, String columnName, CustomIndex
handler) {
+this.polygon = polygon;
+this.handler = handler;
+this.columnName = columnName;
+ }
+
+ /**
+ * This method builds the GeoHash range expressions from the list of ranges
of GeoHash IDs.
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373033839
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
##
@@ -391,7 +391,12 @@ private static void setLocalDictInfo(CarbonTable table,
TableInfo tableInfo) {
*/
private void fillCreateOrderColumn(String tableName) {
List columns = new ArrayList();
-List dimensions = this.tableDimensionsMap.get(tableName);
+List dimensions = new ArrayList();
+for (CarbonDimension dimension : this.tableDimensionsMap.get(tableName)) {
+ if (dimension.getColumnSchema().getSchemaOrdinal() != -1) {
Review comment:
Ok. Added a new property to identify it as an indexColumn.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373033034
##
File path: core/src/main/java/org/apache/carbondata/core/geo/GeoHashImpl.java
##
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.geo;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.util.CustomIndex;
+
+/**
+ * GeoHash default implementation
+ */
+public class GeoHashImpl implements CustomIndex> {
+ /**
+ * Initialize the geohash index handler instance.
+ * @param handlerName
+ * @param properties
+ * @throws Exception
+ */
+ @Override
+ public void init(String handlerName, Map properties) throws
Exception {
+String options = properties.get(CarbonCommonConstants.INDEX_HANDLER);
+if (options == null || options.isEmpty()) {
+ throw new MalformedCarbonCommandException(
+ String.format("%s property is invalid.",
CarbonCommonConstants.INDEX_HANDLER));
+}
+options = options.toLowerCase();
+if (!options.contains(handlerName.toLowerCase())) {
+ throw new MalformedCarbonCommandException(
+ String.format("%s property is invalid. %s is not present.",
+ CarbonCommonConstants.INDEX_HANDLER, handlerName));
+}
+
+String commonKey = CarbonCommonConstants.INDEX_HANDLER + "." + handlerName
+ ".";
+String TYPE = commonKey + "type";
+String SOURCE_COLUMNS = commonKey + "sourcecolumns";
+String SOURCE_COLUMN_TYPES = commonKey + "sourcecolumntypes";
+String TARGET_DATA_TYPE = commonKey + "datatype";
+String ORIGIN_LATITUDE = commonKey + "originlatitude";
+String MIN_LONGITUDE = commonKey + "minlongitude";
+String MAX_LONGITUDE = commonKey + "maxlongitude";
+String MIN_LATITUDE = commonKey + "minlatitude";
+String MAX_LATITUDE = commonKey + "maxlatitude";
+String GRID_SIZE = commonKey + "gridsize";
+String CONVERSION_RATIO = commonKey + "conversionratio";
+
+
+String sourceColumnsOption = properties.get(SOURCE_COLUMNS);
+if (sourceColumnsOption == null) {
+ throw new MalformedCarbonCommandException(
+ String.format("%s property is invalid. %s property is not
specified.",
+ CarbonCommonConstants.INDEX_HANDLER, SOURCE_COLUMNS));
+}
+
+if (sourceColumnsOption.split(",").length != 2) {
+ throw new MalformedCarbonCommandException(
+ String.format("%s property is invalid. %s property must have 2
columns.",
+ CarbonCommonConstants.INDEX_HANDLER, SOURCE_COLUMNS));
+}
+
+String type = properties.get(TYPE);
+if (type != null && !CarbonCommonConstants.GEOHASH.equalsIgnoreCase(type))
{
+ throw new MalformedCarbonCommandException(
+ String.format("%s property is invalid. %s property must be %s
for this class.",
+ CarbonCommonConstants.INDEX_HANDLER, TYPE,
CarbonCommonConstants.GEOHASH));
+}
+
+properties.put(TYPE, CarbonCommonConstants.GEOHASH);
+
+String sourceDataTypes = properties.get(SOURCE_COLUMN_TYPES);
+String[] srcTypes = sourceDataTypes.split(",");
+for (String srcdataType : srcTypes) {
+ if (!"bigint".equalsIgnoreCase(srcdataType)) {
+throw new MalformedCarbonCommandException(
+String.format("%s property is invalid. %s datatypes must be
long.",
+CarbonCommonConstants.INDEX_HANDLER, SOURCE_COLUMNS));
+ }
+}
+
+String dataType = properties.get(TARGET_DATA_TYPE);
+if (dataType != null && !"long".equalsIgnoreCase(dataType)) {
+ throw new MalformedCarbonCommandException(
+ String.format("%s property is invalid. %s property must be long
for this class.",
+ CarbonCommonConstants.INDEX_HANDLER, TARGE
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373032254
##
File path:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
##
@@ -264,8 +307,152 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
s"Carbon Implicit column ${col.column} is not allowed in" +
s" column name while creating table")
}
+}
+ }
+
+ /**
+* The method parses, validates and processes the index_handler property.
+* @param tableProperties Table properties
+* @param tableFields Sequence of table fields
+* @return Sequence of table fields
+*/
+ private def processIndexProperty(tableProperties: mutable.Map[String,
String],
Review comment:
It is moved to CarbonParserUtils.scala
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373030952
##
File path:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
##
@@ -268,6 +309,151 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
}
}
+ /**
+ * The method parses, validates and processes the index_handler property.
+ *
+ * @param tableProperties Table properties
+ * @param tableFields Sequence of table fields
+ * @return Sequence of table fields
+ *
+ */
+ private def processIndexProperty(tableProperties: mutable.Map[String,
String],
+ tableFields: Seq[Field]): Seq[Field] = {
+val option = tableProperties.get(CarbonCommonConstants.INDEX_HANDLER)
+val fields = ListBuffer[Field]()
+if (option.isDefined) {
+ if (option.get.isEmpty) {
+throw new MalformedCarbonCommandException(
+ s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is invalid.
" +
+s"Option value is empty.")
+ }
+
+ val handlers = option.get.split(",")
+ handlers.foreach { e =>
+/* Validate target column name */
+if (tableFields.exists(_.column.equalsIgnoreCase(e))) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"handler value : $e is not allowed. It matches with another
column name in table. " +
+ s"Cannot create column with it.")
+}
+
+val sourceColumnsOption = tableProperties.get(
+ CarbonCommonConstants.INDEX_HANDLER + s".$e.sourcecolumns")
+if (sourceColumnsOption.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property is not specified.")
+} else if (sourceColumnsOption.get.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property cannot be empty.")
+}
+
+/* Validate source columns */
+val sources = sourceColumnsOption.get.split(",")
+if (sources.distinct.length != sources.size) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property " +
+ s"have duplicate columns.")
+}
+
+val sourceTypes = StringBuilder.newBuilder
+sources.foreach { column =>
+ tableFields.find(_.column.equalsIgnoreCase(column)) match {
+case Some(field) =>
sourceTypes.append(field.dataType.get).append(",")
+case None =>
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"Source column : $column in property " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns " +
+ "is not a valid column in table.")
+ }
+}
+
+tableProperties.put(CarbonCommonConstants.INDEX_HANDLER +
+ s".$e.sourcecolumntypes", sourceTypes.dropRight(1).toString())
+
+val handlerType =
tableProperties.get(CarbonCommonConstants.INDEX_HANDLER + s".$e.type")
+val handlerClass =
tableProperties.get(CarbonCommonConstants.INDEX_HANDLER + s".$e.class")
+
+val handlerClassName: String = handlerClass match {
+ case Some(className) =>
+className
+ case None =>
+/* use handler type to find the default implementation */
+if (handlerType.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"Both ${CarbonCommonConstants.INDEX_HANDLER}.$e.class and "
+
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.type properties
are not specified")
+} else if (handlerType.get.equalsIgnoreCase("geohash")) {
+ /* Use geoHash default implementation */
+ val className =
classOf[org.apache.carbondata.core.util.GeoHashDefault].getName
+
tableProperties.put(s"${CarbonCommonConstants.INDEX_HANDLER}.$e.class",
className)
+ className
+} else {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HA
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373030361
##
File path:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
##
@@ -268,6 +309,151 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
}
}
+ /**
+ * The method parses, validates and processes the index_handler property.
+ *
+ * @param tableProperties Table properties
+ * @param tableFields Sequence of table fields
+ * @return Sequence of table fields
+ *
+ */
+ private def processIndexProperty(tableProperties: mutable.Map[String,
String],
+ tableFields: Seq[Field]): Seq[Field] = {
+val option = tableProperties.get(CarbonCommonConstants.INDEX_HANDLER)
+val fields = ListBuffer[Field]()
+if (option.isDefined) {
+ if (option.get.isEmpty) {
+throw new MalformedCarbonCommandException(
+ s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is invalid.
" +
+s"Option value is empty.")
+ }
+
+ val handlers = option.get.split(",")
+ handlers.foreach { e =>
+/* Validate target column name */
+if (tableFields.exists(_.column.equalsIgnoreCase(e))) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"handler value : $e is not allowed. It matches with another
column name in table. " +
+ s"Cannot create column with it.")
+}
+
+val sourceColumnsOption = tableProperties.get(
+ CarbonCommonConstants.INDEX_HANDLER + s".$e.sourcecolumns")
+if (sourceColumnsOption.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property is not specified.")
+} else if (sourceColumnsOption.get.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property cannot be empty.")
+}
+
+/* Validate source columns */
+val sources = sourceColumnsOption.get.split(",")
+if (sources.distinct.length != sources.size) {
+ throw new MalformedCarbonCommandException(
Review comment:
I think, it is not required. Because error indicates that some column names
are duplicated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373028753
##
File path:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
##
@@ -268,6 +309,151 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
}
}
+ /**
+ * The method parses, validates and processes the index_handler property.
+ *
+ * @param tableProperties Table properties
+ * @param tableFields Sequence of table fields
+ * @return Sequence of table fields
+ *
+ */
+ private def processIndexProperty(tableProperties: mutable.Map[String,
String],
+ tableFields: Seq[Field]): Seq[Field] = {
+val option = tableProperties.get(CarbonCommonConstants.INDEX_HANDLER)
+val fields = ListBuffer[Field]()
+if (option.isDefined) {
+ if (option.get.isEmpty) {
+throw new MalformedCarbonCommandException(
+ s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is invalid.
" +
+s"Option value is empty.")
+ }
+
+ val handlers = option.get.split(",")
+ handlers.foreach { e =>
+/* Validate target column name */
+if (tableFields.exists(_.column.equalsIgnoreCase(e))) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"handler value : $e is not allowed. It matches with another
column name in table. " +
+ s"Cannot create column with it.")
+}
+
+val sourceColumnsOption = tableProperties.get(
+ CarbonCommonConstants.INDEX_HANDLER + s".$e.sourcecolumns")
+if (sourceColumnsOption.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property is not specified.")
+} else if (sourceColumnsOption.get.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property cannot be empty.")
+}
+
+/* Validate source columns */
+val sources = sourceColumnsOption.get.split(",")
+if (sources.distinct.length != sources.size) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property " +
+ s"have duplicate columns.")
+}
+
+val sourceTypes = StringBuilder.newBuilder
+sources.foreach { column =>
+ tableFields.find(_.column.equalsIgnoreCase(column)) match {
+case Some(field) =>
sourceTypes.append(field.dataType.get).append(",")
+case None =>
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"Source column : $column in property " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns " +
+ "is not a valid column in table.")
+ }
+}
+
+tableProperties.put(CarbonCommonConstants.INDEX_HANDLER +
+ s".$e.sourcecolumntypes", sourceTypes.dropRight(1).toString())
+
+val handlerType =
tableProperties.get(CarbonCommonConstants.INDEX_HANDLER + s".$e.type")
+val handlerClass =
tableProperties.get(CarbonCommonConstants.INDEX_HANDLER + s".$e.class")
+
+val handlerClassName: String = handlerClass match {
+ case Some(className) =>
+className
+ case None =>
+/* use handler type to find the default implementation */
+if (handlerType.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"Both ${CarbonCommonConstants.INDEX_HANDLER}.$e.class and "
+
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.type properties
are not specified")
+} else if (handlerType.get.equalsIgnoreCase("geohash")) {
+ /* Use geoHash default implementation */
+ val className =
classOf[org.apache.carbondata.core.util.GeoHashDefault].getName
+
tableProperties.put(s"${CarbonCommonConstants.INDEX_HANDLER}.$e.class",
className)
+ className
+} else {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HA
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373028577
##
File path:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
##
@@ -268,6 +309,151 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
}
}
+ /**
+ * The method parses, validates and processes the index_handler property.
+ *
+ * @param tableProperties Table properties
+ * @param tableFields Sequence of table fields
+ * @return Sequence of table fields
+ *
+ */
+ private def processIndexProperty(tableProperties: mutable.Map[String,
String],
+ tableFields: Seq[Field]): Seq[Field] = {
+val option = tableProperties.get(CarbonCommonConstants.INDEX_HANDLER)
+val fields = ListBuffer[Field]()
+if (option.isDefined) {
+ if (option.get.isEmpty) {
+throw new MalformedCarbonCommandException(
+ s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is invalid.
" +
+s"Option value is empty.")
+ }
+
+ val handlers = option.get.split(",")
+ handlers.foreach { e =>
+/* Validate target column name */
+if (tableFields.exists(_.column.equalsIgnoreCase(e))) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"handler value : $e is not allowed. It matches with another
column name in table. " +
+ s"Cannot create column with it.")
+}
+
+val sourceColumnsOption = tableProperties.get(
+ CarbonCommonConstants.INDEX_HANDLER + s".$e.sourcecolumns")
+if (sourceColumnsOption.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property is not specified.")
+} else if (sourceColumnsOption.get.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property cannot be empty.")
+}
+
+/* Validate source columns */
+val sources = sourceColumnsOption.get.split(",")
+if (sources.distinct.length != sources.size) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns
property " +
+ s"have duplicate columns.")
+}
+
+val sourceTypes = StringBuilder.newBuilder
+sources.foreach { column =>
+ tableFields.find(_.column.equalsIgnoreCase(column)) match {
+case Some(field) =>
sourceTypes.append(field.dataType.get).append(",")
+case None =>
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
+ s"Source column : $column in property " +
+ s"${CarbonCommonConstants.INDEX_HANDLER}.$e.sourcecolumns " +
+ "is not a valid column in table.")
+ }
+}
+
+tableProperties.put(CarbonCommonConstants.INDEX_HANDLER +
+ s".$e.sourcecolumntypes", sourceTypes.dropRight(1).toString())
+
+val handlerType =
tableProperties.get(CarbonCommonConstants.INDEX_HANDLER + s".$e.type")
+val handlerClass =
tableProperties.get(CarbonCommonConstants.INDEX_HANDLER + s".$e.class")
+
+val handlerClassName: String = handlerClass match {
+ case Some(className) =>
+className
+ case None =>
+/* use handler type to find the default implementation */
+if (handlerType.isEmpty) {
+ throw new MalformedCarbonCommandException(
+s"Carbon ${CarbonCommonConstants.INDEX_HANDLER} property is
invalid. " +
Review comment:
Modified
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373027979
##
File path:
core/src/main/java/org/apache/carbondata/core/util/GeoHashDefault.java
##
@@ -0,0 +1,255 @@
+package org.apache.carbondata.core.util;
+
+import
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+public class GeoHashDefault implements CustomIndex>
{
+// 角度转弧度的转换因子
+private final static double CONVERT_FACTOR = 180.0;
+// 地球半径
+private final static double EARTH_RADIUS = 6371004.0;
+
+private static double transValue = Math.PI / CONVERT_FACTOR *
EARTH_RADIUS; // 赤道经度1度或者纬度1度对应的地理空间距离
+
+private double oriLongitude = 0; // 坐标原点的经度
+
+private double oriLatitude = 0; // 坐标原点的纬度
+
+private double userDefineMaxLongitude = 0; // 用户定义地图最大的经度
+
+private double userDefineMaxLatitude = 0; // 用户定义地图最大的纬度
+
+private double userDefineMinLongitude = 0; // 用户定义地图最小的经度
+
+private double userDefineMinLatitude = 0; // 用户定义地图最小的纬度
+
+private double CalculateMaxLongitude = 0; // 计算后得出的补齐地图最大的经度
+
+private double CalculateMaxLatitude = 0; // 计算后得出的补齐地图最大的纬度
+
+private int gridSize = 0; //栅格长度单位是米
+
+private double mCos; // 坐标原点纬度的余玄数值
+
+private double deltaY = 0;// 每一个gridSize长度对应Y轴的度数
+
+private double deltaX = 0;// 每一个gridSize长度应X轴的度数
+
+private double deltaYByRatio = 0; // 每一个gridSize长度对应Y轴的度数 * 系数
+
+private double deltaXByRatio = 0; // 每一个gridSize长度应X轴的度数 * 系数
+
+private int cutLevel = 0; // 对整个区域切的刀数（一横一竖为1刀），就是四叉树的深度
+
+private int totalRowNumber = 0;// 整个区域的行数，从左上开始到右下
+
+private int totalCloumnNumber = 0; // 整个区域的列数，从左上开始到右下
+
+private int udfRowStartNumber = 0; // 用户定义区域的开始行数
+
+private int udfRowEndNumber = 0; // 用户定义区域的结束的行数
+
+private int udfCloumnStartNumber = 0; // 用户定义区域的开始列数
+
+private int udfCloumnEndNumber = 0; // 用户定义区域的开始结束列数
+
+private double lon0 = 0; // 栅格最小数值的经度坐标,最小栅格坐标是扩展区域最左上角的经纬度坐标
+
+private double lat0 = 0; // 栅格最小数值的纬度坐标,最小栅格坐标是扩展区域最左上角的经纬度坐标
+
+private double lon0ByRation = 0; // *系数的常量
+private double lat0ByRation = 0; // *系数的常量
+
+private int conversionRatio = 1; // 系数，用于将double类型的经纬度，转换成int类型后计算
+
+
+@Override
+public void validateOption(Map properties) throws
Exception {
+String option = properties.get(CarbonCommonConstants.INDEX_HANDLER);
+if (option == null || option.isEmpty()) {
+throw new MalformedCarbonCommandException(
+String.format("%s property is invalid.",
CarbonCommonConstants.INDEX_HANDLER));
+}
+
+String commonKey = "." + option + ".";
+String sourceColumnsOption =
properties.get(CarbonCommonConstants.INDEX_HANDLER + commonKey +
"sourcecolumns");
+if (sourceColumnsOption == null) {
+throw new MalformedCarbonCommandException(
+String.format("%s property is invalid. %s property is not
specified.",
+CarbonCommonConstants.INDEX_HANDLER,
+CarbonCommonConstants.INDEX_HANDLER + commonKey +
"sourcecolumns"));
+}
+
+if (sourceColumnsOption.split(",").length != 2) {
+throw new MalformedCarbonCommandException(
+String.format("%s property is invalid. %s property must
have 2 columns.",
+CarbonCommonConstants.INDEX_HANDLER,
+CarbonCommonConstants.INDEX_HANDLER + commonKey +
"sourcecolumns"));
+}
+
+String type = properties.get(CarbonCommonConstants.INDEX_HANDLER +
commonKey + "type");
+if (type != null && !"geohash".equalsIgnoreCase(type)) {
+throw new MalformedCarbonCommandException(
+String.format("%s property is invalid. %s property must be
geohash for this class",
+CarbonCommonConstants.INDEX_HANDLER,
+CarbonCommonConstants.INDEX_HANDLER + commonKey +
"type"));
+}
+
+properties.put(CarbonCommonConstants.INDEX_HANDLER + commonKey +
"type", "geohash");
+
+String sourceDataTypes =
properties.get(CarbonCommonConstants.INDEX_HANDLER + commonKey +
"sourcecolumntypes");
+String[] srcTypes = sourceDataTypes.split(",");
+for (String srcdataType : srcTypes) {
+if (!"int".equalsIgnoreCase(srcdataType)) {
+throw new MalformedCarbonCommandException(
+String.format("%s property is invalid. source
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And
VenuReddy2103 commented on a change in pull request #3436:
[CARBONDATA-3548]Geospatial Support: Modified to create and load the table with
a nonschema dimension sort column. And added InPolygon UDF
URL: https://github.com/apache/carbondata/pull/3436#discussion_r373027929
##
File path:
core/src/main/java/org/apache/carbondata/core/util/GeoHashDefault.java
##
@@ -0,0 +1,255 @@
+package org.apache.carbondata.core.util;
+
+import
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+public class GeoHashDefault implements CustomIndex>
{
+// 角度转弧度的转换因子
+private final static double CONVERT_FACTOR = 180.0;
+// 地球半径
+private final static double EARTH_RADIUS = 6371004.0;
+
+private static double transValue = Math.PI / CONVERT_FACTOR *
EARTH_RADIUS; // 赤道经度1度或者纬度1度对应的地理空间距离
+
+private double oriLongitude = 0; // 坐标原点的经度
+
+private double oriLatitude = 0; // 坐标原点的纬度
+
+private double userDefineMaxLongitude = 0; // 用户定义地图最大的经度
+
+private double userDefineMaxLatitude = 0; // 用户定义地图最大的纬度
+
+private double userDefineMinLongitude = 0; // 用户定义地图最小的经度
+
+private double userDefineMinLatitude = 0; // 用户定义地图最小的纬度
+
+private double CalculateMaxLongitude = 0; // 计算后得出的补齐地图最大的经度
+
+private double CalculateMaxLatitude = 0; // 计算后得出的补齐地图最大的纬度
+
+private int gridSize = 0; //栅格长度单位是米
+
+private double mCos; // 坐标原点纬度的余玄数值
+
+private double deltaY = 0;// 每一个gridSize长度对应Y轴的度数
+
+private double deltaX = 0;// 每一个gridSize长度应X轴的度数
+
+private double deltaYByRatio = 0; // 每一个gridSize长度对应Y轴的度数 * 系数
+
+private double deltaXByRatio = 0; // 每一个gridSize长度应X轴的度数 * 系数
+
+private int cutLevel = 0; // 对整个区域切的刀数（一横一竖为1刀），就是四叉树的深度
+
+private int totalRowNumber = 0;// 整个区域的行数，从左上开始到右下
+
+private int totalCloumnNumber = 0; // 整个区域的列数，从左上开始到右下
+
+private int udfRowStartNumber = 0; // 用户定义区域的开始行数
+
+private int udfRowEndNumber = 0; // 用户定义区域的结束的行数
+
+private int udfCloumnStartNumber = 0; // 用户定义区域的开始列数
+
+private int udfCloumnEndNumber = 0; // 用户定义区域的开始结束列数
+
+private double lon0 = 0; // 栅格最小数值的经度坐标,最小栅格坐标是扩展区域最左上角的经纬度坐标
+
+private double lat0 = 0; // 栅格最小数值的纬度坐标,最小栅格坐标是扩展区域最左上角的经纬度坐标
+
+private double lon0ByRation = 0; // *系数的常量
+private double lat0ByRation = 0; // *系数的常量
+
+private int conversionRatio = 1; // 系数，用于将double类型的经纬度，转换成int类型后计算
+
+
+@Override
+public void validateOption(Map properties) throws
Exception {
+String option = properties.get(CarbonCommonConstants.INDEX_HANDLER);
+if (option == null || option.isEmpty()) {
+throw new MalformedCarbonCommandException(
+String.format("%s property is invalid.",
CarbonCommonConstants.INDEX_HANDLER));
+}
+
+String commonKey = "." + option + ".";
+String sourceColumnsOption =
properties.get(CarbonCommonConstants.INDEX_HANDLER + commonKey +
"sourcecolumns");
+if (sourceColumnsOption == null) {
+throw new MalformedCarbonCommandException(
+String.format("%s property is invalid. %s property is not
specified.",
+CarbonCommonConstants.INDEX_HANDLER,
+CarbonCommonConstants.INDEX_HANDLER + commonKey +
"sourcecolumns"));
+}
+
+if (sourceColumnsOption.split(",").length != 2) {
+throw new MalformedCarbonCommandException(
+String.format("%s property is invalid. %s property must
have 2 columns.",
+CarbonCommonConstants.INDEX_HANDLER,
+CarbonCommonConstants.INDEX_HANDLER + commonKey +
"sourcecolumns"));
+}
+
+String type = properties.get(CarbonCommonConstants.INDEX_HANDLER +
commonKey + "type");
+if (type != null && !"geohash".equalsIgnoreCase(type)) {
+throw new MalformedCarbonCommandException(
+String.format("%s property is invalid. %s property must be
geohash for this class",
+CarbonCommonConstants.INDEX_HANDLER,
+CarbonCommonConstants.INDEX_HANDLER + commonKey +
"type"));
+}
+
+properties.put(CarbonCommonConstants.INDEX_HANDLER + commonKey +
"type", "geohash");
+
+String sourceDataTypes =
properties.get(CarbonCommonConstants.INDEX_HANDLER + commonKey +
"sourcecolumntypes");
+String[] srcTypes = sourceDataTypes.split(",");
+for (String srcdataType : srcTypes) {
+if (!"int".equalsIgnoreCase(srcdataType)) {
Review comment:
done
This is an automated message from the
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3598: [WIP] Remove MDK and cardinality in write path
CarbonDataQA1 commented on issue #3598: [WIP] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598#issuecomment-580303559 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1808/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] jackylk opened a new pull request #3598: [WIP] Remove MDK and cardinality in write path
jackylk opened a new pull request #3598: [WIP] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF
CarbonDataQA1 commented on issue #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF URL: https://github.com/apache/carbondata/pull/3436#issuecomment-580281804 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1807/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF
CarbonDataQA1 commented on issue #3436: [CARBONDATA-3548]Geospatial Support: Modified to create and load the table with a nonschema dimension sort column. And added InPolygon UDF URL: https://github.com/apache/carbondata/pull/3436#issuecomment-580245497 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1806/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries
CarbonDataQA1 commented on issue #3495: [CARBONDATA-3532] Support Query Rollup for MV TimeSeries Queries URL: https://github.com/apache/carbondata/pull/3495#issuecomment-580238036 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1805/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3597: [CARBONDATA-3575] Remove redundant exception throws
jackylk commented on issue #3597: [CARBONDATA-3575] Remove redundant exception throws URL: https://github.com/apache/carbondata/pull/3597#issuecomment-580209672 @ajantha-bhat I do not think checkstyle can check this, since it is needs to understand the API semantic This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
CarbonDataQA1 commented on issue #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage URL: https://github.com/apache/carbondata/pull/3595#issuecomment-580196993 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1804/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3596: [CARBONDATA-3673] Remove unused declarations
CarbonDataQA1 commented on issue #3596: [CARBONDATA-3673] Remove unused declarations URL: https://github.com/apache/carbondata/pull/3596#issuecomment-580187468 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1803/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3597: [CARBONDATA-3575] Remove redundant exception throws
jackylk commented on issue #3597: [CARBONDATA-3575] Remove redundant exception throws URL: https://github.com/apache/carbondata/pull/3597#issuecomment-580164635 I checked it using IntelliJ's Analyzer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove
Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
URL: https://github.com/apache/carbondata/pull/3595#discussion_r372838310
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/TableSpec.java
##
@@ -103,7 +102,7 @@ private void addDimensions(List
dimensions) {
} else {
noSortNoDictDimSpec.add(spec);
}
- } else if (dimension.isDirectDictionaryEncoding()) {
+ } else if (dimension.getDataType() == DataTypes.DATE) {
spec = new DimensionSpec(ColumnType.DIRECT_DICTIONARY, dimension,
dictActualPosition++);
Review comment:
This is not possible in this PR since DATE is still a direct dictionary,
need to do in another PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove
Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
URL: https://github.com/apache/carbondata/pull/3595#discussion_r372836145
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
##
@@ -363,14 +363,14 @@ private void
fillDimensionsAndMeasuresForTables(TableSchema tableSchema) {
if (!columnSchema.isInvisible() && columnSchema.isSortColumn()) {
this.numberOfSortColumns++;
}
- if (!columnSchema.getEncodingList().contains(Encoding.DICTIONARY)) {
+ if (columnSchema.getDataType() != DataTypes.DATE) {
Review comment:
Following need to be removed later in another PR
encodings.add(Encoding.DIRECT_DICTIONARY);
encodings.add(Encoding.DICTIONARY);
This is not possible in this PR since DATE is still a direct dictionary,
need to do in another PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove
Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
URL: https://github.com/apache/carbondata/pull/3595#discussion_r372836461
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/TableSpec.java
##
@@ -103,7 +102,7 @@ private void addDimensions(List
dimensions) {
} else {
noSortNoDictDimSpec.add(spec);
}
- } else if (dimension.isDirectDictionaryEncoding()) {
+ } else if (dimension.getDataType() == DataTypes.DATE) {
spec = new DimensionSpec(ColumnType.DIRECT_DICTIONARY, dimension,
dictActualPosition++);
Review comment:
I will refactor ColumnType in another PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove
Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
URL: https://github.com/apache/carbondata/pull/3595#discussion_r372836461
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/TableSpec.java
##
@@ -103,7 +102,7 @@ private void addDimensions(List
dimensions) {
} else {
noSortNoDictDimSpec.add(spec);
}
- } else if (dimension.isDirectDictionaryEncoding()) {
+ } else if (dimension.getDataType() == DataTypes.DATE) {
spec = new DimensionSpec(ColumnType.DIRECT_DICTIONARY, dimension,
dictActualPosition++);
Review comment:
I will refactor ColumnType in another PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
jackylk commented on a change in pull request #3595: [CARBONDATA-3674] remove
Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
URL: https://github.com/apache/carbondata/pull/3595#discussion_r372836145
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
##
@@ -363,14 +363,14 @@ private void
fillDimensionsAndMeasuresForTables(TableSchema tableSchema) {
if (!columnSchema.isInvisible() && columnSchema.isSortColumn()) {
this.numberOfSortColumns++;
}
- if (!columnSchema.getEncodingList().contains(Encoding.DICTIONARY)) {
+ if (columnSchema.getDataType() != DataTypes.DATE) {
Review comment:
Following need to be removed later in another PR
encodings.add(Encoding.DIRECT_DICTIONARY);
encodings.add(Encoding.DICTIONARY);
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
ajantha-bhat commented on issue #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage URL: https://github.com/apache/carbondata/pull/3595#issuecomment-580155563 Also I hope Encoding.Dictionary is not used for local dictionary, I will cross check. May be need to test once compatibility with old store also. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3597: [CARBONDATA-3575] Remove redundant exception throws
ajantha-bhat commented on issue #3597: [CARBONDATA-3575] Remove redundant exception throws URL: https://github.com/apache/carbondata/pull/3597#issuecomment-580153938 @jackylk : how to make sure this PR has removed all the redundant exception throwing ? and make sure new PR don't add any new redundant exception ? @lamber-ken : do we have checkstyle rule for this ? we should enable it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674]
remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
URL: https://github.com/apache/carbondata/pull/3595#discussion_r372827270
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/TableSpec.java
##
@@ -103,7 +102,7 @@ private void addDimensions(List
dimensions) {
} else {
noSortNoDictDimSpec.add(spec);
}
- } else if (dimension.isDirectDictionaryEncoding()) {
+ } else if (dimension.getDataType() == DataTypes.DATE) {
spec = new DimensionSpec(ColumnType.DIRECT_DICTIONARY, dimension,
dictActualPosition++);
Review comment:
ColumnType.DIRECT_DICTIONARY, we are still keeping ? we can refactor this
also
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674]
remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
URL: https://github.com/apache/carbondata/pull/3595#discussion_r372825697
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
##
@@ -363,14 +363,14 @@ private void
fillDimensionsAndMeasuresForTables(TableSchema tableSchema) {
if (!columnSchema.isInvisible() && columnSchema.isSortColumn()) {
this.numberOfSortColumns++;
}
- if (!columnSchema.getEncodingList().contains(Encoding.DICTIONARY)) {
+ if (columnSchema.getDataType() != DataTypes.DATE) {
Review comment:
similar comment for Encoding.DICTIONARY also
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674]
remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
URL: https://github.com/apache/carbondata/pull/3595#discussion_r372825333
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
##
@@ -363,14 +363,14 @@ private void
fillDimensionsAndMeasuresForTables(TableSchema tableSchema) {
if (!columnSchema.isInvisible() && columnSchema.isSortColumn()) {
this.numberOfSortColumns++;
}
- if (!columnSchema.getEncodingList().contains(Encoding.DICTIONARY)) {
+ if (columnSchema.getDataType() != DataTypes.DATE) {
Review comment:
I think all the places we can remove below code also.
encodings.add(Encoding.DIRECT_DICTIONARY);
encodings.add(Encoding.DICTIONARY);
Also can remove the carbonColumn.hasEncoding(Encoding.DIRECT_DICTIONARY)
check from all the places.
so, can lookup the usages of Encoding.DIRECT_DICTIONARY, we can remove
everywhere other than thrift to maintain the compatibility
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674]
remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
URL: https://github.com/apache/carbondata/pull/3595#discussion_r372821631
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/TableSpec.java
##
@@ -103,7 +102,7 @@ private void addDimensions(List
dimensions) {
} else {
noSortNoDictDimSpec.add(spec);
}
- } else if (dimension.isDirectDictionaryEncoding()) {
Review comment:
CarbonDimension has isGlobalDictionaryEncoding and
isDirectDictionaryEncoding, please remove both the methods and handle the
callers
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage URL: https://github.com/apache/carbondata/pull/3595#discussion_r372822576 ## File path: core/src/main/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImpl.java ## @@ -118,7 +118,7 @@ return org.apache.carbondata.format.Encoding.DIRECT_COMPRESS_VARCHAR; case BIT_PACKED: return org.apache.carbondata.format.Encoding.BIT_PACKED; - case DIRECT_DICTIONARY: + case DIRECT_DICTIONARY: Review comment: revert this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674] remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
ajantha-bhat commented on a change in pull request #3595: [CARBONDATA-3674]
remove Encoding.DICTIONARY and Encoding.DIRECT_DICTIONARY usage
URL: https://github.com/apache/carbondata/pull/3595#discussion_r372821631
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/TableSpec.java
##
@@ -103,7 +102,7 @@ private void addDimensions(List
dimensions) {
} else {
noSortNoDictDimSpec.add(spec);
}
- } else if (dimension.isDirectDictionaryEncoding()) {
Review comment:
CarbonDimension has isGlobalDictionaryEncoding and
isDirectDictionaryEncoding, please remove both the methods and handle the
callers
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3597: [CARBONDATA-3575] Remove redundant exception throws
CarbonDataQA1 commented on issue #3597: [CARBONDATA-3575] Remove redundant exception throws URL: https://github.com/apache/carbondata/pull/3597#issuecomment-580139640 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1801/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3596: [CARBONDATA-3673] Remove unused declarations
CarbonDataQA1 commented on issue #3596: [CARBONDATA-3673] Remove unused declarations URL: https://github.com/apache/carbondata/pull/3596#issuecomment-580138078 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1802/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
