[GitHub] carbondata pull request #3019: [CARBONDATA-3194] Integrating Carbon with Pre...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244278116
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataModule.java
---
@@ -17,62 +17,150 @@
package org.apache.carbondata.presto;
-import javax.inject.Inject;
+import java.util.function.Supplier;
import static java.util.Objects.requireNonNull;
-import org.apache.carbondata.presto.impl.CarbonTableConfig;
import org.apache.carbondata.presto.impl.CarbonTableReader;
+import com.facebook.presto.hive.CoercionPolicy;
+import com.facebook.presto.hive.DirectoryLister;
+import com.facebook.presto.hive.FileFormatDataSourceStats;
+import com.facebook.presto.hive.GenericHiveRecordCursorProvider;
+import com.facebook.presto.hive.HadoopDirectoryLister;
+import com.facebook.presto.hive.HdfsConfiguration;
+import com.facebook.presto.hive.HdfsConfigurationUpdater;
+import com.facebook.presto.hive.HdfsEnvironment;
+import com.facebook.presto.hive.HiveClientConfig;
+import com.facebook.presto.hive.HiveClientModule;
+import com.facebook.presto.hive.HiveCoercionPolicy;
+import com.facebook.presto.hive.HiveConnectorId;
+import com.facebook.presto.hive.HiveEventClient;
+import com.facebook.presto.hive.HiveFileWriterFactory;
+import com.facebook.presto.hive.HiveHdfsConfiguration;
+import com.facebook.presto.hive.HiveLocationService;
+import com.facebook.presto.hive.HiveMetadataFactory;
+import com.facebook.presto.hive.HiveNodePartitioningProvider;
+import com.facebook.presto.hive.HivePageSinkProvider;
+import com.facebook.presto.hive.HivePageSourceFactory;
+import com.facebook.presto.hive.HivePartitionManager;
+import com.facebook.presto.hive.HiveRecordCursorProvider;
+import com.facebook.presto.hive.HiveSessionProperties;
+import com.facebook.presto.hive.HiveSplitManager;
+import com.facebook.presto.hive.HiveTableProperties;
+import com.facebook.presto.hive.HiveTransactionManager;
+import com.facebook.presto.hive.HiveTypeTranslator;
+import com.facebook.presto.hive.HiveWriterStats;
+import com.facebook.presto.hive.LocationService;
+import com.facebook.presto.hive.NamenodeStats;
+import com.facebook.presto.hive.OrcFileWriterConfig;
+import com.facebook.presto.hive.OrcFileWriterFactory;
+import com.facebook.presto.hive.PartitionUpdate;
+import com.facebook.presto.hive.RcFileFileWriterFactory;
+import com.facebook.presto.hive.TableParameterCodec;
+import com.facebook.presto.hive.TransactionalMetadata;
+import com.facebook.presto.hive.TypeTranslator;
+import com.facebook.presto.hive.orc.DwrfPageSourceFactory;
+import com.facebook.presto.hive.orc.OrcPageSourceFactory;
+import com.facebook.presto.hive.parquet.ParquetPageSourceFactory;
+import com.facebook.presto.hive.parquet.ParquetRecordCursorProvider;
+import com.facebook.presto.hive.rcfile.RcFilePageSourceFactory;
+import com.facebook.presto.spi.connector.ConnectorNodePartitioningProvider;
+import com.facebook.presto.spi.connector.ConnectorPageSinkProvider;
import com.facebook.presto.spi.connector.ConnectorPageSourceProvider;
import com.facebook.presto.spi.connector.ConnectorSplitManager;
-import com.facebook.presto.spi.type.Type;
-import com.facebook.presto.spi.type.TypeManager;
-import com.fasterxml.jackson.databind.DeserializationContext;
-import com.fasterxml.jackson.databind.deser.std.FromStringDeserializer;
import com.google.inject.Binder;
-import com.google.inject.Module;
import com.google.inject.Scopes;
+import com.google.inject.TypeLiteral;
+import com.google.inject.multibindings.Multibinder;
+import io.airlift.event.client.EventClient;
-import static
com.facebook.presto.spi.type.TypeSignature.parseTypeSignature;
-import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.inject.multibindings.Multibinder.newSetBinder;
import static io.airlift.configuration.ConfigBinder.configBinder;
+import static io.airlift.json.JsonCodecBinder.jsonCodecBinder;
-public class CarbondataModule implements Module {
+import static org.weakref.jmx.ObjectNames.generatedNameOf;
+import static org.weakref.jmx.guice.ExportBinder.newExporter;
+
+public class CarbondataModule extends HiveClientModule {
--- End diff --
ok
---
[GitHub] carbondata issue #3031: [CARBONDATA-3212] Fixed NegativeArraySizeException w...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3031 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10311/ ---
[GitHub] carbondata issue #3031: [CARBONDATA-3212] Fixed NegativeArraySizeException w...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3031 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2263/ ---
[jira] [Created] (CARBONDATA-3214) Optimize the test code
xubo245 created CARBONDATA-3214:
---
Summary: Optimize the test code
Key: CARBONDATA-3214
URL: https://issues.apache.org/jira/browse/CARBONDATA-3214
Project: CarbonData
Issue Type: Improvement
Reporter: xubo245
When I run test case with: mvn -Pspark-2.1 clean install
A file was deleted, the files should be deleted, we should fix it;
It create a new folder after run test case, carbon shouldn't create new folder
after run test case, so we should delete it.
{code:java}
deleted:integration/spark-common-test/src/test/resources/temp/data1.csv
Untracked files:
(use "git add ..." to include in what will be committed)
examples/spark2/testWriteFiles/
{code}
Please check different spark version too.
mvn -Pspark-2.2 clean install
mvn -Pspark-2.3 clean install
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata pull request #3019: [CARBONDATA-3194] Integrating Carbon with Pre...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244276202
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/PrestoFilterUtil.java
---
@@ -78,32 +72,33 @@
private static final String HIVE_DEFAULT_DYNAMIC_PARTITION =
"__HIVE_DEFAULT_PARTITION__";
/**
- * @param carbondataColumnHandle
+ * @param columnHandle
* @return
*/
- private static DataType spi2CarbondataTypeMapper(CarbondataColumnHandle
carbondataColumnHandle) {
-Type colType = carbondataColumnHandle.getColumnType();
-if (colType == BooleanType.BOOLEAN) {
+ private static DataType spi2CarbondataTypeMapper(HiveColumnHandle
columnHandle) {
+HiveType colType = columnHandle.getHiveType();
+if (colType.equals(HiveType.HIVE_BOOLEAN)) {
return DataTypes.BOOLEAN;
-} else if (colType == SmallintType.SMALLINT) {
+} else if (colType.equals(HiveType.HIVE_SHORT)) {
return DataTypes.SHORT;
-} else if (colType == IntegerType.INTEGER) {
+} else if (colType.equals(HiveType.HIVE_INT)) {
return DataTypes.INT;
-} else if (colType == BigintType.BIGINT) {
+} else if (colType.equals(HiveType.HIVE_LONG)) {
return DataTypes.LONG;
-} else if (colType == DoubleType.DOUBLE) {
+} else if (colType.equals(HiveType.HIVE_DOUBLE)) {
return DataTypes.DOUBLE;
-} else if (colType == VarcharType.VARCHAR) {
+} else if (colType.equals(HiveType.HIVE_STRING)) {
return DataTypes.STRING;
-} else if (colType == DateType.DATE) {
+} else if (colType.equals(HiveType.HIVE_DATE)) {
return DataTypes.DATE;
-} else if (colType == TimestampType.TIMESTAMP) {
+} else if (colType.equals(HiveType.HIVE_TIMESTAMP)) {
return DataTypes.TIMESTAMP;
-} else if
(colType.equals(DecimalType.createDecimalType(carbondataColumnHandle.getPrecision(),
-carbondataColumnHandle.getScale( {
- return
DataTypes.createDecimalType(carbondataColumnHandle.getPrecision(),
- carbondataColumnHandle.getScale());
-} else {
+}
+else if (colType.getTypeInfo() instanceof DecimalTypeInfo) {
--- End diff --
ok
---
[GitHub] carbondata pull request #3019: [CARBONDATA-3194] Integrating Carbon with Pre...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244276046
--- Diff:
integration/presto/src/test/scala/org/apache/carbondata/presto/util/CarbonDataStoreCreator.scala
---
@@ -80,7 +80,7 @@ object CarbonDataStoreCreator {
UUID.randomUUID().toString))
// val factFilePath: String = new
File(dataFilePath).getCanonicalPath
val storeDir: File = new File(absoluteTableIdentifier.getTablePath)
- CarbonUtil.deleteFoldersAndFiles(storeDir)
+// CarbonUtil.deleteFoldersAndFiles(storeDir)
--- End diff --
ok
---
[jira] [Created] (CARBONDATA-3213) Add License for all doc
xubo245 created CARBONDATA-3213: --- Summary: Add License for all doc Key: CARBONDATA-3213 URL: https://issues.apache.org/jira/browse/CARBONDATA-3213 Project: CarbonData Issue Type: Improvement Affects Versions: 1.5.1 Reporter: xubo245 There are some doc no license in project, we should add License for all doc, for example: # datamap-developer-guide.md # introduction.md # usecases.md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3213) Add License for all doc
[ https://issues.apache.org/jira/browse/CARBONDATA-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3213: Description: There are some doc no license in project, we should add License for all doc, for example: # datamap-developer-guide.md # introduction.md # usecases.md and so on, please check all doc was: There are some doc no license in project, we should add License for all doc, for example: # datamap-developer-guide.md # introduction.md # usecases.md > Add License for all doc > > > Key: CARBONDATA-3213 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3213 > Project: CarbonData > Issue Type: Improvement >Affects Versions: 1.5.1 >Reporter: xubo245 >Priority: Major > > There are some doc no license in project, we should add License for all doc, > for example: > # datamap-developer-guide.md > # introduction.md > # usecases.md > and so on, please check all doc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #3027: [CARBONDATA-3202]update the schema to session catalo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3027 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2060/ ---
[GitHub] carbondata pull request #3015: [Carbondata-3173] Add the hive /presto docume...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/3015 ---
[GitHub] carbondata issue #3015: [Carbondata-3173] Add the hive /presto documents ind...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/3015 LGTM, Thank you for working on this. @BeyondYourself ---
[GitHub] carbondata pull request #3027: [CARBONDATA-3202]update the schema to session...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3027#discussion_r244271907
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableColRenameDataTypeChangeCommand.scala
---
@@ -262,13 +263,26 @@ private[sql] case class
CarbonAlterTableColRenameDataTypeChangeCommand(
carbonTable: CarbonTable,
tableInfo: TableInfo,
addColumnSchema: ColumnSchema,
- schemaEvolutionEntry: SchemaEvolutionEntry): Unit = {
+ schemaEvolutionEntry: SchemaEvolutionEntry,
+ oldCarbonColumn: CarbonColumn): Unit = {
val schemaConverter = new ThriftWrapperSchemaConverterImpl
-val a =
List(schemaConverter.fromExternalToWrapperColumnSchema(addColumnSchema))
+// get the carbon column in schema order
+val carbonColumns =
carbonTable.getCreateOrderColumn(carbonTable.getTableName).asScala
+ .filter(!_.isInvisible).collect{case carbonColumn =>
carbonColumn.getColumnSchema}
--- End diff --
Move filter operation to collect
---
[GitHub] carbondata pull request #3027: [CARBONDATA-3202]update the schema to session...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3027#discussion_r244271865
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableAddColumnCommand.scala
---
@@ -93,11 +93,17 @@ private[sql] case class
CarbonAlterTableAddColumnCommand(
schemaEvolutionEntry.setAdded(newCols.toList.asJava)
val thriftTable = schemaConverter
.fromWrapperToExternalTableInfo(wrapperTableInfo, dbName,
tableName)
+ // carbon columns based on schema order
+ val carbonColumns =
carbonTable.getCreateOrderColumn(carbonTable.getTableName).asScala
+.collect { case carbonColumn => carbonColumn.getColumnSchema }
+.filter(!_.isInvisible)
--- End diff --
Move filter operation in collect operation by adding if clause in the case
statement
---
[GitHub] carbondata pull request #3027: [CARBONDATA-3202]update the schema to session...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3027#discussion_r244272092
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableColRenameDataTypeChangeCommand.scala
---
@@ -262,13 +263,26 @@ private[sql] case class
CarbonAlterTableColRenameDataTypeChangeCommand(
carbonTable: CarbonTable,
tableInfo: TableInfo,
addColumnSchema: ColumnSchema,
- schemaEvolutionEntry: SchemaEvolutionEntry): Unit = {
+ schemaEvolutionEntry: SchemaEvolutionEntry,
+ oldCarbonColumn: CarbonColumn): Unit = {
val schemaConverter = new ThriftWrapperSchemaConverterImpl
-val a =
List(schemaConverter.fromExternalToWrapperColumnSchema(addColumnSchema))
+// get the carbon column in schema order
+val carbonColumns =
carbonTable.getCreateOrderColumn(carbonTable.getTableName).asScala
+ .filter(!_.isInvisible).collect{case carbonColumn =>
carbonColumn.getColumnSchema}
+// get the schema ordinal of the column for which the datatype changed
or column is renamed
+val schemaOrdinal = carbonColumns.collect {
--- End diff --
Instead of collect try and use foreach
---
[GitHub] carbondata pull request #3027: [CARBONDATA-3202]update the schema to session...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3027#discussion_r244271732
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableAddColumnCommand.scala
---
@@ -93,11 +93,17 @@ private[sql] case class
CarbonAlterTableAddColumnCommand(
schemaEvolutionEntry.setAdded(newCols.toList.asJava)
val thriftTable = schemaConverter
.fromWrapperToExternalTableInfo(wrapperTableInfo, dbName,
tableName)
+ // carbon columns based on schema order
+ val carbonColumns =
carbonTable.getCreateOrderColumn(carbonTable.getTableName).asScala
+.collect { case carbonColumn => carbonColumn.getColumnSchema }
+.filter(!_.isInvisible)
+ // sort the new columns based on schema order
+ val sortedColsBasedActualSchemaOrder = newCols.sortBy(a =>
a.getSchemaOrdinal)
val (tableIdentifier, schemaParts, cols) =
AlterTableUtil.updateSchemaInfo(
carbonTable,
schemaConverter.fromWrapperToExternalSchemaEvolutionEntry(schemaEvolutionEntry),
thriftTable,
- Some(newCols))(sparkSession)
+ Some(carbonColumns ++
sortedColsBasedActualSchemaOrder))(sparkSession)
--- End diff --
`AlterTableUtil.updateSchemaInfo` is not making use of columns passed so
remove the method argument and use columns for changing the hive schema
---
[GitHub] carbondata pull request #3027: [CARBONDATA-3202]update the schema to session...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3027#discussion_r244270219
--- Diff:
integration/spark2/src/main/commonTo2.2And2.3/org/apache/spark/sql/hive/CarbonSessionState.scala
---
@@ -105,47 +106,37 @@ class CarbonHiveSessionCatalog(
.asInstanceOf[HiveExternalCatalog].client
}
- def alterTableRename(oldTableIdentifier: TableIdentifier,
- newTableIdentifier: TableIdentifier,
- newTablePath: String): Unit = {
-getClient().runSqlHive(
- s"ALTER TABLE ${ oldTableIdentifier.database.get }.${
oldTableIdentifier.table } " +
- s"RENAME TO ${ oldTableIdentifier.database.get }.${
newTableIdentifier.table }")
-getClient().runSqlHive(
- s"ALTER TABLE ${ oldTableIdentifier.database.get }.${
newTableIdentifier.table} " +
- s"SET SERDEPROPERTIES" +
- s"('tableName'='${ newTableIdentifier.table }', " +
- s"'dbName'='${ oldTableIdentifier.database.get }', 'tablePath'='${
newTablePath }')")
- }
-
- override def alterTable(tableIdentifier: TableIdentifier,
- schemaParts: String,
- cols:
Option[Seq[org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema]])
- : Unit = {
-getClient()
- .runSqlHive(s"ALTER TABLE ${tableIdentifier.database.get}.${
tableIdentifier.table } " +
- s"SET TBLPROPERTIES(${ schemaParts })")
- }
-
override def alterAddColumns(tableIdentifier: TableIdentifier,
schemaParts: String,
- cols:
Option[Seq[org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema]])
- : Unit = {
+ cols: Option[Seq[ColumnSchema]]): Unit = {
alterTable(tableIdentifier, schemaParts, cols)
+CarbonSessionUtil
+ .alterExternalCatalogForTableWithUpdatedSchema(tableIdentifier,
+cols,
+schemaParts,
+sparkSession)
}
override def alterDropColumns(tableIdentifier: TableIdentifier,
--- End diff --
Unify `alterDropColumns` and `alterAddColumns` into one method...keep
interface methods same but move the common code to 1 method and call it from
the interface methods
---
[GitHub] carbondata pull request #3027: [CARBONDATA-3202]update the schema to session...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3027#discussion_r244270598
--- Diff:
integration/spark2/src/main/commonTo2.2And2.3/org/apache/spark/sql/hive/CarbonSessionUtil.scala
---
@@ -93,4 +98,34 @@ object CarbonSessionUtil {
)
}
+ /**
+ * This method alter the table for datatype change or column rename
operation, and update the
+ * external catalog directly
+ *
+ * @param tableIdentifier tableIdentifier for table
+ * @param colsall the column of table, which are updated
with datatype change of
+ *new column name
+ * @param schemaParts schemaParts
+ * @param sparkSessionsparkSession
+ */
+ def alterExternalCatalogForTableWithUpdatedSchema(tableIdentifier:
TableIdentifier,
+ cols: Option[Seq[ColumnSchema]],
+ schemaParts: String,
+ sparkSession: SparkSession): Unit = {
+val carbonTable =
CarbonEnv.getCarbonTable(tableIdentifier)(sparkSession)
+val colArray: scala.collection.mutable.ArrayBuffer[StructField] =
ArrayBuffer()
+cols.get.foreach(column =>
+ if (!column.isInvisible) {
+colArray += StructField(column.getColumnName,
+ SparkTypeConverter
+.convertCarbonToSparkDataType(column,
+ carbonTable))
+ }
+)
+sparkSession.sessionState.catalog.externalCatalog
+ .alterTableDataSchema(tableIdentifier.database.get,
--- End diff --
add a comment for the usage of API `alterTableDataSchema` to explain its
purpose
---
[jira] [Assigned] (CARBONDATA-3210) Unify the public method in example
[
https://issues.apache.org/jira/browse/CARBONDATA-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiaohui0110 reassigned CARBONDATA-3210:
---
Assignee: xiaohui0110
> Unify the public method in example
> --
>
> Key: CARBONDATA-3210
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3210
> Project: CarbonData
> Issue Type: Improvement
>Affects Versions: 1.5.1
>Reporter: xubo245
>Assignee: xiaohui0110
>Priority: Major
>
> Unify the public method in example, we can move the method to a public util
> class:
> getKeyOnPrefix
> {code:java}
> org.apache.carbondata.examples.S3UsingSDKExample#getKeyOnPrefix
> org.apache.carbondata.examples.S3Example$#getKeyOnPrefix
> org.apache.carbondata.spark.thriftserver.CarbonThriftServer#getKeyOnPrefix
> {code}
> getS3EndPoint
> {code:java}
> org.apache.carbondata.examples.S3UsingSDKExample#getS3EndPoint
> org.apache.carbondata.examples.S3Example#getS3EndPoint
> org.apache.carbondata.spark.thriftserver.CarbonThriftServer#getS3EndPoint
> {code}
> getSparkMaster
> {code:java}
> org.apache.carbondata.examples.S3UsingSDKExample#getSparkMaster
> org.apache.carbondata.examples.S3Example#getSparkMaster
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-3209) Remove unused import
[
https://issues.apache.org/jira/browse/CARBONDATA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Guangxin reassigned CARBONDATA-3209:
-
Assignee: Wang Guangxin
> Remove unused import
>
>
> Key: CARBONDATA-3209
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3209
> Project: CarbonData
> Issue Type: Improvement
>Affects Versions: 1.5.1
>Reporter: xubo245
>Assignee: Wang Guangxin
>Priority: Major
>
> Remove unused import，for example：
> {code:java}
> import java.io.File
> import org.apache.commons.lang3.StringUtils
> import scala.util.Random
> import org.apache.spark.SparkEnv
> {code}
> in org.apache.spark.sql.execution.datasources.SparkCarbonTableFormat。
> There are some unused import in
> org.apache.carbondata.spark.util.GlobalDictionaryUtil，org.apache.carbondata.core.util.DataTypeUtilTest。
> Please find other unused import in other class
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #3031: [CARBONDATA-3212] Fixed NegativeArraySizeException w...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3031 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2059/ ---
[jira] [Assigned] (CARBONDATA-3208) Remove unused parameter from code
[ https://issues.apache.org/jira/browse/CARBONDATA-3208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Runzhong reassigned CARBONDATA-3208: Assignee: Liu Runzhong > Remove unused parameter from code > - > > Key: CARBONDATA-3208 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3208 > Project: CarbonData > Issue Type: Bug >Reporter: xubo245 >Assignee: Liu Runzhong >Priority: Major > > Remove unused parameter from code： > for example， sparkSession parameter in： > org.apache.carbondata.spark.rdd.CarbonColumnDictGenerateRDD -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #3030: [HOTFIX] Optimize the code style in csdk/sdk markdow...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/3030 retest this please ---
[GitHub] carbondata issue #3027: [CARBONDATA-3202]update the schema to session catalo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3027 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2058/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2161 @chandrasaripaka Can you verify and fix this issue in branch-1.5,1.4,1.3? ---
[GitHub] carbondata pull request #3031: [CARBONDATA-3212] Fixed NegativeArraySizeExce...
Github user qiuchenjian commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3031#discussion_r244269580
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/LocalDictColumnPage.java
---
@@ -140,6 +140,7 @@ public boolean isLocalDictGeneratedPage() {
} else {
actualDataColumnPage.putBytes(rowId, bytes);
}
+pageSize = rowId + 1;
--- End diff --
```suggestion
if (pageSize <= rowId) {
pageSize = rowId + 1;
}
```
LocalDictColumnPage construct method has init pageSize in line 80, I think
it's better to compare rowId and pageSize
---
[GitHub] carbondata pull request #3019: [CARBONDATA-3194] Integrating Carbon with Pre...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244269584
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonCreateTableCommand.scala
---
@@ -157,7 +157,7 @@ case class CarbonCreateTableCommand(
| tableName "$tableName",
| dbName "$dbName",
| tablePath "$tablePath",
- | path "$tablePath",
+ | path "${FileFactory.addSchemeIfNotExists(tablePath)}",
--- End diff --
Can you update the comment from line 147 to 150. I feel it need to be
rephrased
---
[GitHub] carbondata pull request #3019: [CARBONDATA-3194] Integrating Carbon with Pre...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244269478
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
---
@@ -369,6 +369,24 @@ public static boolean createNewLockFile(String
filePath, FileType fileType) thro
LOCAL, HDFS, ALLUXIO, VIEWFS, S3
}
+ public static String addSchemeIfNotExists(String filePath) {
--- End diff --
add comment
---
[GitHub] carbondata pull request #3019: [CARBONDATA-3194] Integrating Carbon with Pre...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244269435
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataModule.java
---
@@ -17,62 +17,150 @@
package org.apache.carbondata.presto;
-import javax.inject.Inject;
+import java.util.function.Supplier;
import static java.util.Objects.requireNonNull;
-import org.apache.carbondata.presto.impl.CarbonTableConfig;
import org.apache.carbondata.presto.impl.CarbonTableReader;
+import com.facebook.presto.hive.CoercionPolicy;
+import com.facebook.presto.hive.DirectoryLister;
+import com.facebook.presto.hive.FileFormatDataSourceStats;
+import com.facebook.presto.hive.GenericHiveRecordCursorProvider;
+import com.facebook.presto.hive.HadoopDirectoryLister;
+import com.facebook.presto.hive.HdfsConfiguration;
+import com.facebook.presto.hive.HdfsConfigurationUpdater;
+import com.facebook.presto.hive.HdfsEnvironment;
+import com.facebook.presto.hive.HiveClientConfig;
+import com.facebook.presto.hive.HiveClientModule;
+import com.facebook.presto.hive.HiveCoercionPolicy;
+import com.facebook.presto.hive.HiveConnectorId;
+import com.facebook.presto.hive.HiveEventClient;
+import com.facebook.presto.hive.HiveFileWriterFactory;
+import com.facebook.presto.hive.HiveHdfsConfiguration;
+import com.facebook.presto.hive.HiveLocationService;
+import com.facebook.presto.hive.HiveMetadataFactory;
+import com.facebook.presto.hive.HiveNodePartitioningProvider;
+import com.facebook.presto.hive.HivePageSinkProvider;
+import com.facebook.presto.hive.HivePageSourceFactory;
+import com.facebook.presto.hive.HivePartitionManager;
+import com.facebook.presto.hive.HiveRecordCursorProvider;
+import com.facebook.presto.hive.HiveSessionProperties;
+import com.facebook.presto.hive.HiveSplitManager;
+import com.facebook.presto.hive.HiveTableProperties;
+import com.facebook.presto.hive.HiveTransactionManager;
+import com.facebook.presto.hive.HiveTypeTranslator;
+import com.facebook.presto.hive.HiveWriterStats;
+import com.facebook.presto.hive.LocationService;
+import com.facebook.presto.hive.NamenodeStats;
+import com.facebook.presto.hive.OrcFileWriterConfig;
+import com.facebook.presto.hive.OrcFileWriterFactory;
+import com.facebook.presto.hive.PartitionUpdate;
+import com.facebook.presto.hive.RcFileFileWriterFactory;
+import com.facebook.presto.hive.TableParameterCodec;
+import com.facebook.presto.hive.TransactionalMetadata;
+import com.facebook.presto.hive.TypeTranslator;
+import com.facebook.presto.hive.orc.DwrfPageSourceFactory;
+import com.facebook.presto.hive.orc.OrcPageSourceFactory;
+import com.facebook.presto.hive.parquet.ParquetPageSourceFactory;
+import com.facebook.presto.hive.parquet.ParquetRecordCursorProvider;
+import com.facebook.presto.hive.rcfile.RcFilePageSourceFactory;
+import com.facebook.presto.spi.connector.ConnectorNodePartitioningProvider;
+import com.facebook.presto.spi.connector.ConnectorPageSinkProvider;
import com.facebook.presto.spi.connector.ConnectorPageSourceProvider;
import com.facebook.presto.spi.connector.ConnectorSplitManager;
-import com.facebook.presto.spi.type.Type;
-import com.facebook.presto.spi.type.TypeManager;
-import com.fasterxml.jackson.databind.DeserializationContext;
-import com.fasterxml.jackson.databind.deser.std.FromStringDeserializer;
import com.google.inject.Binder;
-import com.google.inject.Module;
import com.google.inject.Scopes;
+import com.google.inject.TypeLiteral;
+import com.google.inject.multibindings.Multibinder;
+import io.airlift.event.client.EventClient;
-import static
com.facebook.presto.spi.type.TypeSignature.parseTypeSignature;
-import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.inject.multibindings.Multibinder.newSetBinder;
import static io.airlift.configuration.ConfigBinder.configBinder;
+import static io.airlift.json.JsonCodecBinder.jsonCodecBinder;
-public class CarbondataModule implements Module {
+import static org.weakref.jmx.ObjectNames.generatedNameOf;
+import static org.weakref.jmx.guice.ExportBinder.newExporter;
+
+public class CarbondataModule extends HiveClientModule {
--- End diff --
Please add comment for this class
---
[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction
Github user NamanRastogi commented on the issue: https://github.com/apache/carbondata/pull/3029 @qiuchenjian We dont need a parameter to control to control whether to sort the data or not while compacting. The sole purpose of compaction is to improve subsequent query performance, and even though the compaction gets slow, it is doing what is intended. ---
[GitHub] carbondata pull request #3031: [CARBONDATA-3212] Fixed NegativeArraySizeExce...
GitHub user shivamasn opened a pull request: https://github.com/apache/carbondata/pull/3031 [CARBONDATA-3212] Fixed NegativeArraySizeException while querying in specific scenario ### What was the issue? In Local Dictionary, page size was not getting updated. So during fallback, new page was being created with less records giving NegativeArraySizeException while querying data. ### What has been changed? Updated the page size in Local Dictionary. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/shivamasn/carbondata negative_index_exception Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/3031.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3031 commit a8f945d178c00ad11872f33326d7860ee47dfb52 Author: shivamasn Date: 2018-12-28T05:32:51Z NegativeArraySizeException Fixed ---
[GitHub] carbondata issue #3010: [CARBONDATA-3189] Fix PreAggregate Datamap Issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3010 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10309/ ---
[jira] [Created] (CARBONDATA-3212) Select * is failing with java.lang.NegativeArraySizeException in SDK flow
Shivam Goyal created CARBONDATA-3212: Summary: Select * is failing with java.lang.NegativeArraySizeException in SDK flow Key: CARBONDATA-3212 URL: https://issues.apache.org/jira/browse/CARBONDATA-3212 Project: CarbonData Issue Type: Bug Affects Versions: 1.5.1 Reporter: Shivam Goyal Fix For: 1.5.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10308/ ---
[GitHub] carbondata issue #3020: [CARBONDATA-3195]Added validation for Inverted Index...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/3020 LGTM ---
[GitHub] carbondata issue #3014: [CARBONDATA-3201] Added load level SORT_SCOPE
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3014 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10306/ ---
[GitHub] carbondata issue #3010: [CARBONDATA-3189] Fix PreAggregate Datamap Issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3010 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2057/ ---
[GitHub] carbondata issue #3010: [CARBONDATA-3189] Fix PreAggregate Datamap Issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3010 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2262/ ---
[GitHub] carbondata issue #3019: [CARBONDATA-3194] Integrating Carbon with Presto usi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3019 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10305/ ---
[GitHub] carbondata pull request #3015: [Carbondata-3173] Add the hive /presto docume...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/3015#discussion_r244259267 --- Diff: docs/presto-guide.md --- @@ -1,21 +1,148 @@ - -Please follow the below steps to query carbondata in presto +[Presto Multinode Cluster setup For Carbondata](#presto-multinode-cluster-setup-for-carbondata) --- End diff -- Please optimize the upper/lower case ---
[GitHub] carbondata pull request #3015: [Carbondata-3173] Add the hive /presto docume...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/3015#discussion_r244259294 --- Diff: docs/presto-guide.md --- @@ -1,21 +1,148 @@ - -Please follow the below steps to query carbondata in presto +[Presto Multinode Cluster setup For Carbondata](#presto-multinode-cluster-setup-for-carbondata) + +[Presto Single node setup For Carbondata](#presto-single-node-setup-for-carbondata) + +## Presto Multinode Cluster setup For Carbondata +### Installing Presto + + 1. Download the 0.210 version of Presto using: + `wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.210/presto-server-0.210.tar.gz` + + 2. Extract Presto tar file: `tar zxvf presto-server-0.210.tar.gz`. + + 3. Download the Presto CLI for the coordinator and name it presto. + + ``` +wget https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.210/presto-cli-0.210-executable.jar + +mv presto-cli-0.210-executable.jar presto + +chmod +x presto + ``` + + ### Create Configuration Files + + 1. Create `etc` folder in presto-server-0.210 directory. + 2. Create `config.properties`, `jvm.config`, `log.properties`, and `node.properties` files. + 3. Install uuid to generate a node.id. + + ``` + sudo apt-get install uuid + + uuid + ``` + + +# Contents of your node.properties file + + ``` + node.environment=production + node.id= + node.data-dir=/home/ubuntu/data + ``` + +# Contents of your jvm.config file + + ``` + -server + -Xmx16G + -XX:+UseG1GC + -XX:G1HeapRegionSize=32M + -XX:+UseGCOverheadLimit + -XX:+ExplicitGCInvokesConcurrent + -XX:+HeapDumpOnOutOfMemoryError + -XX:OnOutOfMemoryError=kill -9 %p + ``` + +# Contents of your log.properties file + ``` + com.facebook.presto=INFO + ``` + + The default minimum level is `INFO`. There are four levels: `DEBUG`, `INFO`, `WARN` and `ERROR`. + +## Coordinator Configurations + + # Contents of your config.properties + ``` + coordinator=true + node-scheduler.include-coordinator=false + http-server.http.port=8086 + query.max-memory=5GB + query.max-total-memory-per-node=5GB + query.max-memory-per-node=3GB + memory.heap-headroom-per-node=1GB + discovery-server.enabled=true + discovery.uri=:8086 + ``` +The options `node-scheduler.include-coordinator=false` and `coordinator=true` indicate that the node is the coordinator and tells the coordinator not to do any of the computation work itself and to use the workers. + +**Note**: We recommend setting `query.max-memory-per-node` to half of the JVM config max memory, though if your workload is highly concurrent, you may want to use a lower value for `query.max-memory-per-node`. + +Also relation between below two configuration-properties should be like: +If, `query.max-memory-per-node=30GB` +Then, `query.max-memory=<30GB * number of nodes>`. + +### Worker Configurations + +# Contents of your config.properties + + ``` + coordinator=false + http-server.http.port=8086 + query.max-memory=5GB + query.max-memory-per-node=2GB + discovery.uri=:8086 + ``` + +**Note**: `jvm.config` and `node.properties` files are same for all the nodes (worker + coordinator). All the nodes should have different `node.id`. + +### Catalog Configurations + +1. Create a folder named `catalog` in etc directory of presto on all the nodes of the cluster including the coordinator. + +# Configuring Carbondata in Presto +1. Create a file named `carbondata.properties` in the `catalog` folder and set the required properties on all the nodes. + +### Add Plugins + +1. Create a directory named `carbondata` in plugin directory of presto. +2. Copy `carbondata` jars to `plugin/carbondata` directory on all nodes. + +### Start Presto Server on all nodes + +``` +./presto-server-0.210/bin/launcher start +``` +To run it as a background process. + +``` +./presto-server-0.210/bin/launcher run +``` +To run it in foreground. + +### Start Presto CLI +``` +./presto +``` +To connect to carbondata catalog use the following command: + +``` +./presto --server :8086 --catalog carbondata --schema +``` +Execute the following command to ensure the workers are connected. + +``` +select * from system.runtime.nodes; +``` +Now you can use the Presto CLI on the coordinator to query data sources in the catalog using the Presto workers. + +## Presto Single node setup For Carbondata
[GitHub] carbondata pull request #3015: [Carbondata-3173] Add the hive /presto docume...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/3015#discussion_r244259315 --- Diff: docs/presto-guide.md --- @@ -1,21 +1,148 @@ - -Please follow the below steps to query carbondata in presto +[Presto Multinode Cluster setup For Carbondata](#presto-multinode-cluster-setup-for-carbondata) + +[Presto Single node setup For Carbondata](#presto-single-node-setup-for-carbondata) + +## Presto Multinode Cluster setup For Carbondata --- End diff -- Please optimize the upper/lower case ---
[GitHub] carbondata pull request #3015: [Carbondata-3173] Add the hive /presto docume...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/3015#discussion_r244259273 --- Diff: docs/presto-guide.md --- @@ -1,21 +1,148 @@ - -Please follow the below steps to query carbondata in presto +[Presto Multinode Cluster setup For Carbondata](#presto-multinode-cluster-setup-for-carbondata) + +[Presto Single node setup For Carbondata](#presto-single-node-setup-for-carbondata) --- End diff -- Please optimize the upper/lower case ---
[GitHub] carbondata pull request #3015: [Carbondata-3173] Add the hive /presto docume...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/3015#discussion_r244259184 --- Diff: docs/presto-guide.md --- @@ -1,21 +1,148 @@
[GitHub] carbondata issue #2919: [CARBONDATA-3097] Optimize getVersionDetails
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2919 @KanakaKumar Please check it again. ---
[jira] [Created] (CARBONDATA-3211) Optimize the documentation
xubo245 created CARBONDATA-3211: --- Summary: Optimize the documentation Key: CARBONDATA-3211 URL: https://issues.apache.org/jira/browse/CARBONDATA-3211 Project: CarbonData Issue Type: Improvement Affects Versions: 1.5.1 Reporter: xubo245 Optimize the documentation: 1. Optimize the upper/lower case problem,we should unify the expression by using correct format , for example: Creating BloomFilter DataMap Dropping specified datamap Disable Datamap in https://github.com/apache/carbondata/blob/master/docs/datamap/bloomfilter-datamap-guide.md we should change olap to OLAP in https://github.com/apache/carbondata/blob/master/docs/datamap-developer-guide.md 2. Optimize the space error in project, for example: * @param dbName the database name , if not a default database in org.apache.carbondata.presto.server.PrestoServer#startServer => after database name, it shouldn't have space, need change it to "database name," others: org.apache.spark.sql.CarbonDatasourceHadoopRelation#toString failed to get lucene datamap , -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #3020: [CARBONDATA-3195]Added validation for Inverted Index...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3020 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10304/ ---
[GitHub] carbondata issue #3026: [CARBONDATA-3193] Added support to compile carbon CD...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3026 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10302/ ---
[jira] [Created] (CARBONDATA-3210) Unify the public method in example
xubo245 created CARBONDATA-3210:
---
Summary: Unify the public method in example
Key: CARBONDATA-3210
URL: https://issues.apache.org/jira/browse/CARBONDATA-3210
Project: CarbonData
Issue Type: Improvement
Affects Versions: 1.5.1
Reporter: xubo245
Unify the public method in example, we can move the method to a public util
class:
getKeyOnPrefix
{code:java}
org.apache.carbondata.examples.S3UsingSDKExample#getKeyOnPrefix
org.apache.carbondata.examples.S3Example$#getKeyOnPrefix
org.apache.carbondata.spark.thriftserver.CarbonThriftServer#getKeyOnPrefix
{code}
getS3EndPoint
{code:java}
org.apache.carbondata.examples.S3UsingSDKExample#getS3EndPoint
org.apache.carbondata.examples.S3Example#getS3EndPoint
org.apache.carbondata.spark.thriftserver.CarbonThriftServer#getS3EndPoint
{code}
getSparkMaster
{code:java}
org.apache.carbondata.examples.S3UsingSDKExample#getSparkMaster
org.apache.carbondata.examples.S3Example#getSparkMaster
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction
Github user qiuchenjian commented on the issue: https://github.com/apache/carbondata/pull/3029 Need it add a param to control whether to sort when no_sort? Maybe user only want to merge small segments to a big segment, slow data merging may be unaccepted ---
[GitHub] carbondata issue #3010: [CARBONDATA-3189] Fix PreAggregate Datamap Issue
Github user Shubh18s commented on the issue: https://github.com/apache/carbondata/pull/3010 retest this please ---
[GitHub] carbondata issue #3022: [CARBONDATA-3196] [CARBONDATA-3203]Fixed Compaction ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3022 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10303/ ---
[jira] [Assigned] (CARBONDATA-3119) The getOrCreateCarbonSession method 'storePath' parameter only checks for null,Causes the data store path to be an empty string and no exception is reported.Results
[
https://issues.apache.org/jira/browse/CARBONDATA-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
张源 reassigned CARBONDATA-3119:
--
Assignee: 张源
> The getOrCreateCarbonSession method 'storePath' parameter only checks for
> null,Causes the data store path to be an empty string and no exception is
> reported.Results in no results for the final select data
>
>
> Key: CARBONDATA-3119
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3119
> Project: CarbonData
> Issue Type: Improvement
> Components: spark-integration
>Affects Versions: 1.5.0
>Reporter: 张源
>Assignee: 张源
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.5.0
>
> Attachments: carbondata-error.doc
>
> Original Estimate: 12h
> Time Spent: 14h 40m
> Remaining Estimate: 0h
>
> val carbon =
> SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession("");
> if (storePath != null) {
> carbonProperties.addProperty(CarbonCommonConstants.STORE_LOCATION, storePath)
> // In case if it is in carbon.properties for backward compatible
> } else if (carbonProperties.getProperty(CarbonCommonConstants.STORE_LOCATION)
> == null) {
> carbonProperties.addProperty(CarbonCommonConstants.STORE_LOCATION,
> session.sessionState.conf.warehousePath)
> }
> h3. *explain:*
> The getOrCreateCarbonSession method storePath parameter only checks for
> null,Causes the data store path to be an empty string and no exception is
> reported.Results in no results for the final select data.
> Please see the attachment for examples
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata pull request #3019: [CARBONDATA-3194] Integrating Carbon with Pre...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244254924
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/PrestoFilterUtil.java
---
@@ -78,32 +72,33 @@
private static final String HIVE_DEFAULT_DYNAMIC_PARTITION =
"__HIVE_DEFAULT_PARTITION__";
/**
- * @param carbondataColumnHandle
+ * @param columnHandle
* @return
*/
- private static DataType spi2CarbondataTypeMapper(CarbondataColumnHandle
carbondataColumnHandle) {
-Type colType = carbondataColumnHandle.getColumnType();
-if (colType == BooleanType.BOOLEAN) {
+ private static DataType spi2CarbondataTypeMapper(HiveColumnHandle
columnHandle) {
+HiveType colType = columnHandle.getHiveType();
+if (colType.equals(HiveType.HIVE_BOOLEAN)) {
return DataTypes.BOOLEAN;
-} else if (colType == SmallintType.SMALLINT) {
+} else if (colType.equals(HiveType.HIVE_SHORT)) {
return DataTypes.SHORT;
-} else if (colType == IntegerType.INTEGER) {
+} else if (colType.equals(HiveType.HIVE_INT)) {
return DataTypes.INT;
-} else if (colType == BigintType.BIGINT) {
+} else if (colType.equals(HiveType.HIVE_LONG)) {
return DataTypes.LONG;
-} else if (colType == DoubleType.DOUBLE) {
+} else if (colType.equals(HiveType.HIVE_DOUBLE)) {
return DataTypes.DOUBLE;
-} else if (colType == VarcharType.VARCHAR) {
+} else if (colType.equals(HiveType.HIVE_STRING)) {
return DataTypes.STRING;
-} else if (colType == DateType.DATE) {
+} else if (colType.equals(HiveType.HIVE_DATE)) {
return DataTypes.DATE;
-} else if (colType == TimestampType.TIMESTAMP) {
+} else if (colType.equals(HiveType.HIVE_TIMESTAMP)) {
return DataTypes.TIMESTAMP;
-} else if
(colType.equals(DecimalType.createDecimalType(carbondataColumnHandle.getPrecision(),
-carbondataColumnHandle.getScale( {
- return
DataTypes.createDecimalType(carbondataColumnHandle.getPrecision(),
- carbondataColumnHandle.getScale());
-} else {
+}
+else if (colType.getTypeInfo() instanceof DecimalTypeInfo) {
+ DecimalTypeInfo typeInfo = (DecimalTypeInfo) colType.getTypeInfo();
+ return
DataTypes.createDecimalType(typeInfo.getPrecision(),typeInfo.getScale());
+}
+else {
--- End diff --
move up
---
[GitHub] carbondata pull request #3019: [CARBONDATA-3194] Integrating Carbon with Pre...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244254915
--- Diff:
integration/presto/src/main/java/org/apache/carbondata/presto/PrestoFilterUtil.java
---
@@ -78,32 +72,33 @@
private static final String HIVE_DEFAULT_DYNAMIC_PARTITION =
"__HIVE_DEFAULT_PARTITION__";
/**
- * @param carbondataColumnHandle
+ * @param columnHandle
* @return
*/
- private static DataType spi2CarbondataTypeMapper(CarbondataColumnHandle
carbondataColumnHandle) {
-Type colType = carbondataColumnHandle.getColumnType();
-if (colType == BooleanType.BOOLEAN) {
+ private static DataType spi2CarbondataTypeMapper(HiveColumnHandle
columnHandle) {
+HiveType colType = columnHandle.getHiveType();
+if (colType.equals(HiveType.HIVE_BOOLEAN)) {
return DataTypes.BOOLEAN;
-} else if (colType == SmallintType.SMALLINT) {
+} else if (colType.equals(HiveType.HIVE_SHORT)) {
return DataTypes.SHORT;
-} else if (colType == IntegerType.INTEGER) {
+} else if (colType.equals(HiveType.HIVE_INT)) {
return DataTypes.INT;
-} else if (colType == BigintType.BIGINT) {
+} else if (colType.equals(HiveType.HIVE_LONG)) {
return DataTypes.LONG;
-} else if (colType == DoubleType.DOUBLE) {
+} else if (colType.equals(HiveType.HIVE_DOUBLE)) {
return DataTypes.DOUBLE;
-} else if (colType == VarcharType.VARCHAR) {
+} else if (colType.equals(HiveType.HIVE_STRING)) {
return DataTypes.STRING;
-} else if (colType == DateType.DATE) {
+} else if (colType.equals(HiveType.HIVE_DATE)) {
return DataTypes.DATE;
-} else if (colType == TimestampType.TIMESTAMP) {
+} else if (colType.equals(HiveType.HIVE_TIMESTAMP)) {
return DataTypes.TIMESTAMP;
-} else if
(colType.equals(DecimalType.createDecimalType(carbondataColumnHandle.getPrecision(),
-carbondataColumnHandle.getScale( {
- return
DataTypes.createDecimalType(carbondataColumnHandle.getPrecision(),
- carbondataColumnHandle.getScale());
-} else {
+}
+else if (colType.getTypeInfo() instanceof DecimalTypeInfo) {
--- End diff --
move up
---
[GitHub] carbondata pull request #3019: [CARBONDATA-3194] Integrating Carbon with Pre...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244254822
--- Diff:
integration/presto/src/test/scala/org/apache/carbondata/presto/server/PrestoServer.scala
---
@@ -129,6 +130,21 @@ class PrestoServer {
}
}
+ def execute(query: String) = {
+
+Try {
+ val conn: Connection = createJdbcConnection(dbName)
+ logger.info(s"* executing the query * \n $query")
--- End diff --
rename logger to LOGGER
---
[jira] [Updated] (CARBONDATA-3209) Remove unused import
[
https://issues.apache.org/jira/browse/CARBONDATA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xubo245 updated CARBONDATA-3209:
Description:
Remove unused import，for example：
{code:java}
import java.io.File
import org.apache.commons.lang3.StringUtils
import scala.util.Random
import org.apache.spark.SparkEnv
{code}
in org.apache.spark.sql.execution.datasources.SparkCarbonTableFormat。
There are some unused import in
org.apache.carbondata.spark.util.GlobalDictionaryUtil，org.apache.carbondata.core.util.DataTypeUtilTest。
Please find other unused import in other class
was:
Remove unused import，for example：
{code:java}
import java.io.File
import org.apache.commons.lang3.StringUtils
import scala.util.Random
import org.apache.spark.SparkEnv
{code}
in org.apache.spark.sql.execution.datasources.SparkCarbonTableFormat。
There are some unused import in
org.apache.carbondata.spark.util.GlobalDictionaryUtil too。
Please find other unused import in other class
> Remove unused import
>
>
> Key: CARBONDATA-3209
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3209
> Project: CarbonData
> Issue Type: Improvement
>Affects Versions: 1.5.1
>Reporter: xubo245
>Priority: Major
>
> Remove unused import，for example：
> {code:java}
> import java.io.File
> import org.apache.commons.lang3.StringUtils
> import scala.util.Random
> import org.apache.spark.SparkEnv
> {code}
> in org.apache.spark.sql.execution.datasources.SparkCarbonTableFormat。
> There are some unused import in
> org.apache.carbondata.spark.util.GlobalDictionaryUtil，org.apache.carbondata.core.util.DataTypeUtilTest。
> Please find other unused import in other class
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata pull request #3019: [CARBONDATA-3194] Integrating Carbon with Pre...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3019#discussion_r244254647
--- Diff:
integration/presto/src/test/scala/org/apache/carbondata/presto/util/CarbonDataStoreCreator.scala
---
@@ -80,7 +80,7 @@ object CarbonDataStoreCreator {
UUID.randomUUID().toString))
// val factFilePath: String = new
File(dataFilePath).getCanonicalPath
val storeDir: File = new File(absoluteTableIdentifier.getTablePath)
- CarbonUtil.deleteFoldersAndFiles(storeDir)
+// CarbonUtil.deleteFoldersAndFiles(storeDir)
--- End diff --
delete it if not required, same for line 81
---
[jira] [Updated] (CARBONDATA-3209) Remove unused import
[
https://issues.apache.org/jira/browse/CARBONDATA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xubo245 updated CARBONDATA-3209:
Description:
Remove unused import，for example：
{code:java}
import java.io.File
import org.apache.commons.lang3.StringUtils
import scala.util.Random
import org.apache.spark.SparkEnv
{code}
in org.apache.spark.sql.execution.datasources.SparkCarbonTableFormat。
There are some unused import in
org.apache.carbondata.spark.util.GlobalDictionaryUtil too。
Please find other unused import in other class
was:
Remove unused import，for example：
{code:java}
import java.io.File
import org.apache.commons.lang3.StringUtils
import scala.util.Random
import org.apache.spark.SparkEnv
{code}
in org.apache.spark.sql.execution.datasources.SparkCarbonTableFormat。
Please find other unused import in other class
> Remove unused import
>
>
> Key: CARBONDATA-3209
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3209
> Project: CarbonData
> Issue Type: Improvement
>Affects Versions: 1.5.1
>Reporter: xubo245
>Priority: Major
>
> Remove unused import，for example：
> {code:java}
> import java.io.File
> import org.apache.commons.lang3.StringUtils
> import scala.util.Random
> import org.apache.spark.SparkEnv
> {code}
> in org.apache.spark.sql.execution.datasources.SparkCarbonTableFormat。
> There are some unused import in
> org.apache.carbondata.spark.util.GlobalDictionaryUtil too。
> Please find other unused import in other class
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (CARBONDATA-3119) The getOrCreateCarbonSession method 'storePath' parameter only checks for null,Causes the data store path to be an empty string and no exception is reported.Results i
[
https://issues.apache.org/jira/browse/CARBONDATA-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
张源 closed CARBONDATA-3119.
--
Resolution: Fixed
> The getOrCreateCarbonSession method 'storePath' parameter only checks for
> null,Causes the data store path to be an empty string and no exception is
> reported.Results in no results for the final select data
>
>
> Key: CARBONDATA-3119
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3119
> Project: CarbonData
> Issue Type: Improvement
> Components: spark-integration
>Affects Versions: 1.5.0
>Reporter: 张源
>Assignee: 张源
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.5.0
>
> Attachments: carbondata-error.doc
>
> Original Estimate: 12h
> Time Spent: 14h 40m
> Remaining Estimate: 0h
>
> val carbon =
> SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession("");
> if (storePath != null) {
> carbonProperties.addProperty(CarbonCommonConstants.STORE_LOCATION, storePath)
> // In case if it is in carbon.properties for backward compatible
> } else if (carbonProperties.getProperty(CarbonCommonConstants.STORE_LOCATION)
> == null) {
> carbonProperties.addProperty(CarbonCommonConstants.STORE_LOCATION,
> session.sessionState.conf.warehousePath)
> }
> h3. *explain:*
> The getOrCreateCarbonSession method storePath parameter only checks for
> null,Causes the data store path to be an empty string and no exception is
> reported.Results in no results for the final select data.
> Please see the attachment for examples
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2161 ---
[jira] [Created] (CARBONDATA-3209) Remove unused import
xubo245 created CARBONDATA-3209:
---
Summary: Remove unused import
Key: CARBONDATA-3209
URL: https://issues.apache.org/jira/browse/CARBONDATA-3209
Project: CarbonData
Issue Type: Improvement
Affects Versions: 1.5.1
Reporter: xubo245
Remove unused import，for example：
{code:java}
import java.io.File
import org.apache.commons.lang3.StringUtils
import scala.util.Random
import org.apache.spark.SparkEnv
{code}
in org.apache.spark.sql.execution.datasources.SparkCarbonTableFormat。
Please find other unused import in other class
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2161 LGTM. Merging into master branch ---
[jira] [Reopened] (CARBONDATA-3128) HiveExample has some exception
[
https://issues.apache.org/jira/browse/CARBONDATA-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xubo245 reopened CARBONDATA-3128:
-
> HiveExample has some exception
> --
>
> Key: CARBONDATA-3128
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3128
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: xubo245
>Assignee: Nicholas Jiang
>Priority: Major
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> HiveExample has some exception:
> org.apache.carbondata.hiveexample.HiveExample
> Version:
> clean -DskipTests -Pspark-2.1 -Pbuild-with-format -Pspark.version=2.1.1
> install
> {code:java}
> 2018-11-26 15:06:01 INFO CarbonProperties:1512 - Considered value for min
> max byte limit for string is: 200
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.spark.sql.hive.CarbonSessionStateBuilder
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.spark.util.Utils$.classForName(Utils.scala:230)
> at
> org.apache.spark.util.CarbonReflectionUtils$.createObject(CarbonReflectionUtils.scala:322)
> at
> org.apache.spark.util.CarbonReflectionUtils$.getSessionState(CarbonReflectionUtils.scala:218)
> at
> org.apache.spark.sql.CarbonSession.sessionState$lzycompute(CarbonSession.scala:56)
> at
> org.apache.spark.sql.CarbonSession.sessionState(CarbonSession.scala:55)
> at
> org.apache.spark.sql.CarbonSession$CarbonBuilder$$anonfun$getOrCreateCarbonSession$2.apply(CarbonSession.scala:258)
> at
> org.apache.spark.sql.CarbonSession$CarbonBuilder$$anonfun$getOrCreateCarbonSession$2.apply(CarbonSession.scala:258)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at
> org.apache.spark.sql.CarbonSession$CarbonBuilder.getOrCreateCarbonSession(CarbonSession.scala:258)
> at
> org.apache.carbondata.hiveexample.HiveExample$.main(HiveExample.scala:51)
> at org.apache.carbondata.hiveexample.HiveExample.main(HiveExample.scala)
> {code}
> Spark2.3:
> clean -DskipTests -Pspark-2.3 -Pbuild-with-format install
> {code:java}
> 2018-11-26 15:07:58 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind
> to another address
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.spark.internal.config.package$.SHUFFLE_SPILL_NUM_ELEMENTS_FORCE_SPILL_THRESHOLD()Lorg/apache/spark/internal/config/ConfigEntry;
> at org.apache.spark.sql.internal.SQLConf$.(SQLConf.scala:1011)
> at org.apache.spark.sql.internal.SQLConf$.(SQLConf.scala)
> at
> org.apache.spark.sql.internal.StaticSQLConf$.(StaticSQLConf.scala:31)
> at
> org.apache.spark.sql.internal.StaticSQLConf$.(StaticSQLConf.scala)
> at
> org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:867)
> at
> org.apache.carbondata.hiveexample.HiveExample$.main(HiveExample.scala:50)
> at org.apache.carbondata.hiveexample.HiveExample.main(HiveExample.scala)
> {code}
> Please test spark 2.2 too
> We should fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-3128) HiveExample has some exception
[
https://issues.apache.org/jira/browse/CARBONDATA-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xubo245 resolved CARBONDATA-3128.
-
Resolution: Resolved
> HiveExample has some exception
> --
>
> Key: CARBONDATA-3128
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3128
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: xubo245
>Assignee: Nicholas Jiang
>Priority: Major
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> HiveExample has some exception:
> org.apache.carbondata.hiveexample.HiveExample
> Version:
> clean -DskipTests -Pspark-2.1 -Pbuild-with-format -Pspark.version=2.1.1
> install
> {code:java}
> 2018-11-26 15:06:01 INFO CarbonProperties:1512 - Considered value for min
> max byte limit for string is: 200
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.spark.sql.hive.CarbonSessionStateBuilder
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.spark.util.Utils$.classForName(Utils.scala:230)
> at
> org.apache.spark.util.CarbonReflectionUtils$.createObject(CarbonReflectionUtils.scala:322)
> at
> org.apache.spark.util.CarbonReflectionUtils$.getSessionState(CarbonReflectionUtils.scala:218)
> at
> org.apache.spark.sql.CarbonSession.sessionState$lzycompute(CarbonSession.scala:56)
> at
> org.apache.spark.sql.CarbonSession.sessionState(CarbonSession.scala:55)
> at
> org.apache.spark.sql.CarbonSession$CarbonBuilder$$anonfun$getOrCreateCarbonSession$2.apply(CarbonSession.scala:258)
> at
> org.apache.spark.sql.CarbonSession$CarbonBuilder$$anonfun$getOrCreateCarbonSession$2.apply(CarbonSession.scala:258)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at
> org.apache.spark.sql.CarbonSession$CarbonBuilder.getOrCreateCarbonSession(CarbonSession.scala:258)
> at
> org.apache.carbondata.hiveexample.HiveExample$.main(HiveExample.scala:51)
> at org.apache.carbondata.hiveexample.HiveExample.main(HiveExample.scala)
> {code}
> Spark2.3:
> clean -DskipTests -Pspark-2.3 -Pbuild-with-format install
> {code:java}
> 2018-11-26 15:07:58 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind
> to another address
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.spark.internal.config.package$.SHUFFLE_SPILL_NUM_ELEMENTS_FORCE_SPILL_THRESHOLD()Lorg/apache/spark/internal/config/ConfigEntry;
> at org.apache.spark.sql.internal.SQLConf$.(SQLConf.scala:1011)
> at org.apache.spark.sql.internal.SQLConf$.(SQLConf.scala)
> at
> org.apache.spark.sql.internal.StaticSQLConf$.(StaticSQLConf.scala:31)
> at
> org.apache.spark.sql.internal.StaticSQLConf$.(StaticSQLConf.scala)
> at
> org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:867)
> at
> org.apache.carbondata.hiveexample.HiveExample$.main(HiveExample.scala:50)
> at org.apache.carbondata.hiveexample.HiveExample.main(HiveExample.scala)
> {code}
> Please test spark 2.2 too
> We should fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (CARBONDATA-3128) HiveExample has some exception
[
https://issues.apache.org/jira/browse/CARBONDATA-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xubo245 closed CARBONDATA-3128.
---
Resolution: Fixed
> HiveExample has some exception
> --
>
> Key: CARBONDATA-3128
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3128
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 1.5.1
>Reporter: xubo245
>Assignee: Nicholas Jiang
>Priority: Major
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> HiveExample has some exception:
> org.apache.carbondata.hiveexample.HiveExample
> Version:
> clean -DskipTests -Pspark-2.1 -Pbuild-with-format -Pspark.version=2.1.1
> install
> {code:java}
> 2018-11-26 15:06:01 INFO CarbonProperties:1512 - Considered value for min
> max byte limit for string is: 200
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.spark.sql.hive.CarbonSessionStateBuilder
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.spark.util.Utils$.classForName(Utils.scala:230)
> at
> org.apache.spark.util.CarbonReflectionUtils$.createObject(CarbonReflectionUtils.scala:322)
> at
> org.apache.spark.util.CarbonReflectionUtils$.getSessionState(CarbonReflectionUtils.scala:218)
> at
> org.apache.spark.sql.CarbonSession.sessionState$lzycompute(CarbonSession.scala:56)
> at
> org.apache.spark.sql.CarbonSession.sessionState(CarbonSession.scala:55)
> at
> org.apache.spark.sql.CarbonSession$CarbonBuilder$$anonfun$getOrCreateCarbonSession$2.apply(CarbonSession.scala:258)
> at
> org.apache.spark.sql.CarbonSession$CarbonBuilder$$anonfun$getOrCreateCarbonSession$2.apply(CarbonSession.scala:258)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at
> org.apache.spark.sql.CarbonSession$CarbonBuilder.getOrCreateCarbonSession(CarbonSession.scala:258)
> at
> org.apache.carbondata.hiveexample.HiveExample$.main(HiveExample.scala:51)
> at org.apache.carbondata.hiveexample.HiveExample.main(HiveExample.scala)
> {code}
> Spark2.3:
> clean -DskipTests -Pspark-2.3 -Pbuild-with-format install
> {code:java}
> 2018-11-26 15:07:58 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind
> to another address
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.spark.internal.config.package$.SHUFFLE_SPILL_NUM_ELEMENTS_FORCE_SPILL_THRESHOLD()Lorg/apache/spark/internal/config/ConfigEntry;
> at org.apache.spark.sql.internal.SQLConf$.(SQLConf.scala:1011)
> at org.apache.spark.sql.internal.SQLConf$.(SQLConf.scala)
> at
> org.apache.spark.sql.internal.StaticSQLConf$.(StaticSQLConf.scala:31)
> at
> org.apache.spark.sql.internal.StaticSQLConf$.(StaticSQLConf.scala)
> at
> org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:867)
> at
> org.apache.carbondata.hiveexample.HiveExample$.main(HiveExample.scala:50)
> at org.apache.carbondata.hiveexample.HiveExample.main(HiveExample.scala)
> {code}
> Please test spark 2.2 too
> We should fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3207) Optimize the alter table example
[ https://issues.apache.org/jira/browse/CARBONDATA-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3207: Issue Type: Sub-task (was: Improvement) Parent: CARBONDATA-3125 > Optimize the alter table example > > > Key: CARBONDATA-3207 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3207 > Project: CarbonData > Issue Type: Sub-task >Reporter: xubo245 >Priority: Major > > Optimize the alter table example， let‘s the example can run. Make it coherent > up and down, and run in tandem: > https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #3029: [CARBONDATA-3200] No-Sort compaction
Github user qiuchenjian commented on a diff in the pull request: https://github.com/apache/carbondata/pull/3029#discussion_r244252283 --- Diff: processing/src/main/java/org/apache/carbondata/processing/merger/CarbonCompactionExecutor.java --- @@ -105,10 +107,10 @@ public CarbonCompactionExecutor(Map segmentMapping, * * @return List of Carbon iterators --- End diff -- ```suggestion * @return The key represents Carbon iterator * The value represents whether it's sorted ``` Change the description of this method ---
[GitHub] carbondata issue #3030: [HOTFIX] Optimize the code style in csdk/sdk markdow...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3030 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10298/ ---
[GitHub] carbondata issue #3027: [CARBONDATA-3202]update the schema to session catalo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3027 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10301/ ---
[GitHub] carbondata issue #3028: [CARBONDATA-3205]Fix Get Local Dictionary for empty ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3028 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10300/ ---
[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3029 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10299/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2056/ ---
[GitHub] carbondata issue #3010: [CARBONDATA-3189] Fix PreAggregate Datamap Issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3010 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2055/ ---
[GitHub] carbondata issue #3014: [CARBONDATA-3201] Added load level SORT_SCOPE
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3014 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2054/ ---
[GitHub] carbondata issue #3020: [CARBONDATA-3195]Added validation for Inverted Index...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3020 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2052/ ---
[GitHub] carbondata issue #3019: [CARBONDATA-3194] Integrating Carbon with Presto usi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3019 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2053/ ---
[GitHub] carbondata issue #3022: [CARBONDATA-3196] [CARBONDATA-3203]Fixed Compaction ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3022 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2051/ ---
[GitHub] carbondata issue #752: [CARBONDATA-879] Clear driver side dict cache for Car...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/752 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10292/ ---
[GitHub] carbondata issue #3026: [CARBONDATA-3193] Added support to compile carbon CD...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3026 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2050/ ---
[GitHub] carbondata issue #3027: [CARBONDATA-3202]update the schema to session catalo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3027 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2049/ ---
[GitHub] carbondata issue #3028: [CARBONDATA-3205]Fix Get Local Dictionary for empty ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3028 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2048/ ---
[GitHub] carbondata issue #1380: [CARBONDATA-1485] Timestamp no dictionary bug
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1380 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10271/ ---
[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3029 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2047/ ---
[GitHub] carbondata issue #3030: [HOTFIX] Optimize the code style in csdk/sdk markdow...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3030 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2046/ ---
[GitHub] carbondata issue #419: [CARBONDATA-523] Added test cases for carbon data pro...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/419 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10297/ ---
[GitHub] carbondata issue #749: [CARBONDATA-854] Datastax CFS file system support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/749 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10293/ ---
[GitHub] carbondata issue #732: [CARBONDATA-754] improve performance when order by pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/732 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10295/ ---
[GitHub] carbondata issue #1241: [CARBONDATA-1390]Request 1 executor when preferred l...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1241 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10279/ ---
[GitHub] carbondata issue #745: [CARBONDATA-876] Clear segment access count ASAP
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/745 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10294/ ---
[GitHub] carbondata issue #710: [CARBONDATA-833]load data from dataframe,generater da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/710 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10296/ ---
[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/985 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10288/ ---
[GitHub] carbondata issue #825: [CARBONDATA-961] Added condition to skip sort step fo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/825 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10290/ ---
[GitHub] carbondata issue #807: [CARBONDATA-942] off heap sort chunk size should be v...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/807 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10291/ ---
[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Carbon table schema compatible...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/984 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10289/ ---
[GitHub] carbondata issue #994: [CARBONDATA-933]Added Compare Test For BenchMarking O...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/994 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10287/ ---
[GitHub] carbondata issue #1101: [CARBONDATA-1143] fix for null struct type
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1101 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10283/ ---
[GitHub] carbondata issue #1061: [CARBONDATA-1193] ViewFS Support - improvement
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1061 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10285/ ---
[GitHub] carbondata issue #1138: [CARBONDATA-1263]Single pass load does not take defa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1138 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10281/ ---
[GitHub] carbondata issue #1230: [WIP]corrected sdv test cases
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1230 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10280/ ---
[GitHub] carbondata issue #1010: [CARBONDATA-1110] put if clause out of the for claus...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1010 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10286/ ---
