[jira] [Created] (CARBONDATA-2792) Create external table fails post schema restructure
Mohammad Shahid Khan created CARBONDATA-2792:
Summary: Create external table fails post schema restructure
Key: CARBONDATA-2792
URL: https://issues.apache.org/jira/browse/CARBONDATA-2792
Project: CarbonData
Issue Type: Bug
Affects Versions: 1.4.0, 1.5.0, 1.4.2
Reporter: Mohammad Shahid Khan
Assignee: Mohammad Shahid Khan
Steps to reproduce.
sql("create table rstest1 (c1 string,c2 int) STORED BY
'org.apache.carbondata.format'")
sql("Alter table rstest1 drop columns(c2)")
sql(
"Alter table rstest1 add columns(c4 string)
TBLPROPERTIES('DICTIONARY_EXCLUDE'='c4', " +
"'DEFAULT.VALUE.c4'='def')")
sql(s"""CREATE EXTERNAL TABLE rsext STORED BY 'carbondata' LOCATION
'$storeLocation/rstest1'""")
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-2749) In HDFS Empty tablestatus file is written during datalaod, iud or compaction when disk is full.
[ https://issues.apache.org/jira/browse/CARBONDATA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan reassigned CARBONDATA-2749: Assignee: Mohammad Shahid Khan > In HDFS Empty tablestatus file is written during datalaod, iud or compaction > when disk is full. > --- > > Key: CARBONDATA-2749 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2749 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > *Issue*: When a failure happens due to disk full during load, IUD or > Compaction, then while updating the tablestatus file, the tablestaus.tmp file > during atomic file operation remains empty, and in the finally block the > empty tablestaus.tmp file is getting renamed to the actual file. > This leads to empty tablestaus file. > *Problem*: Once such problem happens the tablestatus file can not be > retrieved and the already loaded data can be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2749) In HDFS Empty tablestatus file is written during datalaod, iud or compaction when disk is full.
Mohammad Shahid Khan created CARBONDATA-2749: Summary: In HDFS Empty tablestatus file is written during datalaod, iud or compaction when disk is full. Key: CARBONDATA-2749 URL: https://issues.apache.org/jira/browse/CARBONDATA-2749 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan *Issue*: When a failure happens due to disk full during load, IUD or Compaction, then while updating the tablestatus file, the tablestaus.tmp file during atomic file operation remains empty, and in the finally block the empty tablestaus.tmp file is getting renamed to the actual file. This leads to empty tablestaus file. *Problem*: Once such problem happens the tablestatus file can not be retrieved and the already loaded data can be used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2729) Schema Compatibility problem between version 1.3.0 and 1.4.0
[
https://issues.apache.org/jira/browse/CARBONDATA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Shahid Khan updated CARBONDATA-2729:
-
Description:
The table schema created with 1.3 is not compatible with 1.4.0.
Steps:
Do the below operation on hive meta store.
1. Create table and load data in 1.3 version.
{code:java}
spark.sql("create database carbonversion_1_1");
spark.sql("use carbonversion_1_1");
spark.sql("""create table testIntType (c1 string,c2 int) STORED BY
'org.apache.carbondata.format'""")
spark.sql("insert into testIntType select 'a',200"){code}
2. Switch to 1.4 version and create data map on the table created in step 1.
{code:java}
spark.sql("use carbonversion_1_3");
spark.sql("create datamap dm8 on table testinttype3 using 'preaggregate' as
select c1, sum(c2) from testinttype3 group by c1")
{code}
stack trace
{code:java}
stack trace
Exception in thread "main" java.lang.NullPointerException
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
at
org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101){code}
was:
The table schema created with 1.3 is not compatible with 1.4.0.
stack trace
{code:java}
stack trace
Exception in thread "main" java.lang.NullPointerException
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
at
org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101){code}
> Schema Compatibility problem between version 1.3.0 and 1.4.0
>
>
> Key: CARBONDATA-2729
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2729
> Project: CarbonData
> Issue Type: Bug
> Components: file-format
>Affects Versions: 1.4.0
>Reporter: Mohammad Shahid Khan
>Assignee: Mohammad Shahid Khan
>Priority: Major
> Fix For: 1.4.1
>
>
> The table schema created with 1.3 is not compatible with 1.4.0.
> Steps:
> Do the below operation on hive meta store.
> 1. Create table and load data in 1.3 version.
> {code:java}
> spark.sql("create database carbonversion_1_1");
> spark.sql("use carbonversion_1_1");
> spark.sql("""create table testIntType (c1 string,c2 int) STORED BY
> 'org.apache.carbondata.format'""")
> spark.sql("insert into testIntType select 'a',200"){code}
> 2. Switch to 1.4 version and create data map on the table created in step 1.
> {code:java}
> spark.sql("use carbonversion_1_3");
> spark.sql("create datamap dm8 on table testinttype3 using 'preaggregate' as
> select c1, sum(c2) from testinttype3 group by c1")
> {code}
> stack trace
> {code:java}
> stack trace
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
> at
> org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2729) Schema Compatibility problem between version 1.3.0 and 1.4.0
[
https://issues.apache.org/jira/browse/CARBONDATA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Shahid Khan updated CARBONDATA-2729:
-
Description:
The table schema created with 1.3 is not compatible with 1.4.0.
stack trace
{code:java}
stack trace
Exception in thread "main" java.lang.NullPointerException
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
at
org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101){code}
was:
The table schema created with 1.3 is not compatible with 1.4.0.
> Schema Compatibility problem between version 1.3.0 and 1.4.0
>
>
> Key: CARBONDATA-2729
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2729
> Project: CarbonData
> Issue Type: Bug
> Components: file-format
>Affects Versions: 1.4.0
>Reporter: Mohammad Shahid Khan
>Assignee: Mohammad Shahid Khan
>Priority: Major
> Fix For: 1.4.1
>
>
> The table schema created with 1.3 is not compatible with 1.4.0.
> stack trace
> {code:java}
> stack trace
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
> at
> org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CARBONDATA-2729) Schema Compatibility problem between version 1.3.0 and 1.4.0
Mohammad Shahid Khan created CARBONDATA-2729: Summary: Schema Compatibility problem between version 1.3.0 and 1.4.0 Key: CARBONDATA-2729 URL: https://issues.apache.org/jira/browse/CARBONDATA-2729 Project: CarbonData Issue Type: Bug Components: file-format Affects Versions: 1.4.0 Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan Fix For: 1.4.1 The table schema created with 1.3 is not compatible with 1.4.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2710) Refactor CarbonSparkSqlParser for better code reuse.
Mohammad Shahid Khan created CARBONDATA-2710: Summary: Refactor CarbonSparkSqlParser for better code reuse. Key: CARBONDATA-2710 URL: https://issues.apache.org/jira/browse/CARBONDATA-2710 Project: CarbonData Issue Type: Improvement Reporter: Mohammad Shahid Khan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2674) Streaming with merge index enabled does not consider the merge index file while pruning.
[ https://issues.apache.org/jira/browse/CARBONDATA-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-2674: - Summary: Streaming with merge index enabled does not consider the merge index file while pruning. (was: Streaming with merge index enabled does not consider the index file while pruning.) > Streaming with merge index enabled does not consider the merge index file > while pruning. > - > > Key: CARBONDATA-2674 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2674 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2674) Streaming with merge index enabled does not consider the index file while pruning.
Mohammad Shahid Khan created CARBONDATA-2674: Summary: Streaming with merge index enabled does not consider the index file while pruning. Key: CARBONDATA-2674 URL: https://issues.apache.org/jira/browse/CARBONDATA-2674 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2623) Add DataMap Pre and Pevent listener
Mohammad Shahid Khan created CARBONDATA-2623: Summary: Add DataMap Pre and Pevent listener Key: CARBONDATA-2623 URL: https://issues.apache.org/jira/browse/CARBONDATA-2623 Project: CarbonData Issue Type: Improvement Reporter: Mohammad Shahid Khan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2621) Lock problem in index datamap
[ https://issues.apache.org/jira/browse/CARBONDATA-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-2621: - Summary: Lock problem in index datamap (was: Lock problem in for index datamap) > Lock problem in index datamap > - > > Key: CARBONDATA-2621 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2621 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Major > Fix For: 1.5.0 > > > The locking for the index Datamap is not correct. > The HDFS lock will not work properly, because the lock is getting created the > the local filesystem instead of HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2621) Lock problem in for index datamap
Mohammad Shahid Khan created CARBONDATA-2621: Summary: Lock problem in for index datamap Key: CARBONDATA-2621 URL: https://issues.apache.org/jira/browse/CARBONDATA-2621 Project: CarbonData Issue Type: Bug Affects Versions: 1.4.0 Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan Fix For: 1.5.0 The locking for the index Datamap is not correct. The HDFS lock will not work properly, because the lock is getting created the the local filesystem instead of HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2436) Block pruning problem post the carbon schema restructure.
[ https://issues.apache.org/jira/browse/CARBONDATA-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-2436: - Summary: Block pruning problem post the carbon schema restructure. (was: Blocklet pruning problem post the carbon schema restructure.) > Block pruning problem post the carbon schema restructure. > - > > Key: CARBONDATA-2436 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2436 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Priority: Major > > Currently datamap is pruning with segmentproperties from the 0th blcok of > BlockletDataMap is not > correct. As post restructure if the table is updated then all the block will > not have > symetric schema within the same segments. > Fix: It must be ensured the block could be pruned with the same schema. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2436) Blocklet pruning problem post the carbon schema restructure.
Mohammad Shahid Khan created CARBONDATA-2436: Summary: Blocklet pruning problem post the carbon schema restructure. Key: CARBONDATA-2436 URL: https://issues.apache.org/jira/browse/CARBONDATA-2436 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Currently datamap is pruning with segmentproperties from the 0th blcok of BlockletDataMap is not correct. As post restructure if the table is updated then all the block will not have symetric schema within the same segments. Fix: It must be ensured the block could be pruned with the same schema. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2410) Error message correction when column value length exceeds 320000 charactor
Mohammad Shahid Khan created CARBONDATA-2410: Summary: Error message correction when column value length exceeds 32 charactor Key: CARBONDATA-2410 URL: https://issues.apache.org/jira/browse/CARBONDATA-2410 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2289) If carbon merge index is enabled then after IUD operation if some blocks of a segment is deleted, then during query and IUD operation the driver is throwing FileNotF
[
https://issues.apache.org/jira/browse/CARBONDATA-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Shahid Khan updated CARBONDATA-2289:
-
Description:
Root cause:
The carbonindexmerge file is containing deleted blocks (.*carbondata*) meta
i.e. driver tries to load BlockletDataMap for deleted blocks and fails while
populating BlockMetaInfo.
Steps to reproduce:
sql("drop table if exists mitable")
sql("create table mitable(id int, issue date) stored by 'carbondata'")
sql("insert into table mitable select '1','2000-02-01'")
val table = CarbonMetadata.getInstance().getCarbonTable("default", "mitable")
new CarbonIndexFileMergeWriter()
.mergeCarbonIndexFilesOfSegment("0", table.getTablePath, false)
sql("update mitable set(id)=(2) where issue = '2000-02-01'").show()
sql("clean files for table mitable")
sql("select * from mitable").show()
was:
Root cause:
The carbonindexmerge file is containing deleted blocks (.*carbondata*) meta
i.e. driver tries to load BlockletDataMap for deleted blocks and fails while
populating BlockMetaInfo.
> If carbon merge index is enabled then after IUD operation if some blocks of a
> segment is deleted, then during query and IUD operation the driver is
> throwing FileNotFoundException while preparing BlockMetaInfo.
> -
>
> Key: CARBONDATA-2289
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2289
> Project: CarbonData
> Issue Type: Bug
>Reporter: Mohammad Shahid Khan
>Priority: Minor
>
> Root cause:
> The carbonindexmerge file is containing deleted blocks (.*carbondata*) meta
> i.e. driver tries to load BlockletDataMap for deleted blocks and fails while
> populating BlockMetaInfo.
> Steps to reproduce:
> sql("drop table if exists mitable")
> sql("create table mitable(id int, issue date) stored by 'carbondata'")
> sql("insert into table mitable select '1','2000-02-01'")
> val table = CarbonMetadata.getInstance().getCarbonTable("default", "mitable")
> new CarbonIndexFileMergeWriter()
> .mergeCarbonIndexFilesOfSegment("0", table.getTablePath, false)
> sql("update mitable set(id)=(2) where issue = '2000-02-01'").show()
> sql("clean files for table mitable")
> sql("select * from mitable").show()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CARBONDATA-2289) If carbon merge index is enabled then after IUD operation if some blocks of a segment is deleted, then during query and IUD operation the driver is throwing FileNotF
Mohammad Shahid Khan created CARBONDATA-2289: Summary: If carbon merge index is enabled then after IUD operation if some blocks of a segment is deleted, then during query and IUD operation the driver is throwing FileNotFoundException while preparing BlockMetaInfo. Key: CARBONDATA-2289 URL: https://issues.apache.org/jira/browse/CARBONDATA-2289 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Root cause: The carbonindexmerge file is containing deleted blocks (.*carbondata*) meta i.e. driver tries to load BlockletDataMap for deleted blocks and fails while populating BlockMetaInfo. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2285) spark integration code refactor
Mohammad Shahid Khan created CARBONDATA-2285: Summary: spark integration code refactor Key: CARBONDATA-2285 URL: https://issues.apache.org/jira/browse/CARBONDATA-2285 Project: CarbonData Issue Type: Improvement Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan Spark integration code refactor to improve extensible and re-usability -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-1507) Dataload global sort fails with RpcTimeOutException: Futures timed out after [120 seconds]
[ https://issues.apache.org/jira/browse/CARBONDATA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan resolved CARBONDATA-1507. -- Resolution: Duplicate > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] > -- > > Key: CARBONDATA-1507 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1507 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Major > > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2056) Hadoop Configuration with access key and secret key should be passed while creating InputStream of distributed carbon file.
Mohammad Shahid Khan created CARBONDATA-2056: Summary: Hadoop Configuration with access key and secret key should be passed while creating InputStream of distributed carbon file. Key: CARBONDATA-2056 URL: https://issues.apache.org/jira/browse/CARBONDATA-2056 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2049) CarbonCleanFilesCommand table path problem
Mohammad Shahid Khan created CARBONDATA-2049: Summary: CarbonCleanFilesCommand table path problem Key: CARBONDATA-2049 URL: https://issues.apache.org/jira/browse/CARBONDATA-2049 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan In CarbonCleanFilesCommand datbaseLocation is being passed instead of the tablePath in case of forceclean. And in case of cleanGarbageData, storeLocation is being passed instead of the tablePath. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2044) IDG update for CARBONDATA-2043 Configurable wait time for requesting executors and minimum registered executors ratio to continue the block distribution
Mohammad Shahid Khan created CARBONDATA-2044: Summary: IDG update for CARBONDATA-2043 Configurable wait time for requesting executors and minimum registered executors ratio to continue the block distribution Key: CARBONDATA-2044 URL: https://issues.apache.org/jira/browse/CARBONDATA-2044 Project: CarbonData Issue Type: Sub-task Components: data-load Reporter: Mohammad Shahid Khan Assignee: Gururaj Shetty Added below parameters: *carbon.dynamicallocation.schedulertimeout :* Description: To configure wait time for executors to bring up. default 5 sec, Min 5 sec and max 15 sec. *carbon.scheduler.minregisteredresourcesratio :* Description: To configure minimum registered/started executors ratio to start block distribution. default to 0.8min 0.1, max 1.0 Removed Parameters: *carbon.max.executor.startup.timeout* – Remove it from IDG -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2043) Configurable wait time for requesting executors and minimum registered executors ratio to continue the block distribution
Mohammad Shahid Khan created CARBONDATA-2043: Summary: Configurable wait time for requesting executors and minimum registered executors ratio to continue the block distribution Key: CARBONDATA-2043 URL: https://issues.apache.org/jira/browse/CARBONDATA-2043 Project: CarbonData Issue Type: Improvement Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan * Configurable wait time for requesting executors and minimum registered executors ratio to continue the block distribution - carbon.dynamicallocation.schedulertimeout : to configure wait time. defalt 5sec, Min 5 sec and max 15 sec. - carbon.scheduler.minregisteredresourcesratio : min 0.1, max 1.0 and default to 0.8 to configure minimum registered executors ratio. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2009) REFRESH TABLE Limitation When HiveMetaStore is used
[ https://issues.apache.org/jira/browse/CARBONDATA-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-2009: - Description: Refresh table command will not register the carbon table if the old table is stored in the CarbonHiveMetastore (was: Refresh table when spark.carbon.hive.schema.store is set to true ie when hive meta store is used.) > REFRESH TABLE Limitation When HiveMetaStore is used > --- > > Key: CARBONDATA-2009 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2009 > Project: CarbonData > Issue Type: Sub-task >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Arshad >Priority: Minor > Fix For: 1.3.0 > > > Refresh table command will not register the carbon table if the old table is > stored in the CarbonHiveMetastore -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-2009) REFRESH TABLE Limitation When HiveMetaStore is used
[ https://issues.apache.org/jira/browse/CARBONDATA-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan reassigned CARBONDATA-2009: Assignee: Mohammad Arshad > REFRESH TABLE Limitation When HiveMetaStore is used > --- > > Key: CARBONDATA-2009 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2009 > Project: CarbonData > Issue Type: Sub-task >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Arshad >Priority: Minor > Fix For: 1.3.0 > > > Refresh table when spark.carbon.hive.schema.store is set to true ie when hive > meta store is used. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-2009) REFRESH TABLE Limitation When HiveMetaStore is used
Mohammad Shahid Khan created CARBONDATA-2009: Summary: REFRESH TABLE Limitation When HiveMetaStore is used Key: CARBONDATA-2009 URL: https://issues.apache.org/jira/browse/CARBONDATA-2009 Project: CarbonData Issue Type: Sub-task Reporter: Mohammad Shahid Khan Priority: Minor Refresh table when spark.carbon.hive.schema.store is set to true ie when hive meta store is used. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1955) Delta DataType calculation is incorrect for long type
[ https://issues.apache.org/jira/browse/CARBONDATA-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1955: - Affects Version/s: 1.2.0 > Delta DataType calculation is incorrect for long type > - > > Key: CARBONDATA-1955 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1955 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0 >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1955) Delta DataType calculation is incorrect
[ https://issues.apache.org/jira/browse/CARBONDATA-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1955: - Description: was: For deciding deltaDataType, adaptiveDataType should be passed to the calculate the deltaDataType otherwise it will end up with source type only. DataType fitDelta(DataType dataType, Object max, Object min) DataType fitDeltaForDecimalType(DataType dataType, Object max, Object min, DecimalConverterFactory.DecimalConverterType decimalConverterType) > Delta DataType calculation is incorrect > --- > > Key: CARBONDATA-1955 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1955 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1955) Delta DataType calculation is incorrect for long type
[ https://issues.apache.org/jira/browse/CARBONDATA-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1955: - Summary: Delta DataType calculation is incorrect for long type (was: Delta DataType calculation is incorrect) > Delta DataType calculation is incorrect for long type > - > > Key: CARBONDATA-1955 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1955 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1955) Delta DataType calculation is incorrect
Mohammad Shahid Khan created CARBONDATA-1955: Summary: Delta DataType calculation is incorrect Key: CARBONDATA-1955 URL: https://issues.apache.org/jira/browse/CARBONDATA-1955 Project: CarbonData Issue Type: Bug Affects Versions: 1.3.0 Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan For deciding deltaDataType, adaptiveDataType should be passed to the calculate the deltaDataType otherwise it will end up with source type only. DataType fitDelta(DataType dataType, Object max, Object min) DataType fitDeltaForDecimalType(DataType dataType, Object max, Object min, DecimalConverterFactory.DecimalConverterType decimalConverterType) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1948) Update help document for the change made for CARBONDATA-1929
Mohammad Shahid Khan created CARBONDATA-1948: Summary: Update help document for the change made for CARBONDATA-1929 Key: CARBONDATA-1948 URL: https://issues.apache.org/jira/browse/CARBONDATA-1948 Project: CarbonData Issue Type: Sub-task Reporter: Mohammad Shahid Khan Assignee: Gururaj Shetty Priority: Minor carbon.sort.file.write.buffer.size (minValue = 10 KB, maxValue=10MB, defaultValue =16 KB ) carbon.sort.intermediate.files.limit (minValue = 2, maxValue=50, defaultValue =20 ) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1929) carbon property configuration validation
[ https://issues.apache.org/jira/browse/CARBONDATA-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1929: - Description: Issue 1 : In carbon.properties carbon.timestamp.format=testserwerwerwer12312312312312 is configured and spark submit server is started using the command - bin/spark-submit --master yarn-client --executor-memory 10G --executor-cores 5 --driver-memory 5G --num-executors 3 --class org.apache.carbondata.spark.thriftserver.CarbonThriftServer --jars /srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-common-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-core-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-format-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-hadoop-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-processing-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-spark-common-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-streaming-1.3.0.0100.jar /srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-spark2-1.3.0.0100.jar "hdfs://hacluster/carbonstore" From Beeline User tries to create a table Issue 2 : In carbon.properties the below properties is configured and spark submit server is started using the command carbon.sort.file.write.buffer.size=test carbon.sort.intermediate.files.limit=test >From Beeline User creates a table User tries to load data. Issue 3: In carbon.properties the below properties is configured and spark submit server is started using the command carbon.block.meta.size.reserved.percentage=test >From beeline user creates table and tries to load data. was: Issue 1 : In carbon.properties carbon.timestamp.format=testserwerwerwer12312312312312 is configured and spark submit server is started using the command - bin/spark-submit --master yarn-client --executor-memory 10G --executor-cores 5 --driver-memory 5G --num-executors 3 --class org.apache.carbondata.spark.thriftserver.CarbonThriftServer --jars /srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-common-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-core-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-format-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-hadoop-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-processing-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-spark-common-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-streaming-1.3.0.0100.jar /srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-spark2-1.3.0.0100.jar "hdfs://hacluster/carbonstore" From Beeline User tries to create a table Create table parsing fails as shown below in snap. http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=sEDRdBxs6%2f6a0ax2Ss3KJMdclw69E%2f%2bmwFVkT%2f6WxsOF1uA6IA71DkJFN3iXJAG7vplZsvI%2bl7M%3d Issue 2 : In carbon.properties the below properties is configured and spark submit server is started using the command carbon.sort.file.write.buffer.size=test carbon.sort.intermediate.files.limit=test >From Beeline User creates a table User tries to load data. Issue :Data loading fails as shown in snap. http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=sEDRdBxs6%2f6a0ax2Ss3KJMdclw69E%2f%2bmwFVkT%2f6WxsOF1uA6IA71DjeZqxHmMLFdeVr460BZPL4%3d Issue 3: In carbon.properties the below properties is configured and spark submit server is started using the command carbon.block.meta.size.reserved.percentage=test >From beeline user creates table and tries to load data. Issue :Data loading fails as shown in snap. http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=sEDRdBxs6%2f6a0ax2Ss3KJMdclw69E%2f%2bmwFVkT%2f6WxsOF1uA6IA71Ds5xK0%2fNEhBDzhdFVtoc3sg%3d > carbon property configuration validation > > > Key: CARBONDATA-1929 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1929 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan > > Issue 1 : > In carbon.properties carbon.timestamp.format=testserwerwerwer12312312312312 > is configured and spark submit server is started using the command - > bin/spark-submit --master yarn-client --executor-memory 10G --executor-cores > 5 --driver-memory 5G --num-executors 3 --class > org.apache.carbondata.spark.thriftserver.CarbonThriftServer --jars >
[jira] [Created] (CARBONDATA-1929) carbon property configuration validation
Mohammad Shahid Khan created CARBONDATA-1929: Summary: carbon property configuration validation Key: CARBONDATA-1929 URL: https://issues.apache.org/jira/browse/CARBONDATA-1929 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Issue 1 : In carbon.properties carbon.timestamp.format=testserwerwerwer12312312312312 is configured and spark submit server is started using the command - bin/spark-submit --master yarn-client --executor-memory 10G --executor-cores 5 --driver-memory 5G --num-executors 3 --class org.apache.carbondata.spark.thriftserver.CarbonThriftServer --jars /srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-common-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-core-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-format-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-hadoop-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-processing-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-spark-common-1.3.0.0100.jar,/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-streaming-1.3.0.0100.jar /srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata-spark2-1.3.0.0100.jar "hdfs://hacluster/carbonstore" From Beeline User tries to create a table Create table parsing fails as shown below in snap. http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=sEDRdBxs6%2f6a0ax2Ss3KJMdclw69E%2f%2bmwFVkT%2f6WxsOF1uA6IA71DkJFN3iXJAG7vplZsvI%2bl7M%3d Issue 2 : In carbon.properties the below properties is configured and spark submit server is started using the command carbon.sort.file.write.buffer.size=test carbon.sort.intermediate.files.limit=test >From Beeline User creates a table User tries to load data. Issue :Data loading fails as shown in snap. http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=sEDRdBxs6%2f6a0ax2Ss3KJMdclw69E%2f%2bmwFVkT%2f6WxsOF1uA6IA71DjeZqxHmMLFdeVr460BZPL4%3d Issue 3: In carbon.properties the below properties is configured and spark submit server is started using the command carbon.block.meta.size.reserved.percentage=test >From beeline user creates table and tries to load data. Issue :Data loading fails as shown in snap. http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=sEDRdBxs6%2f6a0ax2Ss3KJMdclw69E%2f%2bmwFVkT%2f6WxsOF1uA6IA71Ds5xK0%2fNEhBDzhdFVtoc3sg%3d -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (CARBONDATA-929) adding optional field number of blocklet in the carbonindex file
[
https://issues.apache.org/jira/browse/CARBONDATA-929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Shahid Khan resolved CARBONDATA-929.
-
Resolution: Won't Fix
Not required for current design.
Please reopen if required in future.
> adding optional field number of blocklet in the carbonindex file
> -
>
> Key: CARBONDATA-929
> URL: https://issues.apache.org/jira/browse/CARBONDATA-929
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Mohammad Shahid Khan
>Assignee: Mohammad Shahid Khan
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The num_blocklet needed to get the number of Blocklet within a block at the
> driver
> {code}
> /**
> * Block index information stored in index file for every block
> */
> struct BlockIndex{
> 1: required i64 num_rows; // Total number of rows in this file
> 2: required string file_name; // Block file name
> 3: required i64 offset; // Offset of the footer
> 4: required carbondata.BlockletIndex block_index; // Blocklet index
> 5: optional i32 num_blocklet; // total number of Blocklet
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CARBONDATA-1913) Global Sort data dataload fails for big with RPC timeout exception
Mohammad Shahid Khan created CARBONDATA-1913: Summary: Global Sort data dataload fails for big with RPC timeout exception Key: CARBONDATA-1913 URL: https://issues.apache.org/jira/browse/CARBONDATA-1913 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan When gloabl sort option is used for big data then for some times it fails with RPC timeout after 120s. This is happening because the driver is not able to unpersist rdd cache with in 120s. The issue is happening due to rdd unpersist blocking call. Sometimes spark is not able to unppersist the rdd in default "spark.rpc.askTimeout" or "spark.network.timeout" time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1341) the data load not failing for bigdecimal datatype even though if action is fail and data is not in valid range
[ https://issues.apache.org/jira/browse/CARBONDATA-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1341: - Description: Create table t1(c1 Decimal(3,2), c2 String) stored by 'carbondata' For column c1, the expected allowed data should be 2 2.1 2.11 Invalid data 2.111 Actual: But when we load / insert data 2.111 carbon is storing as null, when action is fail. Expected : The load should fail if bad record action is fail. > the data load not failing for bigdecimal datatype even though if action is > fail and data is not in valid range > -- > > Key: CARBONDATA-1341 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1341 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Priority: Minor > > Create table t1(c1 Decimal(3,2), c2 String) stored by 'carbondata' > For column c1, the expected allowed data should be > 2 > 2.1 > 2.11 > Invalid data > 2.111 > Actual: But when we load / insert data 2.111 carbon is storing as null, when > action is fail. > Expected : The load should fail if bad record action is fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1887) block pruning not happening is carbon for ShortType and SmallIntType columns
Mohammad Shahid Khan created CARBONDATA-1887: Summary: block pruning not happening is carbon for ShortType and SmallIntType columns Key: CARBONDATA-1887 URL: https://issues.apache.org/jira/browse/CARBONDATA-1887 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan spark.sql( s""" | create table test_numeric_type(c1 int, c2 long, c3 smallint, c4 bigint, c5 short) stored by 'carbondata' """.stripMargin).show() spark.sql( s""" | insert into test_numeric_type select 1,111,111,11,'2019-01-03 12:12:12' """.stripMargin).show() spark.sql( s""" | insert into test_numeric_type select 2,222,222,22,'2020-01-03 12:12:12' """.stripMargin).show() spark.sql( s""" | insert into test_numeric_type select 3,333,333,33,'2021-01-03 12:12:12' """.stripMargin).show() spark.sql( s""" | select * from test_numeric_type where c5> """.stripMargin).show() Only two blocks should be selected but all blocks are selected during query execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1293) update on carbon data failed with carbon.update.persist.enable false
[ https://issues.apache.org/jira/browse/CARBONDATA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1293: - Priority: Major (was: Critical) > update on carbon data failed with carbon.update.persist.enable false > > > Key: CARBONDATA-1293 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1293 > Project: CarbonData > Issue Type: Bug > Components: data-load >Reporter: Mohammad Shahid Khan > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1849) n Create pre aggregate table failled
Mohammad Shahid Khan created CARBONDATA-1849:
Summary: n Create pre aggregate table failled
Key: CARBONDATA-1849
URL: https://issues.apache.org/jira/browse/CARBONDATA-1849
Project: CarbonData
Issue Type: Bug
Components: spark-integration
Reporter: Mohammad Shahid Khan
Create pre aggregate table is failing
spark.sql("""create table carbontable (c1 string,c2 int,c3 string,c5 string)
STORED BY 'org.apache.carbondata.format'""")
spark.sql("insert into carbontable select 'a',1,'aa','aaa'")
spark. sql("insert into carbontable select 'b',1,'aa','aaa'")
spark.sql("insert into carbontable select 'a',10,'aa','aaa'")
spark.sql("create datamap preagg1 on table carbontable using 'preaggregate'
as select c1,sum(c2) from carbontable group by c1")
Exception:
{code}
Exception in thread "main"
org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException:
CSV header in DDL is not proper. Column names in schema and CSV header are not
the same.
at
org.apache.carbondata.spark.util.CommonUtil$.getCsvHeaderColumns(CommonUtil.scala:637)
at
org.apache.carbondata.spark.util.DataLoadingUtil$.buildCarbonLoadModel(DataLoadingUtil.scala:312)
at
org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand.processData(CarbonLoadDataCommand.scala:133)
at
org.apache.spark.sql.execution.command.DataCommand.run(package.scala:71)
at
org.apache.spark.sql.execution.command.preaaggregate.CreatePreAggregateTableCommand.processData(CreatePreAggregateTableCommand.scala:139)
at
org.apache.spark.sql.execution.command.datamap.CarbonCreateDataMapCommand.processData(CarbonCreateDataMapCommand.scala:80)
at
org.apache.spark.sql.execution.command.AtomicRunnableCommand.run(package.scala:86)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
at org.apache.spark.sql.Dataset.(Dataset.scala:185)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
at
org.apache.carbondata.examples.CarbonSessionExample$.main(CarbonSessionExample.scala:35)
at
org.apache.carbondata.examples.CarbonSessionExample.main(CarbonSessionExample.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CARBONDATA-1822) Support DDL to register the CarbonData table from existing carbon table data
Mohammad Shahid Khan created CARBONDATA-1822: Summary: Support DDL to register the CarbonData table from existing carbon table data Key: CARBONDATA-1822 URL: https://issues.apache.org/jira/browse/CARBONDATA-1822 Project: CarbonData Issue Type: New Feature Components: spark-integration Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan For details please refer to the below link http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DDL-for-CarbonData-table-backup-and-recovery-new-feature-td27854.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1249) Wrong order of columns in redirected csv for bad records
[ https://issues.apache.org/jira/browse/CARBONDATA-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1249: - Summary: Wrong order of columns in redirected csv for bad records (was: Wrong order of columns i n redirected csv for bad records) > Wrong order of columns in redirected csv for bad records > > > Key: CARBONDATA-1249 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1249 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Critical > Time Spent: 6h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1249) Wrong order of columns in redirected csv for bad records
[ https://issues.apache.org/jira/browse/CARBONDATA-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1249: - Priority: Minor (was: Critical) > Wrong order of columns in redirected csv for bad records > > > Key: CARBONDATA-1249 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1249 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Minor > Time Spent: 6h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1727) Dataload is successful even in case if the table is droped from other client.
[ https://issues.apache.org/jira/browse/CARBONDATA-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1727: - Summary: Dataload is successful even in case if the table is droped from other client. (was: Dataload is successful even in case if some user drops the table from other client.) > Dataload is successful even in case if the table is droped from other client. > - > > Key: CARBONDATA-1727 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1727 > Project: CarbonData > Issue Type: Bug > Components: data-load, spark-integration >Affects Versions: 1.2.0, 1.3.0 >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Minor > > table drop has highest priority so even if on some table load operation is in > progress, > the table could be droped. > If before finishing the load operation the table is dropped then the load > should fail. > Steps: > 1. Create table t1 > 2. Load data into t1 (big data that takes some time) > 3. when load in progress, then drop the table > Actual Result : The load is successful > Expected Result : Final Load status should be fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1727) Dataload is successful even in case if some user drops the table from other client.
Mohammad Shahid Khan created CARBONDATA-1727: Summary: Dataload is successful even in case if some user drops the table from other client. Key: CARBONDATA-1727 URL: https://issues.apache.org/jira/browse/CARBONDATA-1727 Project: CarbonData Issue Type: Bug Components: data-load, spark-integration Affects Versions: 1.2.0, 1.3.0 Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan Priority: Minor table drop has highest priority so even if on some table load operation is in progress, the table could be droped. If before finishing the load operation the table is dropped then the load should fail. Steps: 1. Create table t1 2. Load data into t1 (big data that takes some time) 3. when load in progress, then drop the table Actual Result : The load is successful Expected Result : Final Load status should be fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1573) Support Database Location Configuration while Creating Database/ Support Creation of carbon Table in the database location
[ https://issues.apache.org/jira/browse/CARBONDATA-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1573: - Summary: Support Database Location Configuration while Creating Database/ Support Creation of carbon Table in the database location (was: Support Database Location Configuration while Creating Database) > Support Database Location Configuration while Creating Database/ Support > Creation of carbon Table in the database location > -- > > Key: CARBONDATA-1573 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1573 > Project: CarbonData > Issue Type: New Feature > Components: core, data-load, data-query, hadoop-integration, > presto-integration, spark-integration >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > Support Creation of carbon table at the database location > *Please refer to for Design and discussion:* > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-Support-Database-Location-Configuration-while-Creating-Database-td23492.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Issue Comment Deleted] (CARBONDATA-1507) Dataload global sort fails with RpcTimeOutException: Futures timed out after [120 seconds]
[ https://issues.apache.org/jira/browse/CARBONDATA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1507: - Comment: was deleted (was: This happening due to rdd unpersist blocking calll. Sometime the RDD unpersist is not happening in default "spark.rp.askTimeout" or "spark.network.timeout" time. Solution, we may make the unpersist call as non blocking. And doing this will not have any functional impact as spark automatically monitors the cache usage on each node and drops out.) > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] > -- > > Key: CARBONDATA-1507 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1507 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Issue Comment Deleted] (CARBONDATA-1507) Dataload global sort fails with RpcTimeOutException: Futures timed out after [120 seconds]
[ https://issues.apache.org/jira/browse/CARBONDATA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1507: - Comment: was deleted (was: This happening due to rdd unpersist blocking calll. Sometime the RDD unpersist is not happening in default "spark.rp.askTimeout" or "spark.network.timeout" time. Solution, we may make the unpersist call as non blocking. And doing this will not have any functional impact as spark automatically monitors the cache usage on each node and drops out.) > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] > -- > > Key: CARBONDATA-1507 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1507 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CARBONDATA-1507) Dataload global sort fails with RpcTimeOutException: Futures timed out after [120 seconds]
[ https://issues.apache.org/jira/browse/CARBONDATA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175990#comment-16175990 ] Mohammad Shahid Khan commented on CARBONDATA-1507: -- This happening due to rdd unpersist blocking calll. Sometime the RDD unpersist is not happening in default "spark.rp.askTimeout" or "spark.network.timeout" time. Solution, we may make the unpersist call as non blocking. And doing this will not have any functional impact as spark automatically monitors the cache usage on each node and drops out. > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] > -- > > Key: CARBONDATA-1507 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1507 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CARBONDATA-1507) Dataload global sort fails with RpcTimeOutException: Futures timed out after [120 seconds]
[ https://issues.apache.org/jira/browse/CARBONDATA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175989#comment-16175989 ] Mohammad Shahid Khan commented on CARBONDATA-1507: -- This happening due to rdd unpersist blocking calll. Sometime the RDD unpersist is not happening in default "spark.rp.askTimeout" or "spark.network.timeout" time. Solution, we may make the unpersist call as non blocking. And doing this will not have any functional impact as spark automatically monitors the cache usage on each node and drops out. > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] > -- > > Key: CARBONDATA-1507 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1507 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CARBONDATA-1507) Dataload global sort fails with RpcTimeOutException: Futures timed out after [120 seconds]
[ https://issues.apache.org/jira/browse/CARBONDATA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175988#comment-16175988 ] Mohammad Shahid Khan commented on CARBONDATA-1507: -- This happening due to rdd unpersist blocking calll. Sometime the RDD unpersist is not happening in default "spark.rp.askTimeout" or "spark.network.timeout" time. Solution, we may make the unpersist call as non blocking. And doing this will not have any functional impact as spark automatically monitors the cache usage on each node and drops out. > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] > -- > > Key: CARBONDATA-1507 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1507 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > Dataload global sort fails with RpcTimeOutException: Futures timed out after > [120 seconds] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1507) Dataload global sort fails with RpcTimeOutException: Futures timed out after [120 seconds]
Mohammad Shahid Khan created CARBONDATA-1507: Summary: Dataload global sort fails with RpcTimeOutException: Futures timed out after [120 seconds] Key: CARBONDATA-1507 URL: https://issues.apache.org/jira/browse/CARBONDATA-1507 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan Dataload global sort fails with RpcTimeOutException: Futures timed out after [120 seconds] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1341) the data load not failing for bigdecimal datatype even though if action is fail and data is not in valid range
Mohammad Shahid Khan created CARBONDATA-1341: Summary: the data load not failing for bigdecimal datatype even though if action is fail and data is not in valid range Key: CARBONDATA-1341 URL: https://issues.apache.org/jira/browse/CARBONDATA-1341 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1309) During Global Sort the DataWriterProcessorStepImpl not getting shutdown even when datalaod is finished.
[ https://issues.apache.org/jira/browse/CARBONDATA-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan reassigned CARBONDATA-1309: Assignee: Mohammad Shahid Khan > During Global Sort the DataWriterProcessorStepImpl not getting shutdown even > when datalaod is finished. > --- > > Key: CARBONDATA-1309 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1309 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > Global Sort is not closing the DataWriterProcessorStepImpl because of this > the thread started to print the number of records processed by the step at > the interval of 10 second is not getting shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1309) During Global Sort the DataWriterProcessorStepImpl not getting shutdown even when datalaod is finished.
Mohammad Shahid Khan created CARBONDATA-1309: Summary: During Global Sort the DataWriterProcessorStepImpl not getting shutdown even when datalaod is finished. Key: CARBONDATA-1309 URL: https://issues.apache.org/jira/browse/CARBONDATA-1309 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Global Sort is not closing the DataWriterProcessorStepImpl because of this the thread started to print the number of records processed by the step at the interval of 10 second is not getting shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1304) Support IUD with single_pass
Mohammad Shahid Khan created CARBONDATA-1304: Summary: Support IUD with single_pass Key: CARBONDATA-1304 URL: https://issues.apache.org/jira/browse/CARBONDATA-1304 Project: CarbonData Issue Type: Bug Components: data-load, data-query Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1293) update on carbon data failed with carbon.update.persist.enable false
Mohammad Shahid Khan created CARBONDATA-1293: Summary: update on carbon data failed with carbon.update.persist.enable false Key: CARBONDATA-1293 URL: https://issues.apache.org/jira/browse/CARBONDATA-1293 Project: CarbonData Issue Type: Bug Components: data-load Reporter: Mohammad Shahid Khan Priority: Critical -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1283) Carbon should continue with the default value if wrong value is set for the configurable parameter.
Mohammad Shahid Khan created CARBONDATA-1283: Summary: Carbon should continue with the default value if wrong value is set for the configurable parameter. Key: CARBONDATA-1283 URL: https://issues.apache.org/jira/browse/CARBONDATA-1283 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Carbon should continue with the default value if wrong value is set for the configurable parameters. for example for below parameters if wrong value is configured then the executions is failing in the respective scenarios. enable.unsafe.sort carbon.custom.block.distribution carbon.enable.vector.reader carbon.lock.type carbon.csv.read.buffersize.byte -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1258) CarbonData should not allow loading Date Type values violating the boundary condition ("0001-01-01" through "9999-12-31")
Mohammad Shahid Khan created CARBONDATA-1258:
Summary: CarbonData should not allow loading Date Type values
violating the boundary condition ("0001-01-01" through "-12-31")
Key: CARBONDATA-1258
URL: https://issues.apache.org/jira/browse/CARBONDATA-1258
Project: CarbonData
Issue Type: Bug
Reporter: Mohammad Shahid Khan
Assignee: Mohammad Shahid Khan
Priority: Critical
CarbonData should not allow loading Date Type values violating the boundary
condition ("0001-01-01" through "-12-31")
For Date Type the value X being loaded must be X >= "0001-01-01" and X <=
"-12-31".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1252) Add BAD_RECORD_PATH option in Load options section in the Carbon Help doc
[ https://issues.apache.org/jira/browse/CARBONDATA-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1252: - Summary: Add BAD_RECORD_PATH option in Load options section in the Carbon Help doc (was: Add BAD_RECORD_PATH options in Load options in the Carbon Help doc) > Add BAD_RECORD_PATH option in Load options section in the Carbon Help doc > - > > Key: CARBONDATA-1252 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1252 > Project: CarbonData > Issue Type: Sub-task >Reporter: Mohammad Shahid Khan >Priority: Minor > Fix For: 1.2.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1252) Add BAD_RECORD_PATH options in Load options in the Carbon Help doc
Mohammad Shahid Khan created CARBONDATA-1252: Summary: Add BAD_RECORD_PATH options in Load options in the Carbon Help doc Key: CARBONDATA-1252 URL: https://issues.apache.org/jira/browse/CARBONDATA-1252 Project: CarbonData Issue Type: Sub-task Reporter: Mohammad Shahid Khan Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CARBONDATA-1117) Update SET & RESET command details in online help documentation
[ https://issues.apache.org/jira/browse/CARBONDATA-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069604#comment-16069604 ] Mohammad Shahid Khan commented on CARBONDATA-1117: -- Please refer PR #https://github.com/apache/carbondata/pull/972 for details For information Updated supported parameters set -v command displays all session parameter list along with default value and usage doc Example : set -v; Supported parameters for dynamic set. carbon.options.bad.records.logger.enable carbon.options.bad.records.action carbon.options.is.empty.data.bad.record carbon.options.sort.scope carbon.options.batch.sort.size.inmb carbon.options.single.pass carbon.options.bad.record.path carbon.options.global.sort.partitions enable.unsafe.sort carbon.custom.block.distribution > Update SET & RESET command details in online help documentation > --- > > Key: CARBONDATA-1117 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1117 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Manohar Vanam >Assignee: Gururaj Shetty > > Update SET & RESET command details in online help documentation > 1. update syntax > 2. Examples with use cases -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1249) Wrong order of columns i n redirected csv for bad records
Mohammad Shahid Khan created CARBONDATA-1249: Summary: Wrong order of columns i n redirected csv for bad records Key: CARBONDATA-1249 URL: https://issues.apache.org/jira/browse/CARBONDATA-1249 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan Priority: Critical -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1218) In case of data-load failure the BadRecordsLogger.badRecordEntry map holding the task Status is not removing the task Entry.
[ https://issues.apache.org/jira/browse/CARBONDATA-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1218: - Description: In case of data-load failure the BadRecordsLogger.badRecordEntry map holding the task Status is not removing the task Entry. Because of this the next load is getting failed even though the data being loaded has no bad records. The map entry must be removed after load completion either success or fail. was: In case of data-load failure the BadRecordsLogger.badRecordEntry map holding the task Status is not removing the task Entry. Because of this the next load is getting failed even though the data being loaded has no bad records. > In case of data-load failure the BadRecordsLogger.badRecordEntry map holding > the task Status is not removing the task Entry. > > > Key: CARBONDATA-1218 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1218 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > In case of data-load failure the BadRecordsLogger.badRecordEntry map holding > the task Status is not removing the task Entry. > Because of this the next load is getting failed even though the data being > loaded has no bad records. > The map entry must be removed after load completion either success or fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1218) In case of data-load failure the BadRecordsLogger.badRecordEntry map holding the task Status is not removing the task Entry.
[ https://issues.apache.org/jira/browse/CARBONDATA-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan reassigned CARBONDATA-1218: Assignee: Mohammad Shahid Khan > In case of data-load failure the BadRecordsLogger.badRecordEntry map holding > the task Status is not removing the task Entry. > > > Key: CARBONDATA-1218 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1218 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > In case of data-load failure the BadRecordsLogger.badRecordEntry map holding > the task Status is not removing the task Entry. > Because of this the next load is getting failed even though the data being > loaded has no bad records. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1218) In case of data-load failure the BadRecordsLogger.badRecordEntry map holding the task Status is not removing the task Entry.
Mohammad Shahid Khan created CARBONDATA-1218: Summary: In case of data-load failure the BadRecordsLogger.badRecordEntry map holding the task Status is not removing the task Entry. Key: CARBONDATA-1218 URL: https://issues.apache.org/jira/browse/CARBONDATA-1218 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan In case of data-load failure the BadRecordsLogger.badRecordEntry map holding the task Status is not removing the task Entry. Because of this the next load is getting failed even though the data being loaded has no bad records. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1210) Exception should be thrown on bad record logger failure to write to log file or csv file.
[ https://issues.apache.org/jira/browse/CARBONDATA-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-1210: - Summary: Exception should be thrown on bad record logger failure to write to log file or csv file. (was: Exception should be thrown on bad record write failure to log file or csv file.) > Exception should be thrown on bad record logger failure to write to log file > or csv file. > - > > Key: CARBONDATA-1210 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1210 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1210) Exception should be thrown on bad record write failure to log file or csv file.
Mohammad Shahid Khan created CARBONDATA-1210: Summary: Exception should be thrown on bad record write failure to log file or csv file. Key: CARBONDATA-1210 URL: https://issues.apache.org/jira/browse/CARBONDATA-1210 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1207) Resource leak problem in CarbonDictionaryWriter
Mohammad Shahid Khan created CARBONDATA-1207: Summary: Resource leak problem in CarbonDictionaryWriter Key: CARBONDATA-1207 URL: https://issues.apache.org/jira/browse/CARBONDATA-1207 Project: CarbonData Issue Type: Bug Components: data-load Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan If in load during dictionary generation some exception happens during the dictionary writing or dictionary meta data writing, then stream will never be closed. This will load DOS on the incremental load. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1101) Avoid widening between wrapper classes
Mohammad Shahid Khan created CARBONDATA-1101: Summary: Avoid widening between wrapper classes Key: CARBONDATA-1101 URL: https://issues.apache.org/jira/browse/CARBONDATA-1101 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Assignee: Mohammad Shahid Khan UnsafeIntermediateFileMerger.writeDataTofile does widening between Short, Integer, and Long wrapper classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-1077) ColumnDict and ALL_DICTIONARY_PATH must be used with SINGLE_PASS='true'
[ https://issues.apache.org/jira/browse/CARBONDATA-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan reassigned CARBONDATA-1077: Assignee: Mohammad Shahid Khan > ColumnDict and ALL_DICTIONARY_PATH must be used with SINGLE_PASS='true' > > > Key: CARBONDATA-1077 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1077 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Add validation that ColumnDict and ALL_DICTIONARY_PATH must be used with > SINGLE_PASS='true'. > Problem: > If columndict is used without single path, and if the external dictionary > file > does not have all the values present in the actual raw data to be loaded then > load will not have proper data. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-1049) avoid logging data into log file
[ https://issues.apache.org/jira/browse/CARBONDATA-1049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007723#comment-16007723 ] Mohammad Shahid Khan commented on CARBONDATA-1049: -- avoid logging raw data into driver and executor log > avoid logging data into log file > > > Key: CARBONDATA-1049 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1049 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-1049) avoid logging data into log file
Mohammad Shahid Khan created CARBONDATA-1049: Summary: avoid logging data into log file Key: CARBONDATA-1049 URL: https://issues.apache.org/jira/browse/CARBONDATA-1049 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-991) add test case for validation the bad record log file creation at the configured bad store location
[ https://issues.apache.org/jira/browse/CARBONDATA-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan reassigned CARBONDATA-991: --- Assignee: Mohammad Shahid Khan > add test case for validation the bad record log file creation at the > configured bad store location > -- > > Key: CARBONDATA-991 > URL: https://issues.apache.org/jira/browse/CARBONDATA-991 > Project: CarbonData > Issue Type: Test >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Trivial > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-991) add test case for validation the bad record log file creation at the configured bad store location
Mohammad Shahid Khan created CARBONDATA-991: --- Summary: add test case for validation the bad record log file creation at the configured bad store location Key: CARBONDATA-991 URL: https://issues.apache.org/jira/browse/CARBONDATA-991 Project: CarbonData Issue Type: Bug Reporter: Mohammad Shahid Khan Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-991) add test case for validation the bad record log file creation at the configured bad store location
[ https://issues.apache.org/jira/browse/CARBONDATA-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-991: Issue Type: Test (was: Bug) > add test case for validation the bad record log file creation at the > configured bad store location > -- > > Key: CARBONDATA-991 > URL: https://issues.apache.org/jira/browse/CARBONDATA-991 > Project: CarbonData > Issue Type: Test >Reporter: Mohammad Shahid Khan >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
