[
https://issues.apache.org/jira/browse/CARBONDATA-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor resolved CARBONDATA-4231.
--------------------------------------
Fix Version/s: 2.2.0
Resolution: Fixed
> On update operation with 3.1v, cloned spark session is used and set
> properties are lost.
> ----------------------------------------------------------------------------------------
>
> Key: CARBONDATA-4231
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4231
> Project: CarbonData
> Issue Type: Bug
> Reporter: SHREELEKHYA GAMPA
> Priority: Major
> Fix For: 2.2.0
>
>
> *Update operation with bad records property fails with 3.1v.*
> *[Steps to reproduce]:*
> 0: jdbc:hive2://linux-221:22550/> set carbon.options.bad.records.action=force;
> +------------------------------------+--------+
> | key | value |
> +------------------------------------+--------+
> | carbon.options.bad.records.action | force |
> +------------------------------------+--------+
> 1 row selected (0.04 seconds)
> 0: jdbc:hive2://linux-221:22550/> create table t_carbn1(item_type_cd int,
> sell_price bigint, profit decimal(10,4), item_name string, update_time
> timestamp) stored a
> +---------+
> | Result |
> +---------+
> +---------+
> No rows selected (2.117 seconds)
> 0: jdbc:hive2://linux-221:22550/> insert into t_carbn1 select 2,
> 100000,23.3,'Apple','2012-11-11 11:11:11';
> INFO : Execution ID: 858
> +-------------+
> | Segment ID |
> +-------------+
> | 0 |
> +-------------+
> 1 row selected (4.278 seconds)
> 0: jdbc:hive2://linux-221:22550/> update t_carbn1 set (item_type_cd) =
> (item_type_cd/1);
> Error: org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.RuntimeException: Update operation failed. DataLoad failure
> *[Root cause]:*
> On update command, persist is called and with latest 3.1 spark changes, spark
> returns a cloned SparkSession from cacheManager with all specified
> configurations disabled. As now its using different sparkSession for 3.1
> which is not initialized in CarbonEnv. So CarbonEnv.init is called where new
> CarbonSessionInfo is created with no sessionParams. So, the properties set
> were not accessible.
> Spark creates cloned spark session based on following properties:
> 1. spark.sql.optimizer.canChangeCachedPlanOutputPartitioning
> 2. spark.sql.sources.bucketing.autoBucketedScan.enabled
> 3. spark.sql.adaptive.enabled
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
