[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350898#comment-17350898 ] Apache Spark commented on SPARK-21588: -- User 'vinodkc' has created a pull request for this issue: https://github.com/apache/spark/pull/18852 > SQLContext.getConf(key, null) should return null, but it throws NPE > --- > > Key: SPARK-21588 > URL: https://issues.apache.org/jira/browse/SPARK-21588 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Burak Yavuz >Assignee: Vinod KC >Priority: Minor > Fix For: 2.1.2, 2.2.1, 2.3.0 > > > SQLContext.get(key) for a key that is not defined in the conf, and doesn't > have a default value defined, throws a NoSuchElementException. In order to > avoid that, I used a null as the default value, which threw a NPE instead. If > it is null, it shouldn't try to parse the default value in `getConfString` -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115625#comment-16115625 ] Vinod KC commented on SPARK-21588: -- https://github.com/apache/spark/pull/18852 > SQLContext.getConf(key, null) should return null, but it throws NPE > --- > > Key: SPARK-21588 > URL: https://issues.apache.org/jira/browse/SPARK-21588 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Burak Yavuz >Priority: Minor > > SQLContext.get(key) for a key that is not defined in the conf, and doesn't > have a default value defined, throws a NoSuchElementException. In order to > avoid that, I used a null as the default value, which threw a NPE instead. If > it is null, it shouldn't try to parse the default value in `getConfString` -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113519#comment-16113519 ] Burak Yavuz commented on SPARK-21588: - that's what I was proposing. `null` seemed more familiar than `` before I looked at the code. > SQLContext.getConf(key, null) should return null, but it throws NPE > --- > > Key: SPARK-21588 > URL: https://issues.apache.org/jira/browse/SPARK-21588 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Burak Yavuz >Priority: Minor > > SQLContext.get(key) for a key that is not defined in the conf, and doesn't > have a default value defined, throws a NoSuchElementException. In order to > avoid that, I used a null as the default value, which threw a NPE instead. If > it is null, it shouldn't try to parse the default value in `getConfString` -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113515#comment-16113515 ] Anton Okolnychyi commented on SPARK-21588: -- Sure, but the converter will not be called if the default value that you pass is "". However, the check can be extended to `defaultValue != null && defaultValue != ""` in the SQLConf#getConfString. > SQLContext.getConf(key, null) should return null, but it throws NPE > --- > > Key: SPARK-21588 > URL: https://issues.apache.org/jira/browse/SPARK-21588 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Burak Yavuz >Priority: Minor > > SQLContext.get(key) for a key that is not defined in the conf, and doesn't > have a default value defined, throws a NoSuchElementException. In order to > avoid that, I used a null as the default value, which threw a NPE instead. If > it is null, it shouldn't try to parse the default value in `getConfString` -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113422#comment-16113422 ] Burak Yavuz commented on SPARK-21588: - [~vinodkc] [~aokolnychyi] It happens when the config has a value converter, example `spark.sql.shuffle.partitions`. Basically any non-string sql conf. > SQLContext.getConf(key, null) should return null, but it throws NPE > --- > > Key: SPARK-21588 > URL: https://issues.apache.org/jira/browse/SPARK-21588 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Burak Yavuz >Priority: Minor > > SQLContext.get(key) for a key that is not defined in the conf, and doesn't > have a default value defined, throws a NoSuchElementException. In order to > avoid that, I used a null as the default value, which threw a NPE instead. If > it is null, it shouldn't try to parse the default value in `getConfString` -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE
[
https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113412#comment-16113412
]
Anton Okolnychyi commented on SPARK-21588:
--
I did not manage to reproduce this. I tried:
{code}
spark.sqlContext.getConf("spark.sql.streaming.checkpointLocation", null) // null
spark.sqlContext.getConf("spark.sql.thriftserver.scheduler.pool", null) // null
spark.sqlContext.getConf("spark.sql.sources.outputCommitterClass", null) // null
spark.sqlContext.getConf("blabla", null) // null
spark.sqlContext.getConf("spark.sql.sources.outputCommitterClass") //
{code}
I got a NPE only when I called getConf(key, null) for a parameter with a
default value. For example,
{code}
spark.sqlContext.getConf("spark.sql.thriftServer.incrementalCollect",
"") //
spark.sqlContext.getConf("spark.sql.thriftServer.incrementalCollect", null) //
NPE
{code}
> SQLContext.getConf(key, null) should return null, but it throws NPE
> ---
>
> Key: SPARK-21588
> URL: https://issues.apache.org/jira/browse/SPARK-21588
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Burak Yavuz
>Priority: Minor
>
> SQLContext.get(key) for a key that is not defined in the conf, and doesn't
> have a default value defined, throws a NoSuchElementException. In order to
> avoid that, I used a null as the default value, which threw a NPE instead. If
> it is null, it shouldn't try to parse the default value in `getConfString`
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108622#comment-16108622 ] Vinod KC commented on SPARK-21588: -- [~brkyvz] Can you share sample code and NPE stack trace? > SQLContext.getConf(key, null) should return null, but it throws NPE > --- > > Key: SPARK-21588 > URL: https://issues.apache.org/jira/browse/SPARK-21588 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Burak Yavuz >Priority: Minor > > SQLContext.get(key) for a key that is not defined in the conf, and doesn't > have a default value defined, throws a NoSuchElementException. In order to > avoid that, I used a null as the default value, which threw a NPE instead. If > it is null, it shouldn't try to parse the default value in `getConfString` -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
