[jira] [Comment Edited] (SPARK-19490) Hive partition columns are case-sensitive

2018-08-28 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595852#comment-16595852 ] Harish edited comment on SPARK-19490 at 8/29/18 2:19 AM: - I see the same issue

[jira] [Commented] (SPARK-19490) Hive partition columns are case-sensitive

2018-08-28 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595852#comment-16595852 ] Harish commented on SPARK-19490: I > Hive partition columns are case-sensitive >

[jira] [Resolved] (SPARK-24765) Add custom Kubernetes scheduler config parameter to spark-submit

2018-07-10 Thread Nihal Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Harish resolved SPARK-24765. -- Resolution: Information Provided Issue is addressed by 

[jira] [Updated] (SPARK-24765) Add custom Kubernetes scheduler config parameter to spark-submit

2018-07-10 Thread Nihal Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Harish updated SPARK-24765: - Description: spark submit currently does not accept any config parameter that can enable the

[jira] [Created] (SPARK-24765) Add custom Kubernetes scheduler config parameter to spark-submit

2018-07-09 Thread Nihal Harish (JIRA)
Nihal Harish created SPARK-24765: Summary: Add custom Kubernetes scheduler config parameter to spark-submit Key: SPARK-24765 URL: https://issues.apache.org/jira/browse/SPARK-24765 Project: Spark

[jira] [Issue Comment Deleted] (SPARK-18681) Throw Filtering is supported only on partition keys of type string exception

2018-06-25 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish updated SPARK-18681: --- Comment: was deleted (was: [~michael] I see the same issue in 2.3.1. I have hive partitioned table with

[jira] [Commented] (SPARK-18681) Throw Filtering is supported only on partition keys of type string exception

2018-06-25 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523049#comment-16523049 ] Harish commented on SPARK-18681: [~michael] I see the same issue in 2.3.1. I have hive partitioned table

[jira] [Comment Edited] (SPARK-19243) Error when selecting from DataFrame containing parsed data from files larger than 1MB

2017-05-03 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995489#comment-15995489 ] Harish edited comment on SPARK-19243 at 5/3/17 7:52 PM: i am getting the same

[jira] [Commented] (SPARK-19243) Error when selecting from DataFrame containing parsed data from files larger than 1MB

2017-05-03 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995489#comment-15995489 ] Harish commented on SPARK-19243: i am getting the same error in spark 2.1.0. I have 10 node cluster with

[jira] [Created] (SPARK-20362) spark submit not considering user defined Configs (Pyspark)

2017-04-17 Thread Harish (JIRA)
Harish created SPARK-20362: -- Summary: spark submit not considering user defined Configs (Pyspark) Key: SPARK-20362 URL: https://issues.apache.org/jira/browse/SPARK-20362 Project: Spark Issue Type:

[jira] [Commented] (SPARK-20022) java.lang.OutOfMemoryError: Unable to acquire 4228 bytes of memory

2017-03-20 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933121#comment-15933121 ] Harish commented on SPARK-20022: Isn't it same as "https://issues.apache.org/jira/browse/SPARK-14363; I

[jira] [Updated] (SPARK-20022) java.lang.OutOfMemoryError: Unable to acquire 4228 bytes of memory

2017-03-19 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish updated SPARK-20022: --- Description: I am getting below error in 2.0.2. Any help? or work around? WARN TaskSetManager: Lost task

[jira] [Updated] (SPARK-20022) java.lang.OutOfMemoryError: Unable to acquire 4228 bytes of memory

2017-03-19 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish updated SPARK-20022: --- Description: I am getting below error in 2.0.2. Any help? WARN TaskSetManager: Lost task 34.0 in stage

[jira] [Created] (SPARK-20022) java.lang.OutOfMemoryError: Unable to acquire 4228 bytes of memory

2017-03-19 Thread Harish (JIRA)
Harish created SPARK-20022: -- Summary: java.lang.OutOfMemoryError: Unable to acquire 4228 bytes of memory Key: SPARK-20022 URL: https://issues.apache.org/jira/browse/SPARK-20022 Project: Spark

[jira] [Commented] (SPARK-18789) Save Data frame with Null column-- exception

2017-03-16 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929259#comment-15929259 ] Harish commented on SPARK-18789: When you create the DF (dynamic) withough knowing the type of the column

[jira] [Commented] (SPARK-18789) Save Data frame with Null column-- exception

2017-03-16 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928926#comment-15928926 ] Harish commented on SPARK-18789: In your example you are defining the schema first and then loading the

[jira] [Updated] (SPARK-18789) Save Data frame with Null column-- exception

2016-12-09 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish updated SPARK-18789: --- Summary: Save Data frame with Null column-- exception (was: Save Data frame with Null column exception) >

[jira] [Created] (SPARK-18789) Save Data frame with Null column exception

2016-12-08 Thread Harish (JIRA)
Harish created SPARK-18789: -- Summary: Save Data frame with Null column exception Key: SPARK-18789 URL: https://issues.apache.org/jira/browse/SPARK-18789 Project: Spark Issue Type: Bug Affects

[jira] [Updated] (SPARK-18496) java.lang.AssertionError: assertion failed

2016-11-17 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish updated SPARK-18496: --- Affects Version/s: 2.0.2 > java.lang.AssertionError: assertion failed >

[jira] [Created] (SPARK-18496) java.lang.AssertionError: assertion failed

2016-11-17 Thread Harish (JIRA)
Harish created SPARK-18496: -- Summary: java.lang.AssertionError: assertion failed Key: SPARK-18496 URL: https://issues.apache.org/jira/browse/SPARK-18496 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

2016-11-14 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665010#comment-15665010 ] Harish commented on SPARK-16845: This issue is still open - https://github.com/apache/spark/pull/15480

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-11-07 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646453#comment-15646453 ] Harish commented on SPARK-17463: I was able to figure out the issue, its not related to this bug. >

[jira] [Created] (SPARK-18238) WARN Executor: 1 block locks were not released by TID

2016-11-02 Thread Harish (JIRA)
Harish created SPARK-18238: -- Summary: WARN Executor: 1 block locks were not released by TID Key: SPARK-18238 URL: https://issues.apache.org/jira/browse/SPARK-18238 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-10-31 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620885#comment-15620885 ] Harish edited comment on SPARK-16740 at 10/31/16 12:38 PM: --- Thank you. I

[jira] [Commented] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-10-30 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620885#comment-15620885 ] Harish commented on SPARK-16740: Thank you. I downloaded the 2.0.2 snapshot with 2.7 Hadoop (i think its

[jira] [Commented] (SPARK-16740) joins.LongToUnsafeRowMap crashes with NegativeArraySizeException

2016-10-30 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620711#comment-15620711 ] Harish commented on SPARK-16740: is this fix is available in 2.0.2 snapshot?. Please confirm >

[jira] [Commented] (SPARK-16522) [MESOS] Spark application throws exception on exit

2016-10-30 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620225#comment-15620225 ] Harish commented on SPARK-16522: I am getting same error in spark 2.0.2 snapshot. Standalone submission.

[jira] [Created] (SPARK-17948) WARN CodeGenerator: Error calculating stats of compiled class

2016-10-14 Thread Harish (JIRA)
Harish created SPARK-17948: -- Summary: WARN CodeGenerator: Error calculating stats of compiled class Key: SPARK-17948 URL: https://issues.apache.org/jira/browse/SPARK-17948 Project: Spark Issue

[jira] [Resolved] (SPARK-17942) OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=

2016-10-14 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish resolved SPARK-17942. Resolution: Works for Me > OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using >

[jira] [Comment Edited] (SPARK-17942) OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=

2016-10-14 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576375#comment-15576375 ] Harish edited comment on SPARK-17942 at 10/14/16 8:20 PM: -- --conf

[jira] [Commented] (SPARK-17942) OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=

2016-10-14 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576375#comment-15576375 ] Harish commented on SPARK-17942: --conf "spark.executor.extraJavaOptions=-XX:ReservedCodeCacheSize=600m"

[jira] [Updated] (SPARK-17942) OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=

2016-10-14 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish updated SPARK-17942: --- Priority: Minor (was: Major) > OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using

[jira] [Created] (SPARK-17942) OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=

2016-10-14 Thread Harish (JIRA)
Harish created SPARK-17942: -- Summary: OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= Key: SPARK-17942 URL: https://issues.apache.org/jira/browse/SPARK-17942

[jira] [Commented] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

2016-10-14 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575421#comment-15575421 ] Harish commented on SPARK-16845: I have posted a scenario in stack overflow

[jira] [Comment Edited] (SPARK-17908) Column names Corrupted in pysaprk dataframe groupBy

2016-10-13 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572478#comment-15572478 ] Harish edited comment on SPARK-17908 at 10/13/16 4:58 PM: -- Yes. Your code

[jira] [Commented] (SPARK-17908) Column names Corrupted in pysaprk dataframe groupBy

2016-10-13 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572478#comment-15572478 ] Harish commented on SPARK-17908: Yes. You are code structure is same as mine.. But i have 70M records

[jira] [Comment Edited] (SPARK-17908) Column names Corrupted in pysaprk dataframe groupBy

2016-10-13 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572373#comment-15572373 ] Harish edited comment on SPARK-17908 at 10/13/16 4:17 PM: -- Sorry.. I didnt put

[jira] [Commented] (SPARK-17908) Column names Corrupted in pysaprk dataframe groupBy

2016-10-13 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572373#comment-15572373 ] Harish commented on SPARK-17908: Sorry.. I didnt put the actual column names of my code in stack trace, i

[jira] [Comment Edited] (SPARK-17908) Column names Corrupted in pysaprk dataframe groupBy

2016-10-13 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572278#comment-15572278 ] Harish edited comment on SPARK-17908 at 10/13/16 4:13 PM: -- Traceback (most

[jira] [Comment Edited] (SPARK-17908) Column names Corrupted in pysaprk dataframe groupBy

2016-10-13 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572278#comment-15572278 ] Harish edited comment on SPARK-17908 at 10/13/16 4:12 PM: -- Traceback (most

[jira] [Comment Edited] (SPARK-17908) Column names Corrupted in pysaprk dataframe groupBy

2016-10-13 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572278#comment-15572278 ] Harish edited comment on SPARK-17908 at 10/13/16 4:09 PM: -- Traceback (most

[jira] [Commented] (SPARK-17908) Column names Corrupted in pysaprk dataframe groupBy

2016-10-13 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572278#comment-15572278 ] Harish commented on SPARK-17908: Traceback (most recent call last): File

[jira] [Created] (SPARK-17908) Column names Corrupted in pysaprk dataframe groupBy

2016-10-13 Thread Harish (JIRA)
Harish created SPARK-17908: -- Summary: Column names Corrupted in pysaprk dataframe groupBy Key: SPARK-17908 URL: https://issues.apache.org/jira/browse/SPARK-17908 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-17901) NettyRpcEndpointRef: Error sending message and Caused by: java.util.ConcurrentModificationException

2016-10-12 Thread Harish (JIRA)
Harish created SPARK-17901: -- Summary: NettyRpcEndpointRef: Error sending message and Caused by: java.util.ConcurrentModificationException Key: SPARK-17901 URL: https://issues.apache.org/jira/browse/SPARK-17901

[jira] [Comment Edited] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-10-11 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566515#comment-15566515 ] Harish edited comment on SPARK-17463 at 10/11/16 8:34 PM: -- My second approach

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-10-11 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566515#comment-15566515 ] Harish commented on SPARK-17463: My second approach was: def testfunc(keys, vals, columnsToStandardize):

[jira] [Comment Edited] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-10-11 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566427#comment-15566427 ] Harish edited comment on SPARK-17463 at 10/11/16 8:06 PM: -- No i dont have any

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-10-11 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566427#comment-15566427 ] Harish commented on SPARK-17463: No i dont have any code like that. I use pyspark .. Please find my code

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-10-11 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566413#comment-15566413 ] Harish commented on SPARK-17463: Ok. thanks for the update. Do we have any work around for the second

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-10-11 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566206#comment-15566206 ] Harish commented on SPARK-17463: Is this fix is part of the https://github.com/apache/spark/pull/15371

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-10-11 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15565493#comment-15565493 ] Harish commented on SPARK-17463: It looks like a show stopper for my current project. Can you please let

[jira] [Comment Edited] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-10-11 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15565493#comment-15565493 ] Harish edited comment on SPARK-17463 at 10/11/16 2:03 PM: -- It looks like a show