[jira] [Commented] (SPARK-17458) Alias specified for aggregates in a pivot are not honored

2016-09-15 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494591#comment-15494591 ] Andrew Ray commented on SPARK-17458: [~hvanhovell]: My JIRA username is a1ray. > Alias specified for

[jira] [Issue Comment Deleted] (SPARK-17458) Alias specified for aggregates in a pivot are not honored

2016-09-15 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ray updated SPARK-17458: --- Comment: was deleted (was: [~hvanhovell] It's a1ray) > Alias specified for aggregates in a pivot

[jira] [Comment Edited] (SPARK-17458) Alias specified for aggregates in a pivot are not honored

2016-09-15 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494361#comment-15494361 ] Andrew Ray edited comment on SPARK-17458 at 9/15/16 8:09 PM: - [~hvanhovell]

[jira] [Commented] (SPARK-17458) Alias specified for aggregates in a pivot are not honored

2016-09-15 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494361#comment-15494361 ] Andrew Ray commented on SPARK-17458: It's a1ray > Alias specified for aggregates in a pivot are not

[jira] [Commented] (SPARK-15917) Define the number of executors in standalone mode with an easy-to-use property

2016-09-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490663#comment-15490663 ] Andrew Or commented on SPARK-15917: --- By the way is there a pull request? > Define the number of

[jira] [Commented] (SPARK-17310) Disable Parquet's record-by-record filter in normal parquet reader and do it in Spark-side

2016-08-30 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448672#comment-15448672 ] Andrew Duffy commented on SPARK-17310: -- +1 to this, see comments on

[jira] [Commented] (SPARK-17227) Allow configuring record delimiter in csv

2016-08-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435539#comment-15435539 ] Andrew Ash commented on SPARK-17227: Rob and I work together, and we've seen datasets in mostly-CSV

[jira] [Updated] (SPARK-17213) Parquet String Pushdown for Non-Eq Comparisons Broken

2016-08-24 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-17213: - Description: Spark defines ordering over strings based on comparison of UTF8 byte arrays, which

[jira] [Updated] (SPARK-17213) Parquet String Pushdown for Non-Eq Comparisons Broken

2016-08-24 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-17213: - Description: Spark defines ordering over strings based on comparison of UTF8 byte arrays, which

[jira] [Created] (SPARK-17213) Parquet String Pushdown for Non-Eq Comparisons Broken

2016-08-24 Thread Andrew Duffy (JIRA)
Andrew Duffy created SPARK-17213: Summary: Parquet String Pushdown for Non-Eq Comparisons Broken Key: SPARK-17213 URL: https://issues.apache.org/jira/browse/SPARK-17213 Project: Spark Issue

[jira] [Commented] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431409#comment-15431409 ] Andrew Davidson commented on SPARK-17172: - Hi Sean I forgot about that older jira issue. I never

[jira] [Commented] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431371#comment-15431371 ] Andrew Davidson commented on SPARK-17172: - Hi Sean the data center was created using spark-ec2

[jira] [Commented] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431018#comment-15431018 ] Andrew Davidson commented on SPARK-17172: - Hi Sean It should be very easy to use the attached

[jira] [Commented] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-21 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430004#comment-15430004 ] Andrew Davidson commented on SPARK-17172: - Hi Sean I do not think it is the same error. In the

[jira] [Updated] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-20 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Davidson updated SPARK-17172: Attachment: hiveUDFBug.ipynb hiveUDFBug.html > pyspak hiveContext can not

[jira] [Commented] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-20 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429465#comment-15429465 ] Andrew Davidson commented on SPARK-17172: - attached a notebook that demonstrates the bug. Also

[jira] [Commented] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-20 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429463#comment-15429463 ] Andrew Davidson commented on SPARK-17172: - related bug report :

[jira] [Created] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-20 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-17172: --- Summary: pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext. Key: SPARK-17172 URL:

[jira] [Commented] (SPARK-17143) pyspark unable to create UDF: java.lang.RuntimeException: org.apache.hadoop.fs.FileAlreadyExistsException: Parent path is not a directory: /tmp tmp

2016-08-18 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427394#comment-15427394 ] Andrew Davidson commented on SPARK-17143: - See email from user's group. I was able to find a work

[jira] [Commented] (SPARK-17143) pyspark unable to create UDF: java.lang.RuntimeException: org.apache.hadoop.fs.FileAlreadyExistsException: Parent path is not a directory: /tmp tmp

2016-08-18 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427278#comment-15427278 ] Andrew Davidson commented on SPARK-17143: - given the exception metioned an issue with /tmp I

[jira] [Updated] (SPARK-17143) pyspark unable to create UDF: java.lang.RuntimeException: org.apache.hadoop.fs.FileAlreadyExistsException: Parent path is not a directory: /tmp tmp

2016-08-18 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Davidson updated SPARK-17143: Attachment: udfBug.html This html version of the notebook shows the output when run in my

[jira] [Updated] (SPARK-17143) pyspark unable to create UDF: java.lang.RuntimeException: org.apache.hadoop.fs.FileAlreadyExistsException: Parent path is not a directory: /tmp tmp

2016-08-18 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Davidson updated SPARK-17143: Attachment: udfBug.ipynb The attached notebook demonstrated the reported bug. Note it

[jira] [Created] (SPARK-17143) pyspark unable to create UDF: java.lang.RuntimeException: org.apache.hadoop.fs.FileAlreadyExistsException: Parent path is not a directory: /tmp tmp

2016-08-18 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-17143: --- Summary: pyspark unable to create UDF: java.lang.RuntimeException: org.apache.hadoop.fs.FileAlreadyExistsException: Parent path is not a directory: /tmp tmp Key: SPARK-17143

[jira] [Created] (SPARK-17091) ParquetFilters rewrite IN to OR of Eq

2016-08-16 Thread Andrew Duffy (JIRA)
Andrew Duffy created SPARK-17091: Summary: ParquetFilters rewrite IN to OR of Eq Key: SPARK-17091 URL: https://issues.apache.org/jira/browse/SPARK-17091 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-17059) Allow FileFormat to specify partition pruning strategy

2016-08-15 Thread Andrew Duffy (JIRA)
Andrew Duffy created SPARK-17059: Summary: Allow FileFormat to specify partition pruning strategy Key: SPARK-17059 URL: https://issues.apache.org/jira/browse/SPARK-17059 Project: Spark Issue

[jira] [Commented] (SPARK-17029) Dataset toJSON goes through RDD form instead of transforming dataset itself

2016-08-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418149#comment-15418149 ] Andrew Ash commented on SPARK-17029: Note RDD form usage from

[jira] [Closed] (SPARK-16780) spark-streaming-kafka_2.10 version 2.0.0 not on maven central

2016-07-29 Thread Andrew B (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew B closed SPARK-16780. > spark-streaming-kafka_2.10 version 2.0.0 not on maven central >

[jira] [Commented] (SPARK-16780) spark-streaming-kafka_2.10 version 2.0.0 not on maven central

2016-07-28 Thread Andrew B (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398105#comment-15398105 ] Andrew B commented on SPARK-16780: -- How are the new artifacts used with the example below?

[jira] [Created] (SPARK-16780) spark-streaming-kafka_2.10 version 2.0.0 not on maven central

2016-07-28 Thread Andrew B (JIRA)
Andrew B created SPARK-16780: Summary: spark-streaming-kafka_2.10 version 2.0.0 not on maven central Key: SPARK-16780 URL: https://issues.apache.org/jira/browse/SPARK-16780 Project: Spark Issue

[jira] [Commented] (SPARK-16665) python import pyspark fails in context.py

2016-07-22 Thread Andrew Jefferson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389135#comment-15389135 ] Andrew Jefferson commented on SPARK-16665: -- This was the result of a previous failed import in

[jira] [Resolved] (SPARK-16665) python import pyspark fails in context.py

2016-07-22 Thread Andrew Jefferson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Jefferson resolved SPARK-16665. -- Resolution: Cannot Reproduce > python import pyspark fails in context.py >

[jira] [Commented] (SPARK-16665) python import pyspark fails in context.py

2016-07-21 Thread Andrew Jefferson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387908#comment-15387908 ] Andrew Jefferson commented on SPARK-16665: -- Pull Request here:

[jira] [Created] (SPARK-16665) python import pyspark fails in context.py

2016-07-21 Thread Andrew Jefferson (JIRA)
Andrew Jefferson created SPARK-16665: Summary: python import pyspark fails in context.py Key: SPARK-16665 URL: https://issues.apache.org/jira/browse/SPARK-16665 Project: Spark Issue

[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN

2016-07-07 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365998#comment-15365998 ] Andrew Duffy commented on SPARK-16265: -- Hi Sean, yeah I can see where you're coming from, but I feel

[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode

2016-07-01 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359277#comment-15359277 ] Andrew Davidson commented on SPARK-15829: - Hi Sean you mention the ec2 script is not supported

[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode

2016-07-01 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359215#comment-15359215 ] Andrew Davidson commented on SPARK-15829: - Hi Sean I am not sure how to check the value of

[jira] [Created] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN

2016-06-28 Thread Andrew Duffy (JIRA)
Andrew Duffy created SPARK-16265: Summary: Add option to SparkSubmit to ship driver JRE to YARN Key: SPARK-16265 URL: https://issues.apache.org/jira/browse/SPARK-16265 Project: Spark Issue

[jira] [Created] (SPARK-16196) Optimize in-memory scan performance using ColumnarBatches

2016-06-24 Thread Andrew Or (JIRA)
Andrew Or created SPARK-16196: - Summary: Optimize in-memory scan performance using ColumnarBatches Key: SPARK-16196 URL: https://issues.apache.org/jira/browse/SPARK-16196 Project: Spark Issue

[jira] [Commented] (SPARK-15917) Define the number of executors in standalone mode with an easy-to-use property

2016-06-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342376#comment-15342376 ] Andrew Or commented on SPARK-15917: --- (1) Yes, right now `spark.executor.instances` doesn't do anything

[jira] [Commented] (SPARK-15917) Define the number of executors in standalone mode with an easy-to-use property

2016-06-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342310#comment-15342310 ] Andrew Or commented on SPARK-15917: --- Yeah, I agree. We need to deal with conflicting options, however,

[jira] [Created] (SPARK-16023) Move InMemoryRelation to its own file

2016-06-17 Thread Andrew Or (JIRA)
Andrew Or created SPARK-16023: - Summary: Move InMemoryRelation to its own file Key: SPARK-16023 URL: https://issues.apache.org/jira/browse/SPARK-16023 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-15749) Make the error message more meaningful

2016-06-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15749. --- Resolution: Fixed Assignee: Huaxin Gao Fix Version/s: 2.0.0 Target

[jira] [Resolved] (SPARK-15868) Executors table in Executors tab should sort Executor IDs in numerical order (not alphabetical order)

2016-06-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15868. --- Resolution: Fixed Assignee: Alex Bozarth Fix Version/s: 2.0.0 Target

[jira] [Resolved] (SPARK-15998) Verification of SQLConf HIVE_METASTORE_PARTITION_PRUNING

2016-06-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15998. --- Resolution: Fixed Fix Version/s: 2.0.0 > Verification of SQLConf

[jira] [Updated] (SPARK-15998) Verification of SQLConf HIVE_METASTORE_PARTITION_PRUNING

2016-06-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15998: -- Assignee: Xiao Li > Verification of SQLConf HIVE_METASTORE_PARTITION_PRUNING >

[jira] [Resolved] (SPARK-15975) Improper Popen.wait() return code handling in dev/run-tests

2016-06-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15975. --- Resolution: Fixed Fix Version/s: 2.0.0 1.6.2 1.5.3 >

[jira] [Resolved] (SPARK-15978) Some improvement of "Show Tables"

2016-06-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15978. --- Resolution: Fixed Assignee: Bo Meng (was: Apache Spark) Fix Version/s:

[jira] [Updated] (SPARK-15576) Add back hive tests blacklisted by SPARK-15539

2016-06-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15576: -- Target Version/s: 2.1.0 (was: 2.0.0) > Add back hive tests blacklisted by SPARK-15539 >

[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode

2016-06-10 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325001#comment-15325001 ] Andrew Davidson commented on SPARK-15829: - Hi Xin I ran netstat on my master. I do not think the

[jira] [Commented] (SPARK-15867) TABLESAMPLE BUCKET semantics don't match Hive's

2016-06-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324960#comment-15324960 ] Andrew Or commented on SPARK-15867: --- I think we should fix it, though looks like it's been an issue for

[jira] [Updated] (SPARK-15867) TABLESAMPLE BUCKET semantics don't match Hive's

2016-06-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15867: -- Affects Version/s: 1.6.0 > TABLESAMPLE BUCKET semantics don't match Hive's >

[jira] [Created] (SPARK-15867) TABLESAMPLE BUCKET semantics don't match Hive's

2016-06-10 Thread Andrew Or (JIRA)
Andrew Or created SPARK-15867: - Summary: TABLESAMPLE BUCKET semantics don't match Hive's Key: SPARK-15867 URL: https://issues.apache.org/jira/browse/SPARK-15867 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode

2016-06-08 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-15829: --- Summary: spark master webpage links to application UI broke when running in cluster mode Key: SPARK-15829 URL: https://issues.apache.org/jira/browse/SPARK-15829

[jira] [Issue Comment Deleted] (SPARK-15722) Wrong data when CTAS specifies schema

2016-06-03 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15722: -- Comment: was deleted (was: User 'andrewor14' has created a pull request for this issue:

[jira] [Resolved] (SPARK-15736) Gracefully handle loss of DiskStore files

2016-06-02 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15736. --- Resolution: Fixed Fix Version/s: 2.0.0 1.6.2 > Gracefully handle loss of

[jira] [Resolved] (SPARK-15718) better error message for writing bucketing data

2016-06-02 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15718. --- Resolution: Fixed Fix Version/s: 2.0.0 > better error message for writing bucketing data >

[jira] [Issue Comment Deleted] (SPARK-15722) Wrong data when CTAS specifies schema

2016-06-02 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15722: -- Comment: was deleted (was: User 'andrewor14' has created a pull request for this issue:

[jira] [Resolved] (SPARK-15711) Ban CREATE TEMP TABLE USING AS SELECT for now

2016-06-02 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15711. --- Resolution: Fixed Fix Version/s: 2.0.0 > Ban CREATE TEMP TABLE USING AS SELECT for now >

[jira] [Updated] (SPARK-15646) When spark.sql.hive.convertCTAS is true, we may still convert the table to a parquet table when TEXTFILE or SEQUENCEFILE is specified.

2016-06-01 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15646: -- Assignee: Yin Huai > When spark.sql.hive.convertCTAS is true, we may still convert the table to a >

[jira] [Resolved] (SPARK-15646) When spark.sql.hive.convertCTAS is true, we may still convert the table to a parquet table when TEXTFILE or SEQUENCEFILE is specified.

2016-06-01 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15646. --- Resolution: Fixed Fix Version/s: 2.0.0 > When spark.sql.hive.convertCTAS is true, we may

[jira] [Created] (SPARK-15722) Wrong data when CTAS specifies schema

2016-06-01 Thread Andrew Or (JIRA)
Andrew Or created SPARK-15722: - Summary: Wrong data when CTAS specifies schema Key: SPARK-15722 URL: https://issues.apache.org/jira/browse/SPARK-15722 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-15715) Altering partition storage information doesn't work in Hive

2016-06-01 Thread Andrew Or (JIRA)
Andrew Or created SPARK-15715: - Summary: Altering partition storage information doesn't work in Hive Key: SPARK-15715 URL: https://issues.apache.org/jira/browse/SPARK-15715 Project: Spark Issue

[jira] [Updated] (SPARK-15711) Ban CREATE TEMP TABLE USING AS SELECT for now

2016-06-01 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15711: -- Assignee: Sean Zhong > Ban CREATE TEMP TABLE USING AS SELECT for now >

[jira] [Resolved] (SPARK-15236) No way to disable Hive support in REPL

2016-05-31 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15236. --- Resolution: Fixed Fix Version/s: 2.0.0 > No way to disable Hive support in REPL >

[jira] [Resolved] (SPARK-15618) Use SparkSession.builder.sparkContext(...) in tests where possible

2016-05-31 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15618. --- Resolution: Fixed Fix Version/s: 2.0.0 > Use SparkSession.builder.sparkContext(...) in tests

[jira] [Updated] (SPARK-15236) No way to disable Hive support in REPL

2016-05-31 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15236: -- Assignee: Xin Wu > No way to disable Hive support in REPL > -- > >

[jira] [Updated] (SPARK-15670) Add deprecate annotation for acumulator V1 interface in JavaSparkContext class

2016-05-31 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15670: -- Assignee: Weichen Xu > Add deprecate annotation for acumulator V1 interface in JavaSparkContext class

[jira] [Resolved] (SPARK-15670) Add deprecate annotation for acumulator V1 interface in JavaSparkContext class

2016-05-31 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15670. --- Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: 2.0.0 > Add deprecate

[jira] [Resolved] (SPARK-15662) Add since annotation for classes in sql.catalog

2016-05-31 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15662. --- Resolution: Fixed Fix Version/s: 2.0.0 > Add since annotation for classes in sql.catalog >

[jira] [Created] (SPARK-15635) ALTER TABLE RENAME doesn't work for datasource tables

2016-05-27 Thread Andrew Or (JIRA)
Andrew Or created SPARK-15635: - Summary: ALTER TABLE RENAME doesn't work for datasource tables Key: SPARK-15635 URL: https://issues.apache.org/jira/browse/SPARK-15635 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-15450) Clean up SparkSession builder for python

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15450. --- Resolution: Fixed Fix Version/s: 2.0.0 > Clean up SparkSession builder for python >

[jira] [Resolved] (SPARK-15534) TRUNCATE TABLE should throw exceptions, not logError

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15534. --- Resolution: Fixed Fix Version/s: 2.0.0 > TRUNCATE TABLE should throw exceptions, not logError

[jira] [Resolved] (SPARK-15535) Remove code for TRUNCATE TABLE ... COLUMN

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15535. --- Resolution: Fixed Fix Version/s: 2.0.0 > Remove code for TRUNCATE TABLE ... COLUMN >

[jira] [Updated] (SPARK-15450) Clean up SparkSession builder for python

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15450: -- Assignee: Eric Liang (was: Andrew Or) > Clean up SparkSession builder for python >

[jira] [Commented] (SPARK-15618) Use SparkSession.builder.sparkContext(...) in tests where possible

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304506#comment-15304506 ] Andrew Or commented on SPARK-15618: --- it needs to be internal. At least it should be private[spark] >

[jira] [Resolved] (SPARK-15569) Executors spending significant time in DiskObjectWriter.updateBytesWritten function

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15569. --- Resolution: Fixed Assignee: Sital Kedia Fix Version/s: 2.0.0 Target

[jira] [Updated] (SPARK-15599) Document createDataset functions in SparkSession

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15599: -- Affects Version/s: 2.0.0 Target Version/s: 2.0.0 Component/s: Documentation > Document

[jira] [Resolved] (SPARK-15599) Document createDataset functions in SparkSession

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15599. --- Resolution: Fixed Fix Version/s: 2.0.0 > Document createDataset functions in SparkSession >

[jira] [Updated] (SPARK-15599) Document createDataset functions in SparkSession

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15599: -- Assignee: Sameer Agarwal > Document createDataset functions in SparkSession >

[jira] [Resolved] (SPARK-15584) Abstract duplicate code: "spark.sql.sources." properties

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15584. --- Resolution: Fixed Fix Version/s: 2.0.0 > Abstract duplicate code: "spark.sql.sources."

[jira] [Updated] (SPARK-15603) Replace SQLContext with SparkSession in ML/MLLib

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15603: -- Fix Version/s: 2.0.0 > Replace SQLContext with SparkSession in ML/MLLib >

[jira] [Updated] (SPARK-15603) Replace SQLContext with SparkSession in ML/MLLib

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15603: -- Assignee: Dongjoon Hyun > Replace SQLContext with SparkSession in ML/MLLib >

[jira] [Updated] (SPARK-15618) Use SparkSession.builder.sparkContext(...) in tests where possible

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15618: -- Priority: Minor (was: Major) > Use SparkSession.builder.sparkContext(...) in tests where possible >

[jira] [Updated] (SPARK-15603) Replace SQLContext with SparkSession in ML/MLLib

2016-05-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15603: -- Affects Version/s: 2.0.0 > Replace SQLContext with SparkSession in ML/MLLib >

[jira] [Created] (SPARK-15618) Use SparkSession.builder.sparkContext(...) in tests where possible

2016-05-27 Thread Andrew Or (JIRA)
Andrew Or created SPARK-15618: - Summary: Use SparkSession.builder.sparkContext(...) in tests where possible Key: SPARK-15618 URL: https://issues.apache.org/jira/browse/SPARK-15618 Project: Spark

[jira] [Resolved] (SPARK-15536) Disallow TRUNCATE TABLE with external tables and views

2016-05-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15536. --- Resolution: Fixed Fix Version/s: 2.0.0 > Disallow TRUNCATE TABLE with external tables and

[jira] [Resolved] (SPARK-15538) Truncate table does not work on data source table

2016-05-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15538. --- Resolution: Fixed Fix Version/s: 2.0.0 > Truncate table does not work on data source table >

[jira] [Created] (SPARK-15596) ALTER TABLE RENAME needs to uncache query

2016-05-26 Thread Andrew Or (JIRA)
Andrew Or created SPARK-15596: - Summary: ALTER TABLE RENAME needs to uncache query Key: SPARK-15596 URL: https://issues.apache.org/jira/browse/SPARK-15596 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-15584) Abstract duplicate code: "spark.sql.sources." properties

2016-05-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15584: -- Assignee: Dongjoon Hyun > Abstract duplicate code: "spark.sql.sources." properties >

[jira] [Created] (SPARK-15594) ALTER TABLE ... SERDEPROPERTIES does not respect partition spec

2016-05-26 Thread Andrew Or (JIRA)
Andrew Or created SPARK-15594: - Summary: ALTER TABLE ... SERDEPROPERTIES does not respect partition spec Key: SPARK-15594 URL: https://issues.apache.org/jira/browse/SPARK-15594 Project: Spark

[jira] [Created] (SPARK-15584) Abstract duplicate code: "spark.sql.sources." properties

2016-05-26 Thread Andrew Or (JIRA)
Andrew Or created SPARK-15584: - Summary: Abstract duplicate code: "spark.sql.sources." properties Key: SPARK-15584 URL: https://issues.apache.org/jira/browse/SPARK-15584 Project: Spark Issue

[jira] [Updated] (SPARK-15584) Abstract duplicate code: "spark.sql.sources." properties

2016-05-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15584: -- Issue Type: Improvement (was: Bug) > Abstract duplicate code: "spark.sql.sources." properties >

[jira] [Commented] (SPARK-15584) Abstract duplicate code: "spark.sql.sources." properties

2016-05-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303054#comment-15303054 ] Andrew Or commented on SPARK-15584: --- [~dongjoon] would you like to work on this? > Abstract duplicate

[jira] [Updated] (SPARK-15584) Abstract duplicate code: "spark.sql.sources." properties

2016-05-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15584: -- Assignee: (was: Andrew Or) > Abstract duplicate code: "spark.sql.sources." properties >

[jira] [Updated] (SPARK-15584) Abstract duplicate code: "spark.sql.sources." properties

2016-05-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15584: -- Priority: Minor (was: Major) > Abstract duplicate code: "spark.sql.sources." properties >

[jira] [Created] (SPARK-15583) Relax ALTER TABLE properties restriction for data source tables

2016-05-26 Thread Andrew Or (JIRA)
Andrew Or created SPARK-15583: - Summary: Relax ALTER TABLE properties restriction for data source tables Key: SPARK-15583 URL: https://issues.apache.org/jira/browse/SPARK-15583 Project: Spark

[jira] [Commented] (SPARK-15579) SparkUI: Storage page is empty even if things are cached

2016-05-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302930#comment-15302930 ] Andrew Or commented on SPARK-15579: --- I tried this on 0f61d6efb45b9ee94fa663f67c4489fbdae2eded, which is

[jira] [Updated] (SPARK-15579) SparkUI: Storage page is empty even if things are cached

2016-05-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15579: -- Description: scala> sc.parallelize(1 to 1, 5000).cache().count() SparkUI storage page is empty.

[jira] [Created] (SPARK-15579) SparkUI: Storage page is empty even if things are cached

2016-05-26 Thread Andrew Or (JIRA)
Andrew Or created SPARK-15579: - Summary: SparkUI: Storage page is empty even if things are cached Key: SPARK-15579 URL: https://issues.apache.org/jira/browse/SPARK-15579 Project: Spark Issue

[jira] [Resolved] (SPARK-15552) Remove unnecessary private[sql] methods in SparkSession

2016-05-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15552. --- Resolution: Fixed Fix Version/s: 2.0.0 > Remove unnecessary private[sql] methods in

<    1   2   3   4   5   6   7   8   9   10   >