[jira] [Updated] (SPARK-34163) Spark Structured Streaming - Kafka avro transformation on optional field Failed

[ https://issues.apache.org/jira/browse/SPARK-34163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-34163: - Description: Hello All, I have a spark structured streaming job to inject data

[jira] [Updated] (SPARK-34163) Spark Structured Streaming - Kafka avro transformation on optional field Failed

[ https://issues.apache.org/jira/browse/SPARK-34163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-34163: - Description: Hello All, I have a spark structured streaming job to inject data

[jira] [Created] (SPARK-34163) Spark Structured Streaming - Kafka avro transformation on optional field Failed

Felix Kizhakkel Jose created SPARK-34163: Summary: Spark Structured Streaming - Kafka avro transformation on optional field Failed Key: SPARK-34163 URL: https://issues.apache.org/jira/browse/SPARK-34163

[jira] [Commented] (SPARK-32583) PySpark Structured Streaming Testing Support

[ https://issues.apache.org/jira/browse/SPARK-32583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175658#comment-17175658 ] Felix Kizhakkel Jose commented on SPARK-32583: -- [~hyukjin.kwon] I couldn't find any test

[jira] [Commented] (SPARK-32583) PySpark Structured Streaming Testing Support

[ https://issues.apache.org/jira/browse/SPARK-32583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175202#comment-17175202 ] Felix Kizhakkel Jose commented on SPARK-32583: -- [~rohitmishr1484] It would have been nice

[jira] [Created] (SPARK-32583) PySpark Structured Streaming Testing Support

Felix Kizhakkel Jose created SPARK-32583: Summary: PySpark Structured Streaming Testing Support Key: SPARK-32583 URL: https://issues.apache.org/jira/browse/SPARK-32583 Project: Spark

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162311#comment-17162311 ] Felix Kizhakkel Jose commented on SPARK-26345: -- [~sha...@uber.com] I don't have permission

[jira] [Updated] (SPARK-31763) DataFrame.inputFiles() not Available

[ https://issues.apache.org/jira/browse/SPARK-31763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-31763: - Issue Type: Bug (was: New Feature) > DataFrame.inputFiles() not Available >

[jira] [Created] (SPARK-31763) DataFrame.inputFiles() not Available

Felix Kizhakkel Jose created SPARK-31763: Summary: DataFrame.inputFiles() not Available Key: SPARK-31763 URL: https://issues.apache.org/jira/browse/SPARK-31763 Project: Spark Issue

[jira] [Commented] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095697#comment-17095697 ] Felix Kizhakkel Jose commented on SPARK-31599: -- Thank you [~gsomogyi]. But this is not a S3

[jira] [Commented] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094917#comment-17094917 ] Felix Kizhakkel Jose commented on SPARK-31599: -- How do I do that?  > Reading from S3

[jira] [Updated] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-31599: - Description: I have a S3 bucket which has data streamed (Parquet format) to it

[jira] [Created] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

Felix Kizhakkel Jose created SPARK-31599: Summary: Reading from S3 (Structured Streaming Bucket) Fails after Compaction Key: SPARK-31599 URL: https://issues.apache.org/jira/browse/SPARK-31599

[jira] [Commented] (SPARK-31072) Default to ParquetOutputCommitter even after configuring s3a committer as "partitioned"

[ https://issues.apache.org/jira/browse/SPARK-31072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071819#comment-17071819 ] Felix Kizhakkel Jose commented on SPARK-31072: -- Hello, Any updates or helps will be much

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070982#comment-17070982 ] Felix Kizhakkel Jose commented on SPARK-26345: -- I have created a Jira in Parquet-mr for

[jira] [Comment Edited] (SPARK-31162) Provide Configuration Parameter to select/enforce the Hive Hash for Bucketing

[ https://issues.apache.org/jira/browse/SPARK-31162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060408#comment-17060408 ] Felix Kizhakkel Jose edited comment on SPARK-31162 at 3/16/20, 6:13 PM:

[jira] [Commented] (SPARK-31162) Provide Configuration Parameter to select/enforce the Hive Hash for Bucketing

[ https://issues.apache.org/jira/browse/SPARK-31162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060408#comment-17060408 ] Felix Kizhakkel Jose commented on SPARK-31162: -- I have seen following in the API

[jira] [Commented] (SPARK-17495) Hive hash implementation

[ https://issues.apache.org/jira/browse/SPARK-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059927#comment-17059927 ] Felix Kizhakkel Jose commented on SPARK-17495: -- [~maropu] [~tejasp] I have created a Jira

[jira] [Commented] (SPARK-19256) Hive bucketing support

[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059925#comment-17059925 ] Felix Kizhakkel Jose commented on SPARK-19256: -- Expose a configuration parameter for Hive

[jira] [Created] (SPARK-31162) Provide Configuration Parameter to select/enforce the Hive Hash for Bucketing

Felix Kizhakkel Jose created SPARK-31162: Summary: Provide Configuration Parameter to select/enforce the Hive Hash for Bucketing Key: SPARK-31162 URL: https://issues.apache.org/jira/browse/SPARK-31162

[jira] [Commented] (SPARK-17495) Hive hash implementation

[ https://issues.apache.org/jira/browse/SPARK-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059893#comment-17059893 ] Felix Kizhakkel Jose commented on SPARK-17495: -- @[~maropu] [~hyukjin.kwon] So we need a new

[jira] [Commented] (SPARK-17495) Hive hash implementation

[ https://issues.apache.org/jira/browse/SPARK-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059847#comment-17059847 ] Felix Kizhakkel Jose commented on SPARK-17495: -- @[~maropu] Is there any configuration

[jira] [Commented] (SPARK-17495) Hive hash implementation

[ https://issues.apache.org/jira/browse/SPARK-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059846#comment-17059846 ] Felix Kizhakkel Jose commented on SPARK-17495: -- Oh, thats great. I will check. > Hive hash

[jira] [Commented] (SPARK-17495) Hive hash implementation

[ https://issues.apache.org/jira/browse/SPARK-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059681#comment-17059681 ] Felix Kizhakkel Jose commented on SPARK-17495: -- Is this going to available in the coming

[jira] [Commented] (SPARK-31072) Default to ParquetOutputCommitter even after configuring s3a committer as "partitioned"

[ https://issues.apache.org/jira/browse/SPARK-31072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055435#comment-17055435 ] Felix Kizhakkel Jose commented on SPARK-31072: -- Could you please provide some insights? >

[jira] [Commented] (SPARK-19256) Hive bucketing support

[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055007#comment-17055007 ] Felix Kizhakkel Jose commented on SPARK-19256: -- [~chengsu] Any updates on this feature

[jira] [Updated] (SPARK-31072) Default to ParquetOutputCommitter even after configuring s3a committer as "partitioned"

[ https://issues.apache.org/jira/browse/SPARK-31072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-31072: - Summary: Default to ParquetOutputCommitter even after configuring s3a committer

[jira] [Updated] (SPARK-31072) Default to ParquetOutputCommitter even after configuring committer as "partitioned"

[ https://issues.apache.org/jira/browse/SPARK-31072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-31072: - Summary: Default to ParquetOutputCommitter even after configuring committer as

[jira] [Comment Edited] (SPARK-31072) Default to ParquetOutputCommitter even after configuring setting committer as "partitioned"

[ https://issues.apache.org/jira/browse/SPARK-31072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053543#comment-17053543 ] Felix Kizhakkel Jose edited comment on SPARK-31072 at 3/6/20, 3:48 PM:

[jira] [Commented] (SPARK-31072) Default to ParquetOutputCommitter even after configuring setting committer as "partitioned"

[ https://issues.apache.org/jira/browse/SPARK-31072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053543#comment-17053543 ] Felix Kizhakkel Jose commented on SPARK-31072: -- [~steve_l],  I have seen some issues you

[jira] [Created] (SPARK-31072) Default to ParquetOutputCommitter even after configuring setting committer as "partitioned"

Felix Kizhakkel Jose created SPARK-31072: Summary: Default to ParquetOutputCommitter even after configuring setting committer as "partitioned" Key: SPARK-31072 URL:

[jira] [Commented] (SPARK-20901) Feature parity for ORC with Parquet

[ https://issues.apache.org/jira/browse/SPARK-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052476#comment-17052476 ] Felix Kizhakkel Jose commented on SPARK-20901: -- Thank you [~dongjoon].  > Feature parity

[jira] [Commented] (SPARK-20901) Feature parity for ORC with Parquet

[ https://issues.apache.org/jira/browse/SPARK-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052384#comment-17052384 ] Felix Kizhakkel Jose commented on SPARK-20901: -- Hi [~dongjoon], I was trying to choose

[jira] [Commented] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972605#comment-16972605 ] Felix Kizhakkel Jose commented on SPARK-29764: -- So it's still hard to follow the

[jira] [Commented] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970451#comment-16970451 ] Felix Kizhakkel Jose commented on SPARK-29764: -- How do I get a help once the priority is

[jira] [Commented] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969301#comment-16969301 ] Felix Kizhakkel Jose commented on SPARK-29764: -- [~hyukjin.kwon] Could you please help me

[jira] [Comment Edited] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968411#comment-16968411 ] Felix Kizhakkel Jose edited comment on SPARK-29764 at 11/6/19 2:46 PM:

[jira] [Comment Edited] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968411#comment-16968411 ] Felix Kizhakkel Jose edited comment on SPARK-29764 at 11/6/19 2:45 PM:

[jira] [Updated] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-29764: - Attachment: SparkParquetSampleCode.docx > Error on Serializing POJO with java

[jira] [Commented] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968411#comment-16968411 ] Felix Kizhakkel Jose commented on SPARK-29764: -- [~hyukjin.kwon] Sorry, I didn't know

[jira] [Commented] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967800#comment-16967800 ] Felix Kizhakkel Jose commented on SPARK-29764: -- The Spark Schema generated for the POJO is:

[jira] [Updated] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-29764: - Description: Hello, I have been doing a proof of concept for data lake

[jira] [Updated] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-29764: - Description: Hello, I have been doing a proof of concept for data lake

[jira] [Updated] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

[ https://issues.apache.org/jira/browse/SPARK-29764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-29764: - Description: Hello, I have been doing a proof of concept for data lake

[jira] [Created] (SPARK-29764) Error on Serializing POJO with java datetime property to a Parquet file

Felix Kizhakkel Jose created SPARK-29764: Summary: Error on Serializing POJO with java datetime property to a Parquet file Key: SPARK-29764 URL: https://issues.apache.org/jira/browse/SPARK-29764