[jira] [Work started] (BEAM-6480) Add AvroIO.sink for IndexedRecord (FileIO compatible)

[ https://issues.apache.org/jira/browse/BEAM-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-6480 started by Ryan Skraba. - > Add AvroIO.sink for IndexedRecord (FileIO compatible) >

[jira] [Commented] (BEAM-6480) Add AvroIO.sink for IndexedRecord (FileIO compatible)

[ https://issues.apache.org/jira/browse/BEAM-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879205#comment-16879205 ] Ryan Skraba commented on BEAM-6480: --- I wrote the sinkViaGeneric to target PCollection as requested here

[jira] [Commented] (BEAM-881) Provide a PTransform in IOs providing a "standard" Avro IndexedRecord

[ https://issues.apache.org/jira/browse/BEAM-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878502#comment-16878502 ] Ryan Skraba commented on BEAM-881: -- [~jbonofre] – Two years later, it seems obvious that 

[jira] [Assigned] (BEAM-4181) Add ReadFiles transform for TfRecordIO

[ https://issues.apache.org/jira/browse/BEAM-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba reassigned BEAM-4181: - Assignee: Ryan Skraba > Add ReadFiles transform for TfRecordIO >

[jira] [Commented] (BEAM-5164) ParquetIOIT fails on Spark and Flink

[ https://issues.apache.org/jira/browse/BEAM-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907232#comment-16907232 ] Ryan Skraba commented on BEAM-5164: --- OK -- one huge complication :/ Relocating the parquet library

[jira] [Assigned] (BEAM-7073) AvroUtils converting generic record to Beam Row causes class cast exception

[ https://issues.apache.org/jira/browse/BEAM-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba reassigned BEAM-7073: - Assignee: Ryan Skraba > AvroUtils converting generic record to Beam Row causes class cast

[jira] [Commented] (BEAM-5164) ParquetIOIT fails on Spark and Flink

[ https://issues.apache.org/jira/browse/BEAM-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905907#comment-16905907 ] Ryan Skraba commented on BEAM-5164: --- Thanks for the link for the context! Is it possible that

[jira] [Comment Edited] (BEAM-5164) ParquetIOIT fails on Spark and Flink

[ https://issues.apache.org/jira/browse/BEAM-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906020#comment-16906020 ] Ryan Skraba edited comment on BEAM-5164 at 8/13/19 9:50 AM: I checked with a

[jira] [Commented] (BEAM-5164) ParquetIOIT fails on Spark and Flink

[ https://issues.apache.org/jira/browse/BEAM-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906020#comment-16906020 ] Ryan Skraba commented on BEAM-5164: --- I checked with a spark local run on Spark 2.4.3, there is no issue

[jira] [Created] (BEAM-7979) Avro incompatibilities with Spark 2.2 and Spark 2.3

Ryan Skraba created BEAM-7979: - Summary: Avro incompatibilities with Spark 2.2 and Spark 2.3 Key: BEAM-7979 URL: https://issues.apache.org/jira/browse/BEAM-7979 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-5164) ParquetIOIT fails on Spark and Flink

[ https://issues.apache.org/jira/browse/BEAM-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906154#comment-16906154 ] Ryan Skraba commented on BEAM-5164: --- I am not confident on the overall strategy with respect to

[jira] [Commented] (BEAM-4379) Make ParquetIO Read splittable

[ https://issues.apache.org/jira/browse/BEAM-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899959#comment-16899959 ] Ryan Skraba commented on BEAM-4379: --- I took a deep look at the reader furnished by the Parquet community

[jira] [Commented] (BEAM-4379) Make ParquetIO Read splittable

[ https://issues.apache.org/jira/browse/BEAM-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902108#comment-16902108 ] Ryan Skraba commented on BEAM-4379: --- I'm looking at what Spark has done for splittable Parquet files --

[jira] [Commented] (BEAM-4379) Make ParquetIO Read splittable

[ https://issues.apache.org/jira/browse/BEAM-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902224#comment-16902224 ] Ryan Skraba commented on BEAM-4379: --- Alright, I was mistaken about one thing -- the current ParquetIO

[jira] [Commented] (BEAM-7829) AvroUtils.toAvroSchema should put a Schema name to pass Avro Schema validation

[ https://issues.apache.org/jira/browse/BEAM-7829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900175#comment-16900175 ] Ryan Skraba commented on BEAM-7829: --- The spark test suite for generating names and namespaces is here:

[jira] [Commented] (BEAM-5164) ParquetIOIT fails on Spark and Flink

[ https://issues.apache.org/jira/browse/BEAM-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903656#comment-16903656 ] Ryan Skraba commented on BEAM-5164: --- See question and possible workarounds on [stack

[jira] [Commented] (BEAM-6883) StreamingSourceMetricsTest takes too long to finish

[ https://issues.apache.org/jira/browse/BEAM-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16892900#comment-16892900 ] Ryan Skraba commented on BEAM-6883: --- More bad news -- it looks like the StreamingSourceMetricsTests is

[jira] [Assigned] (BEAM-6883) StreamingSourceMetricsTest takes too long to finish

[ https://issues.apache.org/jira/browse/BEAM-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba reassigned BEAM-6883: - Assignee: Ryan Skraba (was: Alexey Romanenko) > StreamingSourceMetricsTest takes too long to

[jira] [Assigned] (BEAM-7698) Elasticsearch never has a capital S in mixed case.

[ https://issues.apache.org/jira/browse/BEAM-7698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba reassigned BEAM-7698: - Assignee: Ryan Skraba > Elasticsearch never has a capital S in mixed case. >

[jira] [Created] (BEAM-7698) Elasticsearch never has a capital S in mixed case.

Ryan Skraba created BEAM-7698: - Summary: Elasticsearch never has a capital S in mixed case. Key: BEAM-7698 URL: https://issues.apache.org/jira/browse/BEAM-7698 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-7698) Elasticsearch never has a capital S in mixed case.

[ https://issues.apache.org/jira/browse/BEAM-7698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba updated BEAM-7698: -- Status: Open (was: Triage Needed) > Elasticsearch never has a capital S in mixed case. >

[jira] [Resolved] (BEAM-7698) Elasticsearch never has a capital S in mixed case.

[ https://issues.apache.org/jira/browse/BEAM-7698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba resolved BEAM-7698. --- Resolution: Fixed Fix Version/s: 2.15.0 > Elasticsearch never has a capital S in mixed case. >

[jira] [Commented] (BEAM-8564) Add LZO compression and decompression support

[ https://issues.apache.org/jira/browse/BEAM-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969438#comment-16969438 ] Ryan Skraba commented on BEAM-8564: --- I'm not sure.  I know there is an Apache licensed LZO

[jira] [Commented] (BEAM-8564) Add LZO compression and decompression support

[ https://issues.apache.org/jira/browse/BEAM-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969248#comment-16969248 ] Ryan Skraba commented on BEAM-8564: --- Just for info: historically, there has been issues including LZO

[jira] [Commented] (BEAM-6496) No coverage reported for sdks/java/extensions/sql, due to class does not match errors

[ https://issues.apache.org/jira/browse/BEAM-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949506#comment-16949506 ] Ryan Skraba commented on BEAM-6496: --- Hello! I just gave this a try, and can't reproduce today

[jira] [Commented] (BEAM-7073) AvroUtils converting generic record to Beam Row causes class cast exception

[ https://issues.apache.org/jira/browse/BEAM-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949523#comment-16949523 ] Ryan Skraba commented on BEAM-7073: --- I agree -- I noted that the PR above doesn't have a unit test, and

[jira] [Closed] (BEAM-6496) No coverage reported for sdks/java/extensions/sql, due to class does not match errors

[ https://issues.apache.org/jira/browse/BEAM-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba closed BEAM-6496. - Fix Version/s: Not applicable Resolution: Cannot Reproduce > No coverage reported for

[jira] [Commented] (BEAM-8384) Spark runner is not respecting spark.default.parallelism user defined configuration

[ https://issues.apache.org/jira/browse/BEAM-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949333#comment-16949333 ] Ryan Skraba commented on BEAM-8384: --- Related to BEAM-8191 (exploding number of partitions during

[jira] [Commented] (BEAM-9361) NPE When putting Avro record with enum through SqlTransform

[ https://issues.apache.org/jira/browse/BEAM-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17047461#comment-17047461 ] Ryan Skraba commented on BEAM-9361: --- Interesting! It would be great, of course, if every Avro record was

[jira] [Commented] (BEAM-4379) Make ParquetIO Read splittable

[ https://issues.apache.org/jira/browse/BEAM-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029712#comment-17029712 ] Ryan Skraba commented on BEAM-4379: --- Hello! No progress to report -- feel free to take this if you

[jira] [Assigned] (BEAM-4379) Make ParquetIO Read splittable

[ https://issues.apache.org/jira/browse/BEAM-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba reassigned BEAM-4379: - Assignee: (was: Ryan Skraba) > Make ParquetIO Read splittable >

[jira] [Commented] (BEAM-9315) HadoopFileSystemOptions unable to interpret HADOOP_CONF_DIR with multiple paths

[ https://issues.apache.org/jira/browse/BEAM-9315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037002#comment-17037002 ] Ryan Skraba commented on BEAM-9315: --- Hello! Thanks for taking a look at this -- I'm confused about