[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv
[
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305996#comment-14305996
]
Adnan Khan commented on FLINK-1388:
---
I'll work on this one.
POJO support for writeAsCsv
---
Key: FLINK-1388
URL: https://issues.apache.org/jira/browse/FLINK-1388
Project: Flink
Issue Type: New Feature
Components: Java API
Reporter: Timo Walther
Priority: Minor
It would be great if one could simply write out POJOs in CSV format.
{code}
public class MyPojo {
String a;
int b;
}
{code}
to:
{code}
# CSV file of org.apache.flink.MyPojo: String a, int b
Hello World, 42
Hello World 2, 47
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv
[
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311329#comment-14311329
]
Adnan Khan commented on FLINK-1388:
---
Hey Fabian,
I've been digging through those two classes you mentioned, but I'm still not
clear on how to use the {{PojoTypeInfo}}. So given a custom POJO, how to create
a {{PojoTypeInfo}} instance. Specifically because it takes in a
{{ListPojoField}} as a constructor parameter as seen here {{public
PojoTypeInfo(ClassT typeClass, ListPojoField fields)}}
However looking at the example output in the ticket description above, I can
generate a CSV string using Java's reflect library. I feel I've missed
something.
POJO support for writeAsCsv
---
Key: FLINK-1388
URL: https://issues.apache.org/jira/browse/FLINK-1388
Project: Flink
Issue Type: New Feature
Components: Java API
Reporter: Timo Walther
Assignee: Adnan Khan
Priority: Minor
It would be great if one could simply write out POJOs in CSV format.
{code}
public class MyPojo {
String a;
int b;
}
{code}
to:
{code}
# CSV file of org.apache.flink.MyPojo: String a, int b
Hello World, 42
Hello World 2, 47
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (FLINK-1388) POJO support for writeAsCsv
[
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14324987#comment-14324987
]
Adnan Khan edited comment on FLINK-1388 at 2/17/15 10:11 PM:
-
Oh that makes sense, I'll look at that. I don't see any tests for
{{org.apache.flink.api.java.io.CsvOutputFormat}}. Should I add some similar to
{{org.apache.flink.api.java.io.record.CsvOutputFormatTest}}? But for tuples and
POJOs of course.
was (Author: khnd):
Oh that makes sense, I'll look at that. I don't see any tests for
{{org.apache.flink.api.java.io.CsvOutputFormat}}. Should I add some similar to
{org.apache.flink.api.java.io.record.CsvOutputFormatTest}}? But for tuples and
POJOs of course.
POJO support for writeAsCsv
---
Key: FLINK-1388
URL: https://issues.apache.org/jira/browse/FLINK-1388
Project: Flink
Issue Type: New Feature
Components: Java API
Reporter: Timo Walther
Assignee: Adnan Khan
Priority: Minor
It would be great if one could simply write out POJOs in CSV format.
{code}
public class MyPojo {
String a;
int b;
}
{code}
to:
{code}
# CSV file of org.apache.flink.MyPojo: String a, int b
Hello World, 42
Hello World 2, 47
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv
[
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325020#comment-14325020
]
Adnan Khan commented on FLINK-1388:
---
Hey thanks.
* What exactly do you mean by integration with {{DataSet.internalWriteAsCsv}}?
* When you say TypeExtraction on every tuple - Do you mean the specific calls
to {{TypeExtractor.createTypeInfo()}} on every tuple?
* That makes sense. I'm going to go with Timo's suggestion and try to modify
{{org.apache.flink.api.java.io.CsvOutputFormat}} which extends
{{FileOutputFormat}}
POJO support for writeAsCsv
---
Key: FLINK-1388
URL: https://issues.apache.org/jira/browse/FLINK-1388
Project: Flink
Issue Type: New Feature
Components: Java API
Reporter: Timo Walther
Assignee: Adnan Khan
Priority: Minor
It would be great if one could simply write out POJOs in CSV format.
{code}
public class MyPojo {
String a;
int b;
}
{code}
to:
{code}
# CSV file of org.apache.flink.MyPojo: String a, int b
Hello World, 42
Hello World 2, 47
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv
[
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321633#comment-14321633
]
Adnan Khan commented on FLINK-1388:
---
Hey Timo,
So the rules for defining a POJO according to
[this|https://github.com/apache/flink/blob/master/docs/internal_types_serialization.md]
include this part {quote} All fields in the class (and all superclasses) are
either public or or have a public getter and a setter method that follows the
Java beans naming conventions for getters and setters. {quote}
So that means that it could have private fields. I was thinking we should add
{{field.setAccessible(true)}} in the implementation for {{PojoField}}.
Otherwise something like {{pFieldValue = pField.field.get(myPojo)}} does not
seem to work.
I ran into while I was testing the CSV writer with POJOs with private fields
but public getter/setters.
POJO support for writeAsCsv
---
Key: FLINK-1388
URL: https://issues.apache.org/jira/browse/FLINK-1388
Project: Flink
Issue Type: New Feature
Components: Java API
Reporter: Timo Walther
Assignee: Adnan Khan
Priority: Minor
It would be great if one could simply write out POJOs in CSV format.
{code}
public class MyPojo {
String a;
int b;
}
{code}
to:
{code}
# CSV file of org.apache.flink.MyPojo: String a, int b
Hello World, 42
Hello World 2, 47
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv
[
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321642#comment-14321642
]
Adnan Khan commented on FLINK-1388:
---
Okay thanks, that sounds good.
Also - Where should I put the csv writer code? Right now I have it in
{{org.apache.flink.api.java.typeutils.runtime}}
POJO support for writeAsCsv
---
Key: FLINK-1388
URL: https://issues.apache.org/jira/browse/FLINK-1388
Project: Flink
Issue Type: New Feature
Components: Java API
Reporter: Timo Walther
Assignee: Adnan Khan
Priority: Minor
It would be great if one could simply write out POJOs in CSV format.
{code}
public class MyPojo {
String a;
int b;
}
{code}
to:
{code}
# CSV file of org.apache.flink.MyPojo: String a, int b
Hello World, 42
Hello World 2, 47
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv
[
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321756#comment-14321756
]
Adnan Khan commented on FLINK-1388:
---
Heya,
Can I get some feedback on the CSV writer code in my working branch? Here's the
latest [commit|
https://github.com/khnd/flink/commit/4934f1843cdae1080afb24e8d22a405c70faf13d].
Thanks.
POJO support for writeAsCsv
---
Key: FLINK-1388
URL: https://issues.apache.org/jira/browse/FLINK-1388
Project: Flink
Issue Type: New Feature
Components: Java API
Reporter: Timo Walther
Assignee: Adnan Khan
Priority: Minor
It would be great if one could simply write out POJOs in CSV format.
{code}
public class MyPojo {
String a;
int b;
}
{code}
to:
{code}
# CSV file of org.apache.flink.MyPojo: String a, int b
Hello World, 42
Hello World 2, 47
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv
[
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357418#comment-14357418
]
Adnan Khan commented on FLINK-1388:
---
so i understand - why does it matter if TypeInformation is serializable or not?
aren't we just using it access the fields we're going to write out?
POJO support for writeAsCsv
---
Key: FLINK-1388
URL: https://issues.apache.org/jira/browse/FLINK-1388
Project: Flink
Issue Type: New Feature
Components: Java API
Reporter: Timo Walther
Assignee: Adnan Khan
Priority: Minor
It would be great if one could simply write out POJOs in CSV format.
{code}
public class MyPojo {
String a;
int b;
}
{code}
to:
{code}
# CSV file of org.apache.flink.MyPojo: String a, int b
Hello World, 42
Hello World 2, 47
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv
[
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357717#comment-14357717
]
Adnan Khan commented on FLINK-1388:
---
Does that make any references to TypeInformation in OutputFormat unuseable once
it's shipped to TaskManager on the cluster? Or does it not guarantee anything
about to non-serializable class members like java.lang.reflect.Field.
POJO support for writeAsCsv
---
Key: FLINK-1388
URL: https://issues.apache.org/jira/browse/FLINK-1388
Project: Flink
Issue Type: New Feature
Components: Java API
Reporter: Timo Walther
Assignee: Adnan Khan
Priority: Minor
It would be great if one could simply write out POJOs in CSV format.
{code}
public class MyPojo {
String a;
int b;
}
{code}
to:
{code}
# CSV file of org.apache.flink.MyPojo: String a, int b
Hello World, 42
Hello World 2, 47
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv
[
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358863#comment-14358863
]
Adnan Khan commented on FLINK-1388:
---
Also - Why should pass the name of all the fields that should be written out?
If I have a reference to the POJO itself, I can always just grab the
{{Fields[]}} using Java's reflection api. I feel I've missed something.
POJO support for writeAsCsv
---
Key: FLINK-1388
URL: https://issues.apache.org/jira/browse/FLINK-1388
Project: Flink
Issue Type: New Feature
Components: Java API
Reporter: Timo Walther
Assignee: Adnan Khan
Priority: Minor
It would be great if one could simply write out POJOs in CSV format.
{code}
public class MyPojo {
String a;
int b;
}
{code}
to:
{code}
# CSV file of org.apache.flink.MyPojo: String a, int b
Hello World, 42
Hello World 2, 47
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
