Re: Parquet Writer Operator

Dev, The FileSystemWalWritter closes the temporary as soon as it gets rotated. It renames (finalizes) the temporary file to the actual file until the window is committed. The mapping of temporary file to actual file is present in the checkpointed state. The FileSystemWalReader reads from the

Re: Parquet Writer Operator

Hi, WAL based approach : The FileSystemWAL.FileSystemWALWriter closes a temporary file only after the window is committed.We cannot read any such files till this point. Once this file is committed, in the same committed callback the ParquetOutputOperator will have to read the committed files,

[jira] [Commented] (APEXMALHAR-2120) Fix bugs on KafkaInputOperatorTest AbstractKafkaInputOperator

[ https://issues.apache.org/jira/browse/APEXMALHAR-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334833#comment-15334833 ] ASF GitHub Bot commented on APEXMALHAR-2120: GitHub user brightchen opened a pull

[GitHub] apex-malhar pull request #321: APEXMALHAR-2120 #resolve #comment solve probl...

GitHub user brightchen opened a pull request: https://github.com/apache/apex-malhar/pull/321 APEXMALHAR-2120 #resolve #comment solve problems of KafkaInputOperato… …rTest and AbstractKafkaInputOperator You can merge this pull request into a Git repository by running: $ git

[jira] [Commented] (APEXMALHAR-2120) Fix bugs on KafkaInputOperatorTest AbstractKafkaInputOperator

[ https://issues.apache.org/jira/browse/APEXMALHAR-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334589#comment-15334589 ] ASF GitHub Bot commented on APEXMALHAR-2120: Github user brightchen closed the pull

[GitHub] apex-malhar pull request #320: APEXMALHAR-2120 #resolve #comment solve probl...

Github user brightchen closed the pull request at: https://github.com/apache/apex-malhar/pull/320 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

Re: [ANNOUNCE] New Apache Apex PMC Member: Siyuan Hua

Congratulations!!! On Thu, Jun 16, 2016 at 10:06 AM Bright Chen wrote: > Siyuan, Congratulations! > > thanks > Bright > > > On Jun 16, 2016, at 9:27 AM, Pradeep A. Dalvi wrote: > > > > Congrats Siyuan! > > > > -prad > > > > On Thursday, June 16, 2016,

Re: [ANNOUNCE] New Apache Apex PMC Member: Siyuan Hua

Siyuan, Congratulations! thanks Bright > On Jun 16, 2016, at 9:27 AM, Pradeep A. Dalvi wrote: > > Congrats Siyuan! > > -prad > > On Thursday, June 16, 2016, Dongming Liang wrote: > >> Congrats, Siyuan! >> >> Thanks, >> - Dongming >> >> Dongming

Re: [ANNOUNCE] New Apache Apex PMC Member: Siyuan Hua

Congrats Siyuan! -prad On Thursday, June 16, 2016, Dongming Liang wrote: > Congrats, Siyuan! > > Thanks, > - Dongming > > Dongming LIANG > > dongming.li...@gmail.com > > On Thu, Jun 16, 2016 at 12:19 AM, Shubham Pathak

Re: [ANNOUNCE] New Apache Apex PMC Member: Siyuan Hua

Congrats, Siyuan! Thanks, - Dongming Dongming LIANG dongming.li...@gmail.com On Thu, Jun 16, 2016 at 12:19 AM, Shubham Pathak wrote: > Congrats Siyuan ! > > Thanks, > Shubham > > On Thu, Jun 16, 2016 at 11:17 AM, Ashish Tadose >

[jira] [Commented] (APEXMALHAR-2119) Make DirectoryScanner in AbstractFileInputOperator inheritance friendly.

[ https://issues.apache.org/jira/browse/APEXMALHAR-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333860#comment-15333860 ] ASF GitHub Bot commented on APEXMALHAR-2119: Github user amberarrow commented on a diff

[GitHub] apex-malhar pull request #318: APEXMALHAR-2119 add setters for partition cou...

Github user amberarrow commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/318#discussion_r67351327 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/AbstractFileInputOperator.java --- @@ -1110,6 +1116,16 @@ public String toString()

[jira] [Commented] (APEXMALHAR-1966) Cassandra output operator improvements

[ https://issues.apache.org/jira/browse/APEXMALHAR-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333545#comment-15333545 ] ASF GitHub Bot commented on APEXMALHAR-1966: Github user DT-Priyanka commented on a diff

[GitHub] apex-malhar pull request #295: APEXMALHAR-1966: Update cassandra output opre...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/295#discussion_r67323229 --- Diff: contrib/src/main/java/com/datatorrent/contrib/cassandra/CassandraPOJOOutputOperator.java --- @@ -50,17 +48,21 @@ * @since 2.1.0

[jira] [Commented] (APEXMALHAR-1966) Cassandra output operator improvements

[ https://issues.apache.org/jira/browse/APEXMALHAR-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333409#comment-15333409 ] ASF GitHub Bot commented on APEXMALHAR-1966: Github user DT-Priyanka commented on a diff

[GitHub] apex-malhar pull request #295: APEXMALHAR-1966: Update cassandra output opre...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/295#discussion_r67311141 --- Diff: contrib/src/main/java/com/datatorrent/contrib/cassandra/CassandraPOJOOutputOperator.java --- @@ -162,35 +160,75 @@ public void

Re: [ANNOUNCE] New Apache Apex PMC Member: Siyuan Hua

Congrats Siyuan ! Thanks, Shubham On Thu, Jun 16, 2016 at 11:17 AM, Ashish Tadose wrote: > Congratulations Siyuan. > > Ashish > > > On 16-Jun-2016, at 10:49 AM, Aniruddha Thombare < > anirud...@datatorrent.com> wrote: > > > > Congratulations!!! > > > > Thanks, > > > > A