[jira] [Commented] (APEXMALHAR-2312) NullPointerException in FileSplitterInput only if the file path is specified for attribute instead of directory path

2016-10-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612497#comment-15612497
 ] 

ASF GitHub Bot commented on APEXMALHAR-2312:


Github user asfgit closed the pull request at:

https://github.com/apache/apex-malhar/pull/463


> NullPointerException in FileSplitterInput only if the file path is specified 
> for attribute  instead of directory path
> 
>
> Key: APEXMALHAR-2312
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2312
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Deepak Narkhede
>Assignee: Deepak Narkhede
>Priority: Critical
>
> Problem Statement:
> ==
> NullPointerException seen in FileSplitterInput only if the file path is 
> specified for attribute  instead of directory path.
> Description:
> ===
> 1) TimeBasedDirectoryScanner threads part of scanservice tries to scan the 
> directories/files.
> 2) Each thread checks with help of isIterationCompleted() [referenceTimes] 
> method whether scanned of last iteration are processed by operator thread.
> 3) Previously it used to work because HashMap (referenceTimes) used to return 
> null even if last scanned directory path is null.
> 4) Recently referenceTimes is changed to ConcurrentHashMap, so get() doesn't 
> allow null key's passed to ConcurrentHashMap get() method.
> 5) Hence NullPointerException is seen as if only file path is provided 
> directory path would be empty hence key would be empty.
> Solution:
> 
> Pre-check that directory path is null then we have completed last iterations 
> if only filepath is provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2312) NullPointerException in FileSplitterInput only if the file path is specified for attribute instead of directory path

2016-10-27 Thread Deepak Narkhede (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611295#comment-15611295
 ] 

Deepak Narkhede commented on APEXMALHAR-2312:
-

Changed to Critical because some basic functionality is broken.

> NullPointerException in FileSplitterInput only if the file path is specified 
> for attribute  instead of directory path
> 
>
> Key: APEXMALHAR-2312
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2312
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Deepak Narkhede
>Assignee: Deepak Narkhede
>Priority: Critical
>
> Problem Statement:
> ==
> NullPointerException seen in FileSplitterInput only if the file path is 
> specified for attribute  instead of directory path.
> Description:
> ===
> 1) TimeBasedDirectoryScanner threads part of scanservice tries to scan the 
> directories/files.
> 2) Each thread checks with help of isIterationCompleted() [referenceTimes] 
> method whether scanned of last iteration are processed by operator thread.
> 3) Previously it used to work because HashMap (referenceTimes) used to return 
> null even if last scanned directory path is null.
> 4) Recently referenceTimes is changed to ConcurrentHashMap, so get() doesn't 
> allow null key's passed to ConcurrentHashMap get() method.
> 5) Hence NullPointerException is seen as if only file path is provided 
> directory path would be empty hence key would be empty.
> Solution:
> 
> Pre-check that directory path is null then we have completed last iterations 
> if only filepath is provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2312) NullPointerException in FileSplitterInput only if the file path is specified for attribute instead of directory path

2016-10-23 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15600896#comment-15600896
 ] 

ASF GitHub Bot commented on APEXMALHAR-2312:


Github user deepak-narkhede closed the pull request at:

https://github.com/apache/apex-malhar/pull/463


> NullPointerException in FileSplitterInput only if the file path is specified 
> for attribute  instead of directory path
> 
>
> Key: APEXMALHAR-2312
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2312
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Deepak Narkhede
>Assignee: Deepak Narkhede
>Priority: Minor
>
> Problem Statement:
> ==
> NullPointerException seen in FileSplitterInput only if the file path is 
> specified for attribute  instead of directory path.
> Description:
> ===
> 1) TimeBasedDirectoryScanner threads part of scanservice tries to scan the 
> directories/files.
> 2) Each thread checks with help of isIterationCompleted() [referenceTimes] 
> method whether scanned of last iteration are processed by operator thread.
> 3) Previously it used to work because HashMap (referenceTimes) used to return 
> null even if last scanned directory path is null.
> 4) Recently referenceTimes is changed to ConcurrentHashMap, so get() doesn't 
> allow null key's passed to ConcurrentHashMap get() method.
> 5) Hence NullPointerException is seen as if only file path is provided 
> directory path would be empty hence key would be empty.
> Solution:
> 
> Pre-check that directory path is null then we have completed last iterations 
> if only filepath is provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2312) NullPointerException in FileSplitterInput only if the file path is specified for attribute instead of directory path

2016-10-23 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15600897#comment-15600897
 ] 

ASF GitHub Bot commented on APEXMALHAR-2312:


GitHub user deepak-narkhede reopened a pull request:

https://github.com/apache/apex-malhar/pull/463

APEXMALHAR-2312 Fix NullPointerException for FileSplitterInput Operat…

Problem Statement:
-
NullPointerException seen in FileSplitterInput only if the file path is 
specified for attribute  instead of directory path.

Description:
---
1) TimeBasedDirectoryScanner threads part of scanservice tries to scan the 
directories/files.
2) Each thread checks with help of isIterationCompleted() [referenceTimes] 
method whether scanned of last iteration are processed by operator thread.
3) Previously it used to work because HashMap (referenceTimes) used to 
return null even if last scanned directory path is null.
4) Recently referenceTimes is changed to ConcurrentHashMap, so get() 
doesn't allow null key's passed to ConcurrentHashMap get() method.
5) Hence NullPointerException is seen as if only file path is provided 
directory path would be empty hence key would be empty.

Solution:
---
Pre-check that directory path is null then we have completed last 
iterations if only filepath is provided.

Testing logs with fix for files/directories/sub-directories:
-
2016-10-21 11:20:38,382 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
Directory path: /user/deepak/files Sub-Directory or File path: 
/user/deepak/files/CustomerTxnData2
2016-10-21 11:20:38,382 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
Scan started for input /user/deepak/files
2016-10-21 11:20:38,386 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
scan /user/deepak/files
2016-10-21 11:20:33,372 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
discovered /user/deepak/files/CustomerTxnData 1477028632605
2016-10-21 11:20:33,372 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
discovered /user/deepak/files/CustomerTxnData1 1477028642067
2016-10-21 11:20:33,373 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
discovered /user/deepak/files/CustomerTxnData2 1477028645290
2016-10-21 11:20:33,373 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
scan complete 0 3



2016-10-21 11:25:50,697 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
Directory path: null Sub-Directory or File path: 
/user/deepak/files/CustomerTxnData
2016-10-21 11:25:50,697 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
Scan started for input /user/deepak/files/CustomerTxnData
2016-10-21 11:25:50,702 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
scan /user/deepak/files/CustomerTxnData
2016-10-21 11:25:50,704 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
scan complete

   

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/deepak-narkhede/apex-malhar APEXMALHAR-2312

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/463.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #463


commit 47f29f39393a4e43c8423153d32d12c9622872b5
Author: deepak-narkhede 
Date:   2016-10-21T06:44:34Z

APEXMALHAR-2312 Fix NullPointerException for FileSplitterInput Operator if 
filepath is specified.

Problem Description:
---
1) TimeBasedDirectoryScanner threads part of scanservice tries to scan the 
directories/files.
2) Each thread checks with help of isIterationCompleted() [referenceTimes] 
method whether scanned of last iteration are processed by operator thread.
3) Previously it used to work because HashMap (referenceTimes) used to 
return null even if last scanned directory path is null.
4) Recently referenceTimes is changed to ConcurrentHashMap, so get() 
doesn't allow null key's passed to ConcurrentHashMap get() method.
5) Hence NullPointerException is seen as if only file path is provided 
directory path would be empty hence key would be empty.

Solution:
-
Pre-check that directory path is null then we have completed last 
iterations if only filepath is provided.




> NullPointerException in FileSplitterInput only if the file path is specified 
> for attribute  instead of directory path
>

[jira] [Commented] (APEXMALHAR-2312) NullPointerException in FileSplitterInput only if the file path is specified for attribute instead of directory path

2016-10-21 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15594285#comment-15594285
 ] 

ASF GitHub Bot commented on APEXMALHAR-2312:


GitHub user deepak-narkhede opened a pull request:

https://github.com/apache/apex-malhar/pull/463

APEXMALHAR-2312 Fix NullPointerException for FileSplitterInput Operat…

Problem Statement:
-
NullPointerException seen in FileSplitterInput only if the file path is 
specified for attribute  instead of directory path.

Description:
---
1) TimeBasedDirectoryScanner threads part of scanservice tries to scan the 
directories/files.
2) Each thread checks with help of isIterationCompleted() [referenceTimes] 
method whether scanned of last iteration are processed by operator thread.
3) Previously it used to work because HashMap (referenceTimes) used to 
return null even if last scanned directory path is null.
4) Recently referenceTimes is changed to ConcurrentHashMap, so get() 
doesn't allow null key's passed to ConcurrentHashMap get() method.
5) Hence NullPointerException is seen as if only file path is provided 
directory path would be empty hence key would be empty.

Solution:
---
Pre-check that directory path is null then we have completed last 
iterations if only filepath is provided.

Testing logs with fix for files/directories/sub-directories:
-
2016-10-21 11:20:38,382 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
Directory path: /user/deepak/files Sub-Directory or File path: 
/user/deepak/files/CustomerTxnData2
2016-10-21 11:20:38,382 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
Scan started for input /user/deepak/files
2016-10-21 11:20:38,386 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
scan /user/deepak/files
2016-10-21 11:20:33,372 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
discovered /user/deepak/files/CustomerTxnData 1477028632605
2016-10-21 11:20:33,372 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
discovered /user/deepak/files/CustomerTxnData1 1477028642067
2016-10-21 11:20:33,373 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
discovered /user/deepak/files/CustomerTxnData2 1477028645290
2016-10-21 11:20:33,373 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
scan complete 0 3



2016-10-21 11:25:50,697 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
Directory path: null Sub-Directory or File path: 
/user/deepak/files/CustomerTxnData
2016-10-21 11:25:50,697 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
Scan started for input /user/deepak/files/CustomerTxnData
2016-10-21 11:25:50,702 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
scan /user/deepak/files/CustomerTxnData
2016-10-21 11:25:50,704 DEBUG com.datatorrent.lib.io.fs.FileSplitterInput: 
scan complete

   

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/deepak-narkhede/apex-malhar APEXMALHAR-2312

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/463.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #463


commit 47f29f39393a4e43c8423153d32d12c9622872b5
Author: deepak-narkhede 
Date:   2016-10-21T06:44:34Z

APEXMALHAR-2312 Fix NullPointerException for FileSplitterInput Operator if 
filepath is specified.

Problem Description:
---
1) TimeBasedDirectoryScanner threads part of scanservice tries to scan the 
directories/files.
2) Each thread checks with help of isIterationCompleted() [referenceTimes] 
method whether scanned of last iteration are processed by operator thread.
3) Previously it used to work because HashMap (referenceTimes) used to 
return null even if last scanned directory path is null.
4) Recently referenceTimes is changed to ConcurrentHashMap, so get() 
doesn't allow null key's passed to ConcurrentHashMap get() method.
5) Hence NullPointerException is seen as if only file path is provided 
directory path would be empty hence key would be empty.

Solution:
-
Pre-check that directory path is null then we have completed last 
iterations if only filepath is provided.




> NullPointerException in FileSplitterInput only if the file path is specified 
> for attribute  instead of directory path
>