from:"Piyush Narang \(JIRA\)"

[jira] [Created] (TEZ-3588) Tez containers exiting with Yarn status = -105 (killed by app master) instead of 0

2017-01-24 Thread Piyush Narang (JIRA)

Piyush Narang created TEZ-3588:
--

 Summary: Tez containers exiting with Yarn status = -105 (killed by 
app master) instead of 0
 Key: TEZ-3588
 URL: https://issues.apache.org/jira/browse/TEZ-3588
 Project: Apache Tez
  Issue Type: Bug
Reporter: Piyush Narang


Noticed that Tez containers on exit aren't setting the containerExitStatus to 
0. Instead they seem to be setting this to -105 (killed by app master). This is 
the case even when the DAG succeeds and the given container completed 
successfully. 

Chatting with [~mingma] and he mentioned that this might require a fix similar 
to what was done in case of Hadoop - 
https://issues.apache.org/jira/browse/MAPREDUCE-5465. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2658) Create a CLI utility tool to track Tez DAG/Application Stats

2016-10-03 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15543220#comment-15543220
 ] 

Piyush Narang commented on TEZ-2658:


hey [~jeagles] / [~hitesh]. The jira that I was working on to add some APIs is 
here: https://issues.apache.org/jira/browse/TEZ-3369 (associated PR: 
https://github.com/apache/tez/pull/13). Unfortunately I haven't had the time to 
work on some of Hitesh's feedback. Got pulled into a few other tasks at work. I 
think I should be able to pick it back up in the next 2-3 weeks. I don't want 
to block you if you need it more urgently. So if you'd like to pick it up, let 
me know. 

> Create a CLI utility tool to track Tez DAG/Application Stats
> 
>
> Key: TEZ-2658
> URL: https://issues.apache.org/jira/browse/TEZ-2658
> Project: Apache Tez
>  Issue Type: Improvement
>Reporter: Saikat
>Assignee: Saikat
> Attachments: TEZ-2658.1.patch, TEZ-2658.2.patch, TEZ-2658.3.patch, 
> TEZ-2658.4.patch, TEZ-2658.5.patch, TEZ-2658.6.patch, TEZ-2658.7.patch, 
> TEZ-2658.8.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-08-19 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428986#comment-15428986
 ] 

Piyush Narang commented on TEZ-3369:


Thanks a ton [~hitesh] :-)

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild 
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That 
> way the cascading code wouldn't need updating if the uri changed / json 
> format changed, it would end up being updated in these clients as well. I 
> suspect adding this to DAGClient would require more work as it'll also need 
> to be supported by the RPCClient and I don't think there are the relevant 
> protos and such available. 
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient 
> (currently it just implements Closeable). This will not require any changes 
> on the Cascading side as DAGClientTimelineImpl will continue to be a 
> DAGClient. 
> Full stack trace:
> {code}
> Exception in thread "flow 
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
>  java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
>   at 
> cascading.flow.tez.planner.Hadoop2TezFlowStepJob$1.getJobStatusClient(Hadoop2TezFlowStepJob.java:117)
>   at 
> cascading.flow.tez.planner.Hadoop2TezFlowStepJob$1.getJobStatusClient(Hadoop2TezFlowStepJob.java:105)
>   at 
> cascading.stats.tez.TezStepStats$1.getJobStatusClient(TezStepStats.java:60)
>   at 
>

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-08-19 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428791#comment-15428791
 ] 

Piyush Narang commented on TEZ-3369:


Ping [~hitesh], can you take a quick look at the PR to see if it looks 
reasonable? 

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild 
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That 
> way the cascading code wouldn't need updating if the uri changed / json 
> format changed, it would end up being updated in these clients as well. I 
> suspect adding this to DAGClient would require more work as it'll also need 
> to be supported by the RPCClient and I don't think there are the relevant 
> protos and such available. 
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient 
> (currently it just implements Closeable). This will not require any changes 
> on the Cascading side as DAGClientTimelineImpl will continue to be a 
> DAGClient. 
> Full stack trace:
> {code}
> Exception in thread "flow 
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
>  java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
>   at 
> cascading.flow.tez.planner.Hadoop2TezFlowStepJob$1.getJobStatusClient(Hadoop2TezFlowStepJob.java:117)
>   at 
> cascading.flow.tez.planner.Hadoop2TezFlowStepJob$1.getJobStatusClient(Hadoop2TezFlowStepJob.java:105)
>   at 
> cascading.stats.tez.TezStepStats$1.getJobStatusClient(TezStepStats.java:60)
>   at 
>

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-08-12 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419453#comment-15419453
 ] 

Piyush Narang commented on TEZ-3369:


[~hitesh] / [~cwensel] / [~mingma] - put up a WIP PR on GitHub - 
https://github.com/apache/tez/pull/13
This isn't the final ready to merge version as there are still a few things to 
tackle but I wanted to get some feedback from you folks earlier rather than 
later. So if you could take a look at the API / return types / high level 
approach that would be great. I'll continue to refine and clean up stuff in the 
meanwhile. 

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild 
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That 
> way the cascading code wouldn't need updating if the uri changed / json 
> format changed, it would end up being updated in these clients as well. I 
> suspect adding this to DAGClient would require more work as it'll also need 
> to be supported by the RPCClient and I don't think there are the relevant 
> protos and such available. 
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient 
> (currently it just implements Closeable). This will not require any changes 
> on the Cascading side as DAGClientTimelineImpl will continue to be a 
> DAGClient. 
> Full stack trace:
> {code}
> Exception in thread "flow 
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
>  java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
>

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-08-08 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412213#comment-15412213
 ] 

Piyush Narang commented on TEZ-3369:


Thanks Andre. Based on what I see I guess the intention is to populate 
structures like: 
[TezAttempt|https://github.com/cwensel/cascading/blob/d0a6ca02af2126ce9490f58925b9a2a801fb9f65/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/TezSliceStats.java#L37].
 If I understand correctly, that currently isn't being populated (I might be 
missing something / ..) 

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild 
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That 
> way the cascading code wouldn't need updating if the uri changed / json 
> format changed, it would end up being updated in these clients as well. I 
> suspect adding this to DAGClient would require more work as it'll also need 
> to be supported by the RPCClient and I don't think there are the relevant 
> protos and such available. 
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient 
> (currently it just implements Closeable). This will not require any changes 
> on the Cascading side as DAGClientTimelineImpl will continue to be a 
> DAGClient. 
> Full stack trace:
> {code}
> Exception in thread "flow 
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
>  java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
>   at 
>

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-08-05 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410357#comment-15410357
 ] 

Piyush Narang commented on TEZ-3369:


Yeah so I guess regarding the static info, I'm thinking of returning (for a 
given dagId):
{code}
message DAGInformationProto {
  required string name = 1;
  required string dagId = 2;
  repeated VertexInformationProto vertices = 4;
}
message VertexInformationProto {
  required string name = 1;
  required string id = 2;
}
{code}

I think Cascading can look up / construct the vertex ID -> name mappings from 
this object. 

The dynamic stuff is needed not just while it is running but also after 
completion. I'm thinking of something on the lines of:
{code}
message TaskInformationProto {
  required string id = 1;
  required TaskStateProto state = 2;
  required double progress = 3;
  required int64 scheduledTime = 4;
  required int64 startTime = 5;
  required int64 endTime = 6;
  required string successfulAttemptId = 7;
  required string diagnostics = 8;
  repeated TezCounterGroupProto counters = 9;
  //collection of all attempts? 
}
{code}

Doesn't seem like Cascading needs the task attempt details as of now (apart 
from which attemptId was successful), but it does seem like an API that could 
be added in the future. 
To call this API:
getTaskInformation(dagId, vertexId, taskId) -> returns a TaskInfo object

It would be nice to retrieve progress / status details for a bunch of tasks. I 
think we'll need to have this supported by pagination (else the data might be 
too much in case of vertices that have a lot of tasks). I believe this should 
be possible if we use the vertexMap (LinkedHashMap) in 
[DagImpl|https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java#L1804].
 

Looked at the code in DAGClientImpl and I see what you mean by the [client 
redirecting|https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientImpl.java#L233].
 Will need to update the new api implementation to use this. I think we 
hopefully should have the data available in the Timeline side, so it should 
mostly just be implementing on the RPC / AM side. 

Does this sound reasonable? I can put out the full proto of the RPCs if that'll 
help make things clear. 

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public

[jira] [Comment Edited] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-08-04 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408750#comment-15408750
 ] 

Piyush Narang edited comment on TEZ-3369 at 8/5/16 1:36 AM:


Ok, spent some time checking out the Tez code, I'm thinking of the following 
APIs. [~hitesh], would be great to get your thoughts.
1) getDAGInformation(dagID) -> returns a DAGInformation struct which has 
details like dag name, id, list of vertices (just names, ids)
2) getTaskInformation(dagId, vertexId, taskId) -> returns a TaskInfo object 
(has taskId, state, scheduled time, start time, endTime, counters etc)
3) getTaskInformationList(dagId, vertexId, startTaskId, limit) -> returns a 
List. This provides a paginated view of the tasks for a vertex. 

Had a follow up question based on looking at the rpc / http clients. Seems like 
the RPC implementation only supports vending out DAG details for the current 
DAG (whereas ATS seems to provide info for tasks belonging to others as well?). 
Is there any way we can retrieve this in the RPC server too? Would it be ok to 
keep cached copies of the DAG objects that are not current in the DAGAppMaster? 


was (Author: pnarang):
Ok, spent some time checking out the Tez code, here I'm thinking of the 
following APIs. [~hitesh], would be great to get your thoughts.
1) getDAGInformation(dagID) -> returns a DAGInformation struct which has 
details like dag name, id, list of vertices (just names, ids)
2) getTaskInformation(dagId, vertexId, taskId) -> returns a TaskInfo object 
(has taskId, state, scheduled time, start time, endTime, counters etc)
3) getTaskInformationList(dagId, vertexId, startTaskId, limit) -> returns a 
List. This provides a paginated view of the tasks for a vertex. 

Had a follow up question based on looking at the rpc / http clients. Seems like 
the RPC implementation only supports vending out DAG details for the current 
DAG (whereas ATS seems to provide info for tasks belonging to others as well?). 
Is there any way we can retrieve this in the RPC server too? Would it be ok to 
keep cached copies of the DAG objects that are not current in the DAGAppMaster? 

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
>

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-08-04 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408750#comment-15408750
 ] 

Piyush Narang commented on TEZ-3369:


Ok, spent some time checking out the Tez code, here I'm thinking of the 
following APIs. [~hitesh], would be great to get your thoughts.
1) getDAGInformation(dagID) -> returns a DAGInformation struct which has 
details like dag name, id, list of vertices (just names, ids)
2) getTaskInformation(dagId, vertexId, taskId) -> returns a TaskInfo object 
(has taskId, state, scheduled time, start time, endTime, counters etc)
3) getTaskInformationList(dagId, vertexId, startTaskId, limit) -> returns a 
List. This provides a paginated view of the tasks for a vertex. 

Had a follow up question based on looking at the rpc / http clients. Seems like 
the RPC implementation only supports vending out DAG details for the current 
DAG (whereas ATS seems to provide info for tasks belonging to others as well?). 
Is there any way we can retrieve this in the RPC server too? Would it be ok to 
keep cached copies of the DAG objects that are not current in the DAGAppMaster? 

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild 
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That 
> way the cascading code wouldn't need updating if the uri changed / json 
> format changed, it would end up being updated in these clients as well. I 
> suspect adding this to DAGClient would require more work as it'll also need 
> to be supported by the RPCClient and I don't think there are the relevant 
> protos and such available. 
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient 
>

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-07-29 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400219#comment-15400219
 ] 

Piyush Narang commented on TEZ-3369:


[~hitesh], makes sense to skip out dependency info from the status responses. 
I'd imagine that including the vertexID would be ok though right? (Not too much 
more info. )
Anyway, if we're providing a different way to get at that information, we 
should probably be ok. Not totally clear what extra DAG info the Pig folks 
wanted based on their linked jira it seemed like counters to help estimate the 
amount of parallelism. 

I'm guessing the 
[DAGPlan|https://github.com/apache/tez/blob/master/tez-api/src/main/proto/DAGApiRecords.proto#L200]
 object is the one we want to base this API off and trim down. I'll look into 
what the definition of that should be and the code around adding it as an API. 

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild 
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That 
> way the cascading code wouldn't need updating if the uri changed / json 
> format changed, it would end up being updated in these clients as well. I 
> suspect adding this to DAGClient would require more work as it'll also need 
> to be supported by the RPCClient and I don't think there are the relevant 
> protos and such available. 
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient 
> (currently it just implements Closeable). This will not require any changes 
> on the Cascading side as DAGClientTimelineImpl will continue to be a 
> DAGClient. 
> Full stack trace:
> {code}
> Exception in thread "flow 
>

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-07-29 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399849#comment-15399849
 ] 

Piyush Narang commented on TEZ-3369:


[~hitesh] that makes sense. From what I understand just adding the basic stuff 
will help unblock integration with Cascading. That currently breaks on the new 
version of Tez. I think some of the other aspects would be great as well but 
we're not blocked on them (some of the other Cascading use cases might be). 

Do you guys have a sense of the design / API you have in mind for this? I could 
take a stab at an initial implementation if no one's already working on this. 
Not super familiar with the Tez code so have a few questions to help save time:
1) Would it be ok to add the vertexID and vertexChildren details to the 
VertexStatus object + proto?
2) To confirm, the server side code is: 
[DAGClientAMProtocolBlockingPBServerImpl|https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/api/client/rpc/DAGClientAMProtocolBlockingPBServerImpl.java#L58]?

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild 
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That 
> way the cascading code wouldn't need updating if the uri changed / json 
> format changed, it would end up being updated in these clients as well. I 
> suspect adding this to DAGClient would require more work as it'll also need 
> to be supported by the RPCClient and I don't think there are the relevant 
> protos and such available. 
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient 
> (currently it just implements Closeable). This will not require any changes 
>

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-07-27 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396697#comment-15396697
 ] 

Piyush Narang commented on TEZ-3369:


[~cwensel], to confirm, apart from the getVertexID Cascading also needs the 
other two methods right - getVertexChildren & getVertexChild? If just the 
vertexID was included in the VertexStatus you'd still need a way to get the 
child(ren)?

[~hitesh], given these methods, what do you guys think is the best way forward? 
Looking at the other jira that Chris linked, there was a suggestion of 
TezATSLib but I think Chris mentioned that they have encountered / heard of 
folks with scale issues with ATS (though it seems as though the current 
Cascading implementation still uses ATS?). 



> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild 
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That 
> way the cascading code wouldn't need updating if the uri changed / json 
> format changed, it would end up being updated in these clients as well. I 
> suspect adding this to DAGClient would require more work as it'll also need 
> to be supported by the RPCClient and I don't think there are the relevant 
> protos and such available. 
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient 
> (currently it just implements Closeable). This will not require any changes 
> on the Cascading side as DAGClientTimelineImpl will continue to be a 
> DAGClient. 
> Full stack trace:
> {code}
> Exception in thread "flow 
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
>  java.lang.ClassCastException:

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-07-25 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392723#comment-15392723
 ] 

Piyush Narang commented on TEZ-3369:


[~hitesh] thanks for getting back. Based on what I understand after short 
forays into the Cascading code, the extra APIs that Cascading is using right 
now are in 
[TimelineClient|https://github.com/cwensel/cascading/blob/wip-3.2/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java]:
{code}
public interface TimelineClient
  {
  String getVertexID( String vertexName ) throws IOException, TezException;

  Iterator getVertexChildren( String vertexID, int limit, String 
startTaskID ) throws IOException, TezException;

  TaskStatus getVertexChild( String taskID ) throws TezException;
  }
{code}

They have an implementation 
[TezTimelineClient|https://github.com/cwensel/cascading/blob/wip-3.2/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
 that implements this interface as well as the DAGClient interface. For the 
DAGClient APIs, they mostly seem to defer to the dag client object that is 
passed in (I think this is returned from Tez). For the TimelineClient calls 
(getVertexID, getVertexChildren, getVertexChild) they end up using the http 
based endpoints taking advantage of the code in DAGClientTimelineImpl (which 
implements the http client in Tez). 

If its just these 3 calls, do you have a sense of how long it would take to 
implement (I'm not too familiar with the rpc implementation)? One option is to 
go ahead with my suggested approach 2 (making DAGClientInternal extend 
DAGClient) and file a follow up jira so that we can track doing this the right 
way and still be unblocked?

If someone with more Cascading chops ([~cwensel]?) could chime in if I'm 
missing something / if there's other APIs that Cascading wants apart from these 
3, that would be great. I can also spin up a new thread on Cascading dev where 
we could ask this. 



> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
>

[jira] [Comment Edited] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-07-21 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388709#comment-15388709
 ] 

Piyush Narang edited comment on TEZ-3369 at 7/22/16 1:14 AM:
-

Tacked on a proof of concept PR of option 2 above - making DAGClientInternal 
extend DAGClient. Of course if option 1 (or a different one) are preferred can 
update accordingly. Put this up to facilitate a conversation. 


was (Author: pnarang):
Tacked on a proof of concept PR of option 2 above - making DAGClientInternal 
extend DAGClient. 

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild 
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That 
> way the cascading code wouldn't need updating if the uri changed / json 
> format changed, it would end up being updated in these clients as well. I 
> suspect adding this to DAGClient would require more work as it'll also need 
> to be supported by the RPCClient and I don't think there are the relevant 
> protos and such available. 
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient 
> (currently it just implements Closeable). This will not require any changes 
> on the Cascading side as DAGClientTimelineImpl will continue to be a 
> DAGClient. 
> Full stack trace:
> {code}
> Exception in thread "flow 
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
>  java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
>   at 
>

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-07-21 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388709#comment-15388709
 ] 

Piyush Narang commented on TEZ-3369:


Tacked on a proof of concept PR of option 2 above - making DAGClientInternal 
extend DAGClient. 

> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of 
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient 
> | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break | 
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit | 
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
>  (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient 
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>  
> As there's no good way to get the vertexID / vertexChildren / vertexChild 
> (correct me if I'm wrong), they end up extending from the 
> DAGClientTimelineImpl which has the http client and json parsing code to 
> allow [things like this | 
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
>   public String getVertexID( String vertexName ) throws IOException, 
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be 
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format = 
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild 
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That 
> way the cascading code wouldn't need updating if the uri changed / json 
> format changed, it would end up being updated in these clients as well. I 
> suspect adding this to DAGClient would require more work as it'll also need 
> to be supported by the RPCClient and I don't think there are the relevant 
> protos and such available. 
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient 
> (currently it just implements Closeable). This will not require any changes 
> on the Cascading side as DAGClientTimelineImpl will continue to be a 
> DAGClient. 
> Full stack trace:
> {code}
> Exception in thread "flow 
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
>  java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
>   at 
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
>   at 
> cascading.flow.tez.planner.Hadoop2TezFlowStepJob$1.getJobStatusClient(Hadoop2TezFlowStepJob.java:117)
>   at 
> cascading.flow.tez.planner.Hadoop2TezFlowStepJob$1.getJobStatusClient(Hadoop2TezFlowStepJob.java:105)
>   at 
> cascading.stats.tez.TezStepStats$1.getJobStatusClient(TezStepStats.java:60)
>   at

[jira] [Created] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

2016-07-21 Thread Piyush Narang (JIRA)

Piyush Narang created TEZ-3369:
--

 Summary: Fixing Tez's DAGClients to work with Cascading
 Key: TEZ-3369
 URL: https://issues.apache.org/jira/browse/TEZ-3369
 Project: Apache Tez
  Issue Type: Bug
Reporter: Piyush Narang


Hi,
We seem to be running into issues when we try to use the newest version of Tez 
(0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
{code}
java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient cannot 
be cast to org.apache.tez.dag.api.client.DAGClient
at 
cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
{code}
(Full stack trace at the end)

Relevant Cascading code is:
1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient | 
https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
2) [TezTimelineClient extends from DAGClientTimelineImpl | 
https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
3) [DAGClientTimelineImpl extends from DAGClientInternal | 
https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
4) [DAGClientInternal extends Closeable which is why things break | 
https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].

This behavior was 'broken' in this [commit | 
https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667] 
(release 0.8.3)

The TezTimelineClient in Cascading seems to do two things:
1) DAGClient functionalities - ends up delegating to the inner DAGClient object.
2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this 
[interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
 

As there's no good way to get the vertexID / vertexChildren / vertexChild 
(correct me if I'm wrong), they end up extending from the DAGClientTimelineImpl 
which has the http client and json parsing code to allow [things like this | 
https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:

{code}
@Override
  public String getVertexID( String vertexName ) throws IOException, 
TezException
{
// the filter 'vertexName' is in the 'otherinfo' field, so it must be 
requested, otherwise timeline server throws
// an NPE. to be safe, we include both fields in the result
String format = 
"%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID, 
dagId, vertexName, FILTER_BY_FIELDS );

JSONObject jsonRoot = getJsonRootEntity( url );
JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
...
{code}

Some options I can think of:
1) Ideally these methods getVertexID / getVertexChildren / getVertexChild would 
be part of DAGClient? Or even part of the DAGClientTimelineImpl? That way the 
cascading code wouldn't need updating if the uri changed / json format changed, 
it would end up being updated in these clients as well. I suspect adding this 
to DAGClient would require more work as it'll also need to be supported by the 
RPCClient and I don't think there are the relevant protos and such available. 
2) A simpler fix would be to have DAGClientInternal extend DAGClient (currently 
it just implements Closeable). This will not require any changes on the 
Cascading side as DAGClientTimelineImpl will continue to be a DAGClient. 

Full stack trace:
{code}
Exception in thread "flow 
com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
 java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient 
cannot be cast to org.apache.tez.dag.api.client.DAGClient
at 
cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
at 
cascading.flow.tez.planner.Hadoop2TezFlowStepJob$1.getJobStatusClient(Hadoop2TezFlowStepJob.java:117)
at 
cascading.flow.tez.planner.Hadoop2TezFlowStepJob$1.getJobStatusClient(Hadoop2TezFlowStepJob.java:105)
at 
cascading.stats.tez.TezStepStats$1.getJobStatusClient(TezStepStats.java:60)
at 
cascading.stats.tez.TezStepStats$1.getJobStatusClient(TezStepStats.java:56)
at cascading.stats.CounterCache.cachedCounters(CounterCache.java:229)
at cascading.stats.CounterCache.cachedCounters(CounterCache.java:187)
at cascading.stats.CounterCache.getCounterValue(CounterCache.java:167)
at 
cascading.stats.BaseCachedStepStats.getCounterValue(BaseCachedStepStats.java:105)
at

[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-19 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385046#comment-15385046
 ] 

Piyush Narang commented on TEZ-3348:


Thanks for the help on this [~hitesh] :-)

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Fix For: 0.9.0, 0.8.5
>
> Attachments: 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-18 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382816#comment-15382816
 ] 

Piyush Narang commented on TEZ-3348:


[~hitesh] - Sorry I'll keep that in mind the next time :-). Realized that the 
last patch I uploaded was generated incorrectly which is why the tests & 
findbugs failed. Uploaded a new one (which is the same as the PR). Hopefully 
Jenkins won't crib this time. 

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-18 Thread Piyush Narang (JIRA)


 [ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piyush Narang updated TEZ-3348:
---
Attachment: 11.patch

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-18 Thread Piyush Narang (JIRA)


 [ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piyush Narang updated TEZ-3348:
---
Attachment: (was: 11.patch)

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-18 Thread Piyush Narang (JIRA)


 [ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piyush Narang updated TEZ-3348:
---
Attachment: (was: 11.patch)

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-18 Thread Piyush Narang (JIRA)


 [ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piyush Narang updated TEZ-3348:
---
Attachment: (was: 11.patch)

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-15 Thread Piyush Narang (JIRA)


 [ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piyush Narang updated TEZ-3348:
---
Attachment: 11.patch

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch, 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-15 Thread Piyush Narang (JIRA)


 [ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piyush Narang updated TEZ-3348:
---
Attachment: 11.patch

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-15 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380325#comment-15380325
 ] 

Piyush Narang commented on TEZ-3348:


[~hitesh] updated the patch but I can't seem to trigger the jenkins build. If 
you get the time, could you help retrigger please? Thanks.

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-15 Thread Piyush Narang (JIRA)


 [ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piyush Narang updated TEZ-3348:
---
Attachment: 11.patch

Updated patch

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-15 Thread Piyush Narang (JIRA)


 [ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piyush Narang updated TEZ-3348:
---
Attachment: (was: 11_07142016.patch)

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-14 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378345#comment-15378345
 ] 

Piyush Narang commented on TEZ-3348:


Attached a new patch to fix the javac warnings - 11_07142016.patch. Not sure if 
the failing test (TestContainerReuse) is reliable. I ran it a bunch of times 
and it seems to go through locally.

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11_07142016.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-14 Thread Piyush Narang (JIRA)


 [ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piyush Narang updated TEZ-3348:
---
Attachment: 11_07142016.patch

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11_07142016.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-13 Thread Piyush Narang (JIRA)


[ 
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376139#comment-15376139
 ] 

Piyush Narang commented on TEZ-3348:


Thanks for getting back Hitesh - Put out a PR - 
https://github.com/apache/tez/pull/11

> NullPointerException in Tez MROutput while trying to write using Parquet's 
> DeprecatedParquetOutputFormat
> 
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Piyush Narang
>
> Trying to run some Tez MR jobs that write out some data using Parquet to 
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.Path.(Path.java:105)
>   at org.apache.hadoop.fs.Path.(Path.java:94)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
>   at 
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
>   at 
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the 
> workOutputPath - 
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
> ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in 
> the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the 
> MROutput.initCommitter method - 
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>  
> This call however, is made after the call to access the workOutputPath as 
> part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>   oldApiTaskAttemptContext =
>   new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>   jobConf, taskAttemptId,
>   new MRTaskReporter(getContext()));
>   initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>   oldOutputFormat = jobConf.getOutputFormat();
>   outputFormatClassName = oldOutputFormat.getClass().getName();
>   FileSystem fs = FileSystem.get(jobConf);
>   String finalName = getOutputName();
>   oldRecordWriter =
>   oldOutputFormat.getRecordWriter(
>   fs, jobConf, finalName, new 
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds 
> reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

2016-07-13 Thread Piyush Narang (JIRA)

Piyush Narang created TEZ-3348:
--

 Summary: NullPointerException in Tez MROutput while trying to 
write using Parquet's DeprecatedParquetOutputFormat
 Key: TEZ-3348
 URL: https://issues.apache.org/jira/browse/TEZ-3348
 Project: Apache Tez
  Issue Type: Bug
Reporter: Piyush Narang


Trying to run some Tez MR jobs that write out some data using Parquet to HDFS. 
When I try to do so, end up seeing a NPE in the Parquet code:
{code}
java.lang.NullPointerException
at org.apache.hadoop.fs.Path.(Path.java:105)
at org.apache.hadoop.fs.Path.(Path.java:94)
at 
org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
at 
org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
at 
org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
at 
org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
at 
org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
{code}

The flow seems to be:
1) The Parquet deprecated output format class tries to read the workOutputPath 
- 
https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
2) This calls FileOutputFormat.getWorkOutputPath(...) - 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR 
("mapreduce.task.output.dir") constant. 
4) This ends up being null and in the Parquet code we end up with an NPE in the 
Path class. 

Looking at the Tez code, we are setting the workOutputPath in the 
MROutput.initCommitter method - 
https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
 

This call however, is made after the call to access the workOutputPath as part 
of outputFormat.getRecordWriter(). 

I tried out a run where I moved this initCommitter call up:
{code}
else {
  oldApiTaskAttemptContext =
  new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
  jobConf, taskAttemptId,
  new MRTaskReporter(getContext()));
  initCommitter(jobConf, useNewApi); // before the getRecordWriter call

  oldOutputFormat = jobConf.getOutputFormat();
  outputFormatClassName = oldOutputFormat.getClass().getName();

  FileSystem fs = FileSystem.get(jobConf);
  String finalName = getOutputName();

  oldRecordWriter =
  oldOutputFormat.getRecordWriter(
  fs, jobConf, finalName, new 
MRReporter(getContext().getCounters()));
}
{code}

I tried out a run with this and it seems to succeed. If this sounds reasonable, 
I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (TEZ-3588) Tez containers exiting with Yarn status = -105 (killed by app master) instead of 0

[jira] [Commented] (TEZ-2658) Create a CLI utility tool to track Tez DAG/Application Stats

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Comment Edited] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Comment Edited] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Created] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading

[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

[jira] [Created] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

31 matches

Site Navigation

Mail list logo

Footer information