[jira] [Commented] (PIG-3145) Parameters in core-site.xml and mapred-site.xml are not correctly substituted

2013-02-25 Thread Cheolsoo Park (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586476#comment-13586476
 ] 

Cheolsoo Park commented on PIG-3145:


To be clear, I ran unit test with hadoop23 which excludes TestHBaseStorage. 
That's why I didn't identify the issue.

Using my new patch, I can verify that TestHBaseStorage passes with hadoop20. 

 Parameters in core-site.xml and mapred-site.xml are not correctly substituted
 -

 Key: PIG-3145
 URL: https://issues.apache.org/jira/browse/PIG-3145
 Project: Pig
  Issue Type: Bug
Reporter: Cheolsoo Park
Assignee: Cheolsoo Park
 Fix For: 0.12

 Attachments: PIG-3145-2.patch, PIG-3145.patch


 To reproduce the issue, please do the following:
 # Parameterize the address of name node in core-site.xml.
 {code}
   property
 namefs.default.name/name
 valuehdfs://${foo}:8020/value
   /property
 {code}
 # Set the value of foo via -D option.
 {code}
 export PIG_OPTS=-Dfoo=mr1-0.cheolsoo.com
 {code}
 # Pig fails with the following error.
 {code}
 2013-01-28 18:54:02,786 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
 to hadoop file system at: hdfs://${foo}:8020
 2013-01-28 18:54:02,805 [main] ERROR org.apache.pig.Main - ERROR 2999: 
 Unexpected internal error. null
 Details at logfile: /home/cheolsoo/pig-cdh/pig_1359428042522.log
 {code}
 Note that the parameter $\{foo\} in core-site.xml is not expanded. This is 
 because the addresses of name node and job tracker are read directly from 
 core-site.xml instead of reading via Configuration.get().
 {code:title=HExecutionEngine.java}
 // properties is Java Properties
 cluster = properties.getProperty(JOB_TRACKER_LOCATION);
 nameNode = properties.getProperty(FILE_SYSTEM_LOCATION);
 {code}
 Replacing these lines with Configuration.get() fixes the issue.
 {code:title=HExecutionEngine.java}
 // jc is Hadoop Configuration
 cluster = jc.get(JOB_TRACKER_LOCATION);
 nameNode = jc.get(FILE_SYSTEM_LOCATION);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3145) Parameters in core-site.xml and mapred-site.xml are not correctly substituted

2013-02-01 Thread Santhosh Srinivasan (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13569146#comment-13569146
 ] 

Santhosh Srinivasan commented on PIG-3145:
--

+1 - the changes look good.

 Parameters in core-site.xml and mapred-site.xml are not correctly substituted
 -

 Key: PIG-3145
 URL: https://issues.apache.org/jira/browse/PIG-3145
 Project: Pig
  Issue Type: Bug
Reporter: Cheolsoo Park
Assignee: Cheolsoo Park
 Attachments: PIG-3145.patch


 To reproduce the issue, please do the following:
 # Parameterize the address of name node in core-site.xml.
 {code}
   property
 namefs.default.name/name
 valuehdfs://${foo}:8020/value
   /property
 {code}
 # Set the value of foo via -D option.
 {code}
 export PIG_OPTS=-Dfoo=mr1-0.cheolsoo.com
 {code}
 # Pig fails with the following error.
 {code}
 2013-01-28 18:54:02,786 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
 to hadoop file system at: hdfs://${foo}:8020
 2013-01-28 18:54:02,805 [main] ERROR org.apache.pig.Main - ERROR 2999: 
 Unexpected internal error. null
 Details at logfile: /home/cheolsoo/pig-cdh/pig_1359428042522.log
 {code}
 Note that the parameter $\{foo\} in core-site.xml is not expanded. This is 
 because the addresses of name node and job tracker are read directly from 
 core-site.xml instead of reading via Configuration.get().
 {code:title=HExecutionEngine.java}
 // properties is Java Properties
 cluster = properties.getProperty(JOB_TRACKER_LOCATION);
 nameNode = properties.getProperty(FILE_SYSTEM_LOCATION);
 {code}
 Replacing these lines with Configuration.get() fixes the issue.
 {code:title=HExecutionEngine.java}
 // jc is Hadoop Configuration
 cluster = jc.get(JOB_TRACKER_LOCATION);
 nameNode = jc.get(FILE_SYSTEM_LOCATION);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3145) Parameters in core-site.xml and mapred-site.xml are not correctly substituted

2013-01-30 Thread Jarek Jarcec Cecho (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566856#comment-13566856
 ] 

Jarek Jarcec Cecho commented on PIG-3145:
-

+1 (non-binding)

The change seems reasonable to me. 

Jarcec

 Parameters in core-site.xml and mapred-site.xml are not correctly substituted
 -

 Key: PIG-3145
 URL: https://issues.apache.org/jira/browse/PIG-3145
 Project: Pig
  Issue Type: Bug
Reporter: Cheolsoo Park
Assignee: Cheolsoo Park
 Attachments: PIG-3145.patch


 To reproduce the issue, please do the following:
 # Parameterize the address of name node in core-site.xml.
 {code}
   property
 namefs.default.name/name
 valuehdfs://${foo}:8020/value
   /property
 {code}
 # Set the value of foo via -D option.
 {code}
 export PIG_OPTS=-Dfoo=mr1-0.cheolsoo.com
 {code}
 # Pig fails with the following error.
 {code}
 2013-01-28 18:54:02,786 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
 to hadoop file system at: hdfs://${foo}:8020
 2013-01-28 18:54:02,805 [main] ERROR org.apache.pig.Main - ERROR 2999: 
 Unexpected internal error. null
 Details at logfile: /home/cheolsoo/pig-cdh/pig_1359428042522.log
 {code}
 Note that the parameter $\{foo\} in core-site.xml is not expanded. This is 
 because the addresses of name node and job tracker are read directly from 
 core-site.xml instead of reading via Configuration.get().
 {code:title=HExecutionEngine.java}
 // properties is Java Properties
 cluster = properties.getProperty(JOB_TRACKER_LOCATION);
 nameNode = properties.getProperty(FILE_SYSTEM_LOCATION);
 {code}
 Replacing these lines with Configuration.get() fixes the issue.
 {code:title=HExecutionEngine.java}
 // jc is Hadoop Configuration
 cluster = jc.get(JOB_TRACKER_LOCATION);
 nameNode = jc.get(FILE_SYSTEM_LOCATION);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3145) Parameters in core-site.xml and mapred-site.xml are not correctly substituted

2013-01-28 Thread Prashant Kommireddi (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565033#comment-13565033
 ] 

Prashant Kommireddi commented on PIG-3145:
--

Hi Cheolsoo, this might also be in a similar category as PIG-3135. Currently 
Pig expects the site xml files on the classpath - again Configuration is not 
used.

 Parameters in core-site.xml and mapred-site.xml are not correctly substituted
 -

 Key: PIG-3145
 URL: https://issues.apache.org/jira/browse/PIG-3145
 Project: Pig
  Issue Type: Bug
Reporter: Cheolsoo Park
Assignee: Cheolsoo Park

 To reproduce the issue, please do the following:
 # Parameterize the address of name node in core-site.xml.
 {code}
   property
 namefs.default.name/name
 valuehdfs://${foo}:8020/value
   /property
 {code}
 # Set the value of foo via -D option.
 {code}
 export PIG_OPTS=-Dfoo=mr1-0.cheolsoo.com
 {code}
 # Pig fails with the following error.
 {code}
 2013-01-28 18:54:02,786 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
 to hadoop file system at: hdfs://${foo}:8020
 2013-01-28 18:54:02,805 [main] ERROR org.apache.pig.Main - ERROR 2999: 
 Unexpected internal error. null
 Details at logfile: /home/cheolsoo/pig-cdh/pig_1359428042522.log
 {code}
 Note that the parameter $\{foo\} in core-site.xml is not expanded. This is 
 because the addresses of name node and job tracker are read directly from 
 core-site.xml instead of reading via Configuration.get().
 {code:title=HExecutionEngine.java}
 // properties is Java Properties
 cluster = properties.getProperty(JOB_TRACKER_LOCATION);
 nameNode = properties.getProperty(FILE_SYSTEM_LOCATION);
 {code}
 Replacing these lines with Configuration.get() fixes the issue.
 {code:title=HExecutionEngine.java}
 // jc is Hadoop Configuration
 cluster = jc.get(JOB_TRACKER_LOCATION);
 nameNode = jc.get(FILE_SYSTEM_LOCATION);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3145) Parameters in core-site.xml and mapred-site.xml are not correctly substituted

2013-01-28 Thread Cheolsoo Park (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565038#comment-13565038
 ] 

Cheolsoo Park commented on PIG-3145:


Hi Prashant,

Thank you very much for pointing that out.

Indeed, PIg-3135 is in a similar category, but but I think that my problem is a 
bit different from yours. In my case, *-site.xml files are present in 
classpath, but system properties (-Dkey=value) that are passed to JVM are not 
honored. They are different, aren't they?

 Parameters in core-site.xml and mapred-site.xml are not correctly substituted
 -

 Key: PIG-3145
 URL: https://issues.apache.org/jira/browse/PIG-3145
 Project: Pig
  Issue Type: Bug
Reporter: Cheolsoo Park
Assignee: Cheolsoo Park

 To reproduce the issue, please do the following:
 # Parameterize the address of name node in core-site.xml.
 {code}
   property
 namefs.default.name/name
 valuehdfs://${foo}:8020/value
   /property
 {code}
 # Set the value of foo via -D option.
 {code}
 export PIG_OPTS=-Dfoo=mr1-0.cheolsoo.com
 {code}
 # Pig fails with the following error.
 {code}
 2013-01-28 18:54:02,786 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
 to hadoop file system at: hdfs://${foo}:8020
 2013-01-28 18:54:02,805 [main] ERROR org.apache.pig.Main - ERROR 2999: 
 Unexpected internal error. null
 Details at logfile: /home/cheolsoo/pig-cdh/pig_1359428042522.log
 {code}
 Note that the parameter $\{foo\} in core-site.xml is not expanded. This is 
 because the addresses of name node and job tracker are read directly from 
 core-site.xml instead of reading via Configuration.get().
 {code:title=HExecutionEngine.java}
 // properties is Java Properties
 cluster = properties.getProperty(JOB_TRACKER_LOCATION);
 nameNode = properties.getProperty(FILE_SYSTEM_LOCATION);
 {code}
 Replacing these lines with Configuration.get() fixes the issue.
 {code:title=HExecutionEngine.java}
 // jc is Hadoop Configuration
 cluster = jc.get(JOB_TRACKER_LOCATION);
 nameNode = jc.get(FILE_SYSTEM_LOCATION);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3145) Parameters in core-site.xml and mapred-site.xml are not correctly substituted

2013-01-28 Thread Prashant Kommireddi (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565042#comment-13565042
 ] 

Prashant Kommireddi commented on PIG-3145:
--

Agreed. I am guessing your changes would go into HExecutionEngine primarily. 
Would be good to get both changes in around the same time.

 Parameters in core-site.xml and mapred-site.xml are not correctly substituted
 -

 Key: PIG-3145
 URL: https://issues.apache.org/jira/browse/PIG-3145
 Project: Pig
  Issue Type: Bug
Reporter: Cheolsoo Park
Assignee: Cheolsoo Park

 To reproduce the issue, please do the following:
 # Parameterize the address of name node in core-site.xml.
 {code}
   property
 namefs.default.name/name
 valuehdfs://${foo}:8020/value
   /property
 {code}
 # Set the value of foo via -D option.
 {code}
 export PIG_OPTS=-Dfoo=mr1-0.cheolsoo.com
 {code}
 # Pig fails with the following error.
 {code}
 2013-01-28 18:54:02,786 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
 to hadoop file system at: hdfs://${foo}:8020
 2013-01-28 18:54:02,805 [main] ERROR org.apache.pig.Main - ERROR 2999: 
 Unexpected internal error. null
 Details at logfile: /home/cheolsoo/pig-cdh/pig_1359428042522.log
 {code}
 Note that the parameter $\{foo\} in core-site.xml is not expanded. This is 
 because the addresses of name node and job tracker are read directly from 
 core-site.xml instead of reading via Configuration.get().
 {code:title=HExecutionEngine.java}
 // properties is Java Properties
 cluster = properties.getProperty(JOB_TRACKER_LOCATION);
 nameNode = properties.getProperty(FILE_SYSTEM_LOCATION);
 {code}
 Replacing these lines with Configuration.get() fixes the issue.
 {code:title=HExecutionEngine.java}
 // jc is Hadoop Configuration
 cluster = jc.get(JOB_TRACKER_LOCATION);
 nameNode = jc.get(FILE_SYSTEM_LOCATION);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira